• Squish __ftsc_date bug

    From Oli@2:280/464.47 to All on Saturday, January 30, 2021 12:11:23
    When messages are %RESCANed hpt doesn't use the __ftsc_date field from the Squish message base, but the 2-second precision DOS date. Which means some messages (~50%) are exported with the time one second off. Some tosser's dupe checkers might fail ...

    ---
    * Origin: . (2:280/464.47)
  • From Kai Richter@2:240/77 to Oli on Monday, February 01, 2021 12:43:48
    Hello Oli!

    30 Jan 21, Oli wrote to All:

    When messages are %RESCANed hpt doesn't use the __ftsc_date field from
    the Squish message base, but the 2-second precision DOS date. Which
    means some messages (~50%) are exported with the time one second off.

    I've never seen uneven timestamps in my editor. Hell, at the moment of writing i see 12:43:49 in the header on the from/to lines. But after saving the mail it
    switched to 12:43:48, lucky me. ;-)

    As far as i remember squish does have a 2 seconds resolution. But i don't know why. If i do not see further than my nose i would say 2 second resolution is the standard and if you have uneven timestamps in your msgbase than that is the
    bug.

    Some tosser's dupe checkers might fail ...

    Is this speculation/theory or could you name a real issue?

    Regards

    Kai

    --- GoldED+/LNX 1.1.4.7
    * Origin: Monobox (2:240/77)
  • From Kai Richter@2:240/77 to Oli on Tuesday, February 02, 2021 22:45:46
    Hello Oli!

    01 Feb 21, Oli wrote to Kai Richter:

    say 2 second resolution is the standard and if you have uneven
    timestamps in your msgbase than that is the bug.

    The DOS time format uses a 2 seconds resolution. The creation and
    received date is stored has DOS time in the Squish message base.

    After export the pkt message does no longer have creation date? As for FTS-0001
    it's the last edited date?

    Some tosser's dupe checkers might fail ...

    Isn't the MSGID the solution for that problem?

    Is this speculation/theory or could you name a real issue?

    This is what other nodes have reported and I have no reason to doubt
    it. Don't you think there are tossers out there which believe that
    messages with different time stamps are not the same?

    I don't know (out there). I think because of ASCII DateTime would not effect DOS 16-bit file format date stamps it should be ok to see different time stamps
    as different mails.

    I don't doubt that the problem exist but for fault confirmation and trouble shooting it is easier to know the software in use.

    doesn't matter, because per Squish specification the __ftsc_date

    I will forward you my messy research for the format. In short: For FTS-0001 DateTime is the creation date of the message. A packed message use the last edited date. For type 2 pkt format it's the pkt creation date.

    After reading those specs i would recomment msgid for dupe checking. ;-)

    should be used on export (rescan). What would be the benefit of not
    using the exact time when it's available?

    Same like if using: Nothing? Even with a fix available - every hpt system have to use that fix to get rid of the problem. And i think we both know that that is most unlikely to happen.

    Regards

    Kai

    --- GoldED+/LNX 1.1.4.7
    * Origin: Monobox (2:240/77)
  • From Oli@2:280/464.47 to Kai Richter on Wednesday, February 03, 2021 12:23:47
    Kai wrote (2021-02-02):

    Hello Oli!

    01 Feb 21, Oli wrote to Kai Richter:

    say 2 second resolution is the standard and if you have uneven
    timestamps in your msgbase than that is the bug.

    The DOS time format uses a 2 seconds resolution. The creation and
    received date is stored has DOS time in the Squish message base.

    After export the pkt message does no longer have creation date? As for FTS-0001 it's the last edited date?

    Let's consult the Squish Developers Kit. This stuff is stored in the message header in the .sqd file.

    The XMSG structure has the following format:
    [...]
    date_written SCOMBO 164 Date that the message was written.

    date_arrived SCOMBO 168 Date that the message was placed in
    [...]
    __ftsc_date char[20] 218 FTS-0001 compatible date. Squish
    applications should not access this
    field directly. This field is used
    exclusively by tossers and scanners
    for preserving the original ASCII
    message date. Squish applications
    should use the binary dates in
    date_written and date_arrived to
    retrieve the message date.

    If a message is exported from the Squish message base and has an non-empty __ftsc_date field, the tosser should use the contents of that field for the date. If the message has been created locally (no __ftsc_date), the tosser uses
    the date_written field. (SCOMBO is DOS datetime)

    Some tosser's dupe checkers might fail ...

    Isn't the MSGID the solution for that problem?

    Yes, MSGIDs is very effective in sort out dupes, but not all messages have a MSGID (see your argument below: old software).

    Is this speculation/theory or could you name a real issue?

    This is what other nodes have reported and I have no reason to doubt
    it. Don't you think there are tossers out there which believe that
    messages with different time stamps are not the same?

    I don't know (out there). I think because of ASCII DateTime would not effect DOS 16-bit file format date stamps it should be ok to see
    different time stamps as different mails.

    I don't doubt that the problem exist but for fault confirmation and trouble shooting it is easier to know the software in use.

    hpt does it wrong. I tested it. What else is there to trouble shoot?

    doesn't matter, because per Squish specification the __ftsc_date

    I will forward you my messy research for the format. In short: For FTS-0001 DateTime is the creation date of the message. A packed message use the last edited date. For type 2 pkt format it's the pkt creation date.

    After reading those specs i would recomment msgid for dupe checking. ;-)

    not sure this has anything to do with __ftsc_date in the Squish message base.

    should be used on export (rescan). What would be the benefit of not
    using the exact time when it's available?

    Same like if using: Nothing? Even with a fix available - every hpt system have to use that fix to get rid of the problem. And i think we both know that that is most unlikely to happen.

    It only affects systems that use a Squish database and have downlinks that do a
    %RESCAN. If it's not fixed it will be there forever. If it were fixed, affected
    systems could be upgraded to a newer hpt version. Just because some sysops are to lazy to use up-to-date software doesn't mean we shouldn't care about bugs and bug fixing.

    It's just a simple bug ...

    ---
    * Origin: . (2:280/464.47)
  • From Kai Richter@2:240/77 to Oli on Thursday, February 04, 2021 00:19:20
    Hello Oli!

    03 Feb 21, Oli wrote to Kai Richter:

    The XMSG structure has the following format:

    Thanks, you found the XMSG definition i was looking for.

    I don't doubt that the problem exist but for fault confirmation
    and trouble shooting it is easier to know the software in use.

    hpt does it wrong. I tested it. What else is there to trouble shoot?

    The other side of the medal. As said i'm no coder and my search for hpt found this one:

    https://github.com/huskyproject/hpt/search?q=__ftsc_date&type=

    src/scan.c
    96 msg->destAddr.point = xmsg.dest.point;
    97 strcpy((char *)msg->datetime, (char *)xmsg.__ftsc_date);
    98 xstrcat(&(msg->subjectLine), (char *)xmsg.subj);

    This looks like the xmsg.* is the squish base message.
    It looks like the ftsc date is overwritten by some copy -> wrong, i'm no coder.

    https://www.cplusplus.com/reference/cstring/strcpy/

    char * strcpy ( char * destination, const char * source );

    Unlike usual copy the C++ strcpy works reversed, the scan.c of hpt get the original squish base ftsc data and uses it for the datetime field.

    It's just a simple bug ...

    If it is it would be fixed already.
    You reported this one a year ago, don't you?

    My knowledge does end here. It was interesting but i can't help you further.

    It only affects systems that use a Squish database and have downlinks
    that do a %RESCAN.

    And it does apply to mails without msgid only. Good luck.

    Regards

    Kai

    --- GoldED+/LNX 1.1.4.7
    * Origin: Monobox (2:240/77)
  • From Oli@2:280/464.47 to Kai Richter on Wednesday, February 10, 2021 09:56:01
    Kai wrote (2021-02-03):

    It's just a simple bug ...

    If it is it would be fixed already.
    You reported this one a year ago, don't you?

    I don't know why it doesn't get fixed. Maybe nobody wants to touch that part of
    the code anymore. A project where critical bugs doesn't get fixed anymore is kind of abandoned.

    ---
    * Origin: . (2:280/464.47)
  • From andrew clarke@3:633/267 to Oli on Wednesday, February 10, 2021 20:32:56
    10 Feb 21 09:56, you wrote to Kai Richter:

    It's just a simple bug ...

    If it is it would be fixed already.
    You reported this one a year ago, don't you?

    I don't know why it doesn't get fixed. Maybe nobody wants to touch that part of the code anymore. A project where critical bugs doesn't get
    fixed anymore is kind of abandoned.

    I suspect the patch below fixes this, but I've not tested it.

    It's also necessary to verify it doesn't break when rescanning *.MSG and JAM bases.

    diff --git a/src/scanarea.c b/src/scanarea.c
    index 60ec71bf..8926aa30 100644
    -+- a/src/scanarea.c
    +++ b/src/scanarea.c
    @@ -88,7 +88,7 @@ void makeMsg(HMSG hmsg, XMSG xmsg, s_message * msg, s_area * echo, int action)
    msg->destAddr.point = xmsg.dest.point;
    msg->attributes = xmsg.attr & ~MSGLOCAL; /* msg should not have MSGLOCAL bit set */

    - sc_time((union stamp_combo *)&(xmsg.date_written), (char *)msg->datetime); + strcpy((char *)msg->datetime, (char *) xmsg.__ftsc_date);
    xstrcat(&msg->toUserName, (char *)xmsg.to);
    xstrcat(&msg->fromUserName, (char *)xmsg.from);
    xstrcat(&msg->subjectLine, (char *)xmsg.subj);

    --- GoldED+/BSD 1.1.5-b20180707
    * Origin: Blizzard of Ozz, Melbourne, Victoria, Australia (3:633/267)
  • From Oli@2:280/464.47 to andrew clarke on Wednesday, February 10, 2021 11:04:00
    andrew wrote (2021-02-10):

    I suspect the patch below fixes this, but I've not tested it.

    It's also necessary to verify it doesn't break when rescanning *.MSG and JAM bases.

    - sc_time((union stamp_combo *)&(xmsg.date_written), (char
    *)msg->datetime);
    + strcpy((char *)msg->datetime, (char *) xmsg.__ftsc_date);

    What happens if xmsg.__ftsc_date is empty? Local messages don't have an __ftsc_date and it could also be missing for other reasons (messages converted from another message base format, copied/moved in a message editor or tossed by
    a tosser that doesn't copy the __ftsc_date).

    ---
    * Origin: . (2:280/464.47)
  • From Michael Dukelsky@2:5020/1042 to Oli on Wednesday, February 10, 2021 12:17:56
    Hello Oli,

    Wednesday February 10 2021, Oli wrote to Kai Richter:

    It's just a simple bug ...

    If it is it would be fixed already.
    You reported this one a year ago, don't you?

    I don't know why it doesn't get fixed. Maybe nobody wants to touch
    that part of the code anymore. A project where critical bugs doesn't
    get fixed anymore is kind of abandoned.

    So, where is a pull request from you? I don't see it.

    As to me, I have a different view on what is critical and what is not so critical. I have a long list of what should be done in Husky and I pick the items from the list according to my idea of which item is more important to me.
    I may also pick a somewhat less important item if it requires a little time. And I have my life besides Fidonet, particularly I still have a job despite the
    fact that I am 73 y/o. So I cannot work for Fidonet full time, but only when I have some time for it.

    Michael

    ... node (at) f1042 (dot) ru
    --- GoldED+/LNX 1.1.5-b20180707
    * Origin: Moscow, Russia (2:5020/1042)
  • From andrew clarke@3:633/267 to Oli on Wednesday, February 10, 2021 23:38:16
    10 Feb 21 11:04, you wrote to me:

    I suspect the patch below fixes this, but I've not tested it.

    It's also necessary to verify it doesn't break when rescanning
    *.MSG and JAM bases.

    - sc_time((union stamp_combo *)&(xmsg.date_written), (char
    *)msg->datetime); + strcpy((char *)msg->datetime, (char *)
    xmsg.__ftsc_date);

    What happens if xmsg.__ftsc_date is empty? Local messages don't have an __ftsc_date and it could also be missing for other reasons (messages converted from another message base format, copied/moved in a message editor or tossed by a tosser that doesn't copy the __ftsc_date).

    Good point. But it's easy to test for that condition.

    if (*xmsg.__ftsc_date == '\0')
    {
    /* __ftsc_date is empty, so use xmsg.date_written */
    sc_time( ...
    }
    else
    {
    /* use xmsg.__ftsc_date */
    strcpy( ...
    }

    --- GoldED+/BSD 1.1.5-b20180707
    * Origin: Blizzard of Ozz, Melbourne, Victoria, Australia (3:633/267)
  • From Oli@2:280/464.47 to andrew clarke on Wednesday, February 10, 2021 15:17:59
    andrew wrote (2021-02-10):

    10 Feb 21 11:04, you wrote to me:

    What happens if xmsg.__ftsc_date is empty? Local messages don't
    have an __ftsc_date and it could also be missing for other reasons
    (messages converted from another message base format, copied/moved
    in a message editor or tossed by a tosser that doesn't copy the
    __ftsc_date).

    Good point. But it's easy to test for that condition.

    if (*xmsg.__ftsc_date == '\0')
    {
    /* __ftsc_date is empty, so use xmsg.date_written */
    sc_time( ...
    }
    else
    {
    /* use xmsg.__ftsc_date */
    strcpy( ...
    }

    That seems to work:

    $ pktview outbound/23e73b00.pkt | grep 'Date'

    Date and time : 09 Feb 21 13:14:00
    Date and time : 09 Feb 21 06:31:33
    Date and time : 09 Feb 21 05:08:19
    Date and time : 07 Feb 21 18:49:35
    Date and time : 10 Feb 21 03:55:02
    Date and time : 10 Feb 21 14:58:42

    Last one is a message with an empty __ftsc_date field.
    (I had to use an hex editor, because Golded does write the __ftsc_date for messages created in the editor, even the Squish developer documentation strongly advises against it).

    I haven't tested JAM and .MSG message bases.

    ---
    * Origin: . (2:280/464.47)
  • From Kai Richter@2:240/77 to Oli on Thursday, February 11, 2021 00:39:12
    Hello Oli!

    10 Feb 21, Oli wrote to andrew clarke:

    Date and time : 10 Feb 21 14:58:42

    Last one is a message with an empty __ftsc_date field.
    (I had to use an hex editor, because Golded does write the __ftsc_date
    for messages created in the editor, even the Squish developer
    documentation strongly advises against it).

    __ftsc_date char[20] 218 FTS-0001 compatible date. Squish
    applications should not access this
    field directly. This field is used
    exclusively by tossers and scanners
    for preserving the original ASCII
    message date.

    If the first tosser/scanner touches the message, where does the original ASCII date comes from?

    Regards

    Kai

    --- GoldED+/LNX 1.1.4.7
    * Origin: Monobox (2:240/77)
  • From Oli@2:280/464.47 to Michael Dukelsky on Sunday, February 14, 2021 13:14:42
    Michael wrote (2021-02-10):

    I don't know why it doesn't get fixed. Maybe nobody wants to touch
    that part of the code anymore. A project where critical bugs doesn't
    get fixed anymore is kind of abandoned.

    So, where is a pull request from you? I don't see it.

    I'm not a hpt user and I'm not involved with the projects.

    As to me, I have a different view on what is critical and what is not so critical. I have a long list of what should be done in Husky and I pick the items from the list according to my idea of which item is more important to me. I may also pick a somewhat less important item if it requires a little time. And I have my life besides Fidonet, particularly
    I still have a job despite the fact that I am 73 y/o. So I cannot work
    for Fidonet full time, but only when I have some time for it.

    I didn't address you directly. Of course you are free to do with hpt / husky whatever you like and label bugs and todos as you wish.

    hpt is a software that interacts with the network. When there is a bug that could cause problems in the network, I call that critical. That's my opinion, nothing more. It's an open source project, nobody have to do anything. But if these problems don't get fixed over an extended period of time, I don't think it's wrong to address it and remind people that they are using broken software.
    If people get annoyed by these reminders, there is an easy solution: fix the bugs that affects the network. Or don't and ignore it. But don't expect that people will not complain.

    Btw, hunting down a bug, writing a bug report and testing the fixes is also time consuming.

    ---
    * Origin: . (2:280/464.47)