View Issue Details

IDProjectCategoryView StatusLast Update
00035043 - Current Dev ListBugpublic2019-11-14 20:15
ReporterPD9FERAssigned ToDOUG 
PrioritynormalSeveritymajorReproducibilityalways
Status assignedResolutionopen 
Product Version 
Target VersionFixed in Version 
Summary0003504: Using QSO Forwarding FT4 is put in the log as MFSK
DescriptionThe problem lies in using QSO forwarding with WSJT-X

WSJT-X Sends ADIF Data as Mode:MFSK Submode:FT4
This causes Logbook to add the QSO as MFSK
Need a workaround so Logbook looks at the Submode string send over via QSO Forwarding.

So logging with JTALERT-X sends Mode FT4
QSO<EOH><CALL:4>LY5O<QSO_DATE:8>20190923<TIME_ON:6>165100<TIME_OFF:6>165300<FREQ:8>14.08084<FREQ_RX:8>14.08084<BAND:3>20m<BAND_RX:3>20m<MODE:3>FT4<RST_SENT:3>-03<RST_RCVD:3>-07<QSL_SENT:1>N<QSL_RCVD:1>N<NAME:7>Bronius<QTH:8>Amerikos<CQZ:2>15<ITUZ:2>29<PFX:3>LY5<CONT:2>EU<DXCC:3>146<COUNTRY:9>Lithuania<GRIDSQUARE:6>KO14wv<DISTANCE:4>1247<TX_PWR:1>5<A_INDEX:1>0<K_INDEX:1>0<SFI:1>0<COMMENT:33>Tnx for FT4 Contact, 73 de PD9FER<MY_GRIDSQUARE:6>JO22OQ<MY_CQ_ZONE:2>14<MY_ITU_ZONE:2>27<STATION_CALLSIGN:6>PD9FER<OPERATOR:6>PD9FER<QSO_COMPLETE:1>Y<EOR>


But WSJT-X via QSO Forwarding produces MFSK
 <call:4>LY5O <gridsquare:4>KO14 <mode:4>MFSK <submode:3>FT4 <rst_sent:3>-03 <rst_rcvd:3>-07 <qso_date:8>20190923 <time_on:6>165138 <qso_date_off:8>20190923 <time_off:6>165337

We need to add some code that monitors QSO Forwarding .
When something with Submode FT4 is seen, then set Mode to FT4 (rewrite the Mode)
Steps To ReproduceIn Logbook set Tools > Configure > Modes to Defaults
Run some FT4 QSO's with WSJT-X via QSO Forwarding.

Note that they will be Logged as MFSK but should be FT4.
Additional InformationComment from Tim:

What I'm seeing here we are chasing our tails. We have people using JTAlert and JTAlert logs to LB and logs the MODE as FT4. Here is an ADI from JTAlert. Take note of the <MODE:3>FT4 field with NO <submode> field.

 <CALL:4>LY5O<QSO_DATE:8>20190923<TIME_ON:6>165100<TIME_OFF:6>165300<FREQ:8>14.08084<FREQ_RX:8>14.08084<BAND:3>20m<BAND_RX:3>20m<MODE:3>FT4<RST_SENT:3>-03<RST_RCVD:3>-07<QSL_SENT:1>N<QSL_RCVD:1>N<NAME:7>Bronius<QTH:8>Amerikos<CQZ:2>15<ITUZ:2>29<PFX:3>LY5<CONT:2>EU<DXCC:3>146<COUNTRY:9>Lithuania<GRIDSQUARE:6>KO14wv<DISTANCE:4>1247<TX_PWR:1>5<A_INDEX:1>0<K_INDEX:1>0<SFI:1>0<COMMENT:33>Tnx for FT4 Contact, 73 de PD9FER<MY_GRIDSQUARE:6>JO22OQ<MY_CQ_ZONE:2>14<MY_ITU_ZONE:2>27<STATION_CALLSIGN:6>PD9FER<OPERATOR:6>PD9FER<QSO_COMPLETE:1>Y<EOR>

 Here is the very same contact being logged using QSO FWD. Note that <mode:4>MFSK <submode:3>FT4. This is completely different from the way it's logged from JTAlert.

 <call:4>LY5O <gridsquare:4>KO14 <mode:4>MFSK <submode:3>FT4 <rst_sent:3>-03 <rst_rcvd:3>-07 <qso_date:8>20190923 <time_on:6>165138 <qso_date_off:8>20190923 <time_off:6>165337 <band:3>20m <freq:9>14.080840 <station_callsign:6>PD9FER <my_gridsquare:6>JO22OQ <tx_pwr:2>15 <comment:33>Tnx for FT4 Contact, 73 de PD9FER <operator:6>PD9FER <eor>

 The way QSO FWD is handling the <mode:4>MFSK <submode:3>FT4 is the correct way this should be handled, however, since we don't have a "submode" field in our logbook, we need the qso to be logged so that the <submode:3>FT4 shows up in the "Mode" column of the HRD Logbook when we are logging using the QSO FWD. This way both the logging from JTAlert AND QSO FWD are logging the same in our logbook.

 In our Logbook, when the contact is logged, we need it to show as "FT4" in the Mode field for 2 reasons. #1 - Since we don't have the "Mode" and "Submode" fields, we need to have "FT4" indicated in the "Mode" field so we can identify the contact as an FT4 contact at a later date. #3 - If our log shows these FT4 contacts as "MFSK" in the Mode column, if we have logged contacts we actually work in the MFSK mode, and have that logged for those contacts, in the future when we look at our logbooks, how are we going to tell which are FT4 contacts and which are true MFSK contacts?

 For more evidence that we are logging incorrectly, please see the test Ron Whitsel and I did in OSTicket #803567.
TagsNo tags attached.
ModuleLogbook
Sub-ModuleFunctional
TestingNot Started

Relationships

related to 0003394 closedPD9FER Ham Radio Deluxe Add FT4 Mode to standard Mode list in Logbook 

Activities

WA9PIE

2019-10-23 21:38

administrator   ~0008996

I pulled this over from the backlog. This is the issue that customers are complaining about.

g3ucq

2019-10-29 09:04

updater   ~0009079

I can confirm this issue.

g3ucq

2019-10-29 09:25

updater   ~0009080

With an FT4 QSO forwarded from WSJT-X as MFSK, a LOTW download does not match

K7ZCZ

2019-10-29 16:45

administrator   ~0009092

Here, it seems like the term "QSO Forwarding" is being used to describe the Logbook listening for QSOs broadcast by other software. I'd call this "QSO Listening". These actions are from the point of view of the Logbook. If the Logbook is forwarding, it's sending to other apps. If the Logbook is listening, it's listing from other apps.

With that in hand, it seems like the report involves the QSO Listener being sent, from WSJT-X, an ADIF record that contains a submode and a mode. Because we don't support SubMode, we just ignore the SubMode value. For FT4 QSOs, ignoring the SubMode value isn't desirable because we'll record "MFSK" as the mode. The complaint is that, instead, we really want to record a mode of "FT4" so that such records can participate in workarounds elsewhere in the product.

Further, it seems like we're saying that JTAlert sends a QSO to the Logbook QSO Listener that has a value of "FT4" for the Mode field and no value for the SubMode field. Sounds like we're fine with that behaviour and no fix is necessary.

With the above interpretations, then, my understanding is that we want to augment the QSO Listener in the Logbook so that when records are received with a value of "MFSK" in the Mode field, and a value of "FT4" in the SubMode field, they're mapped to a record with no SubMode value and a Mode value of "FT4".

Is this correct?

Pursuing this fix isn't difficult to implement. But this path would have us making records in customer databases that aren't exactly much better than they are today because they don't map to the reality we want to model in the scope of the database. We lose fidelity. There are about 100 submodes in the ADIF spec, and this approach addresses only one. Ignoring all others, like USB and LSB being SubModes of SSB.

The product only unmaps Mode/SubMode when trying to match LotW records and not when dealing with any other QSL service. And not when exporting or importing from any other data srouce or file type. And not when restoring or writing our own backup for our own product. Or in any other circumstances.

Is this pollution of customer data and asymmetric mapping acceptable? I would assume not, since it probably makes future fixes precarious. At the very least, we should investigate symmetrical mappings for other services and reason out what our path might be when updating the product to address SubMode fully.

When we do accept such a record from QSO Listening and add it to the Logbook, we're obliged to forward it in the QSO forwarding feature for other listening applications. If the described mapping is implemented, should that forwarded record contain a value of "FT4" for the Mode field and no value for SubMode; or should it have the Mode value of "MFSK" and the SubMode value of "FT4", as originally received?

WA9PIE

2019-10-29 16:49

administrator   ~0009093

As I understand it, I believe that is correct.

This is a reasonable work-around until we can support submodes.

g3ucq

2019-10-29 17:03

updater   ~0009095

Today I made some FT4 qsos.
Having been forwarded from WSJT-X the qsos appeared as MFSK in the Logbook.
I uploaded them to LOTW.
The LOTW download showed the Mode as DATA and there were no matches to my FT4/MFSK qsos.
I changed the Mode in the Logbook from MFSK to FT4 and then the LOTW download reported matches.
After the download, the FT4 qsos did not have "Verified (match)" against them. I had to do that manually.
Is that acceptable?

K7ZCZ

2019-10-29 17:42

administrator   ~0009096

This shelf set contains what should be a fix, given the description above.
https://hrdsoftware.visualstudio.com/HRD/_versionControl/shelveset?ss=workaround%20in%20UDP%20Listener%20for%20FT4%20mode%3Bmikeblas%40msn.com

I have strong reservations about this approach so I won't be checking it in. Any of the other developers can review the code and, if they'd like, check add it to the product.

WA9PIE

2019-10-29 18:32

administrator   ~0009097

Given your reservations, do you have a better idea?

PD9FER

2019-10-30 03:00

updater   ~0009098

Mike B, you are correct.
Same way as I had in mind.

KB3NPH

2019-11-04 11:17

administrator   ~0009131

I got off the phone with Ron Whitsel this morning and here is exactly what needs to be done with this FT4 issue.

The biggest problem is with the ADIF that comes in via QSO Forwarding directly from WSJT-X. I'll get to that in shortly.

The major thing is we MUST have the mode shown in our Logbook as "FT4", and not MFSK, Not DATA nothing but FT4 should show as the mode in our logbook and it should be uploaded as FT4 to LoTW, not MFSK, not DATA but FT4.

Here is the first hurtle we need to jump over, go through or around. Customers who are using JTAlert have two options available for logging FT4 contacts which are used in the ADIF that's sent from JTAlert to the HRD Logbook. The FIRST and CORRECT way is to set the JTAlert so that it creates an ADIF with the "Mode = FT4". There is NO sub mode, no MFSK no nothing, just a simple Mode = FT4 and FT4 is what shows up as the mode in the HRD Logbook. First and foremost, in the Modes table in the HRD Logbook, we MUST have it set to "Mode = FT4" and "ADIF = FT4". This should be the default in the modes table in HRD Logbook.

Now for the actual logging of the contact. This is where we MUST communicate with our customers who are using JTAlert that JTAlert has TWO ways of logging the FT4 contacts that are used when creating the ADIF that is transferred to the HRD Logbook. There is a setting in the JTAlert software where you can configure it so that the MODE = FT4. There is NO sub mode mentioned and no submode in the ADIF that is transferred from JTAlert to the HRD Logbook. When it's written to the HRD Logbook it shows the mode as FT4 in our logbook database. We need to STRESS to our customers who are using JTAlert that THIS is the proper setting to select.

The other option in JTAlert creates an ADIF that logs the Mode = MFSK and a Submode = FT4. This is the option we want OUR customers to ignore and not use when using JTAlert to log between WSJT-X and the HRD Logbook. We really need to thank Laurie for providing the JTAlert users with this option so that JTAlert will be compatible with OUR logbook and also with those who are fully ADIF3 complient and support the logging of sub modes.

Now that we have FT4 being logged as the mode in the HRD logbook from JTAlert, we need to look at the ADIF being sent to the HRD Logbook by WSJT-X for the users who select to use our QSO Forwarding instead of JTAlert to log the contacts to the HRD Logbook.

When we work a contact in WSJT-X and receive the popup that sends the record to the HRD Logbook, the mode in the popup says "FT4". When the ADIF is sent from WSJT-X to the HRD Logbook, it has the "Mode = MFSK" and "Submode = FT4". Here's what we have to do in this case. When our clients are using the QSO Forwarding, we need to intercept this ADIF and have our code change the "Mode = MFSK" to "Mode = FT4, and then log it to the HRD Logbook with the Mode FT4 in the Logbook record.

Now, if we can get the loging from JTAlert and the logging via QSO Forwarding direct from WSJT-X to BOTH create a record in the HRD Logbook with the "Mode: FT4".... we have half the battle won.

The NEXT battle comes when we uplaod these FT4 contacts to LoTW. They should be uploaded to LoTW so that when they land on the LoTW website, the Mode is FT4, which will happen if the mode is FT4 in all of these HRD Logbook records. This is the way it SHOULD be.

The next part of this battle is when we download from LoTW and get the FT4 contacts downloaded from LoTW to MATCH the FT4 contacts in the HRD Logbook. If we REVERT back to the code that was in Beta Build 227 than handled the downloads from LoTW, the FT4 contacts in the ADIF downloaded from LoTW DID MATCH the FT4 contacts in the HRD Logbook. This was proven by a test Ron and I did and reported it earlier in this bug ticket.

If we can somehow do EXACTLY as I have written this note, we will have this FT4 issue resolved to the point where we can release 6.7 and this should satisfy our customers that this has been resolved.

Issue History

Date Modified Username Field Change
2019-10-18 06:41 PD9FER New Issue
2019-10-18 06:46 PD9FER Relationship added related to 0003394
2019-10-23 21:37 WA9PIE Project 1 - Backlog => 3 - Current Dev List
2019-10-23 21:38 WA9PIE Assigned To => K7ZCZ
2019-10-23 21:38 WA9PIE Status new => assigned
2019-10-23 21:38 WA9PIE Category Enhancement => Bug
2019-10-23 21:38 WA9PIE Note Added: 0008996
2019-10-29 09:04 g3ucq Note Added: 0009079
2019-10-29 09:25 g3ucq Note Added: 0009080
2019-10-29 16:45 K7ZCZ Note Added: 0009092
2019-10-29 16:49 WA9PIE Note Added: 0009093
2019-10-29 17:03 g3ucq Note Added: 0009095
2019-10-29 17:42 K7ZCZ Note Added: 0009096
2019-10-29 17:42 K7ZCZ Assigned To K7ZCZ => WA9PIE
2019-10-29 18:32 WA9PIE Assigned To WA9PIE => K7ZCZ
2019-10-29 18:32 WA9PIE Note Added: 0009097
2019-10-30 03:00 PD9FER Note Added: 0009098
2019-11-01 10:32 K7ZCZ Assigned To K7ZCZ => DOUG
2019-11-02 07:30 WA9PIE Assigned To DOUG => K7ZCZ
2019-11-04 11:17 KB3NPH Note Added: 0009131
2019-11-14 20:15 WA9PIE Assigned To K7ZCZ => DOUG