View Issue Details

IDProjectCategoryView StatusLast Update
0002795Ham Radio DeluxeBugpublic2018-09-11 13:18
ReporterWA9PIE 
Assigned ToK7ZCZ 
PrioritynormalSeveritymajorReproducibilityalways
Status closedResolutionfixed 
Product Version6.4.0.858 
Target VersionFixed in Version6.4.0.886 
Summary0002795: QSO Forwarding (UDP broadcast) API in Logbook doesn't work (for WSJT-X).
DescriptionThe (so-called) "QSO Forwarding" feature in Logbook doesn't work. This method of broadcasting QSO information in a standard format over a UDP broadcast on the localhost is used by many logging programs these days to enable QSO information to flow between them. Popular programs like WSJT-X and N1MM do this... but there are many other examples (I'll include links to them below).

These applications have a SEND port and a RECEIVE port. An application sends a broadcast over an assigned UDP/IP port (2333, for example) and other applications listening to that port capture that data and take some action on it. In the case of Logbook, it's there to create a QSO record as-if it were created natively in Logbook (with callsign lookup, etc).
Steps To ReproduceIn Logbook, Tools/Configure/QSO Forwarding.
Set the SEND and RECEIVE ports up there to match the configuration of a 3rd party software (like WSJT-X).
Make a QSO in WSJT-X and click the "Log QSO" button. When the resulting dialog box comes up, click "Ok".
At this point, a UDP broadcast will be sent (captured in the PCAP attached to this record).
No QSO is created in Logbook.
TagsNo tags attached.
ModuleLogbook
Sub-ModuleInterfacing
Testing Beta Successful

Relationships

related to 0002858 closedWA9PIE Logbook: QSO Forwarding options are not correctly scoped 
related to 0002867 closedK7ZCZ Logbook: Databases Manager could use some enhancement 

Activities

WA9PIE

2018-06-30 10:49

administrator  

WSJT-x dumpfile02.pcap (56,948 bytes)

vk2byi

2018-07-01 23:30

viewer   ~0005590

WSJT-X implements the ‘9. Sending Log Data to N1MM+’ specification as part of the N1MM Contest Logger application.
     • Refer: ‘http://n1mm.hamdocs.com/tiki-index.php?page=UDP+Broadcasts#Sending_Log_Data_to_N1MM_

Whereas Logbook implements part of the ‘5. XML Schema and Broadcast Field Lists’ specification, the ‘5.2.1. Contact Info UDP Packet’ specification
     • Refer ‘http://n1mm.hamdocs.com/tiki-index.php?page=UDP+Broadcasts#Contact_Info_UDP_packet

These message formats are different which is why a contact is not being added in Logbook. The only way that a contact can be forwarded directly from WSJT-X to Logbook is if a new API that supports the ‘9. Sending Log Data to N1MM+’ specification is implemented.

Existing users of WSJT-X and Logbook use JTAlert-X (https://hamapps.com/) which captures the QSO details being logged by WSJT-X and logs the contact in Logbook directly if ‘HRD V5/V6’ logging is enabled in JTAlertX, or via my QSO Relay application if ‘Standard ADIF File’ logging is enabled in JTAlertX. In either case, both JTAlertX and QSO Relay using the TCP/IP Logbook API on port 7826 to add the contact to Logbook.

As far as the QSO Forwarding feature in Logbook is concerned, the RECEIVE port side of this feature does work, however, there is a bug in the SEND port side. I have attached ‘QSO Forwarding Timestamp.pdf’ that documents this defect.

Also, the QSO Forwarding RECEIVE port side will not process a Contact Info message that contains an encoding directive in the XML header. In the absence of the optional encoding declaration, an XML processor receiving this XML should assume it to be UTF-8, it being the standard. However, it might be better to declare the encoding as UTF-8 to avoid errors, by using <?xml version="1.0" encoding="UTF-8"?> as the XML header. However, if the encoding directive is included, nothing is logged, and nothing is rebroadcast currently. I have also attached ‘QSO Forwarding XML Header.pdf’ that documents this test failing.

It would be my suggestion that the QSO Forwarding feature in Logbook be extended to implement the N1MM ‘5.2 Contact’ specification (i.e. sections 5.2.1, 5.2.2 and 5.2.3) or something similar so that contacts in Logbook can be inserted, updated or deleted by third-party applications via the RECEIVE port, or third-party applications can be notified of insertion, updates or deletes in Logbook via the SEND port.

QSO Forwarding XML Header.pdf (12,311 bytes)
QSO Forwarding Timestamp.pdf (112,317 bytes)

WA9PIE

2018-08-24 00:17

administrator   ~0006002

MikeB...

With the information I've added... and VK2BYI has added... can you take a look at this and see what additional information is required to fix this?

In particular, I'd like this to work so that WSJT-X can be used to log QSOs without JTAlert. But from what Chris points out here, this likely also resolves the matter for N1MM contest fans as well.

K7ZCZ

2018-08-25 14:24

manager   ~0006006

The N1MM documentation that Chris links is helpful, though not exactingly specific.

Let's examine what it says, and let me inject what my interpreations are.

There's Section 5, which describes an XML-like format for some data. As far as I can tell, a datagram of this format is sent from N1MM to any application which happens to be listening. A transmission from N1MM means that N1MM has logged a QSO and wishes to broadcast that fact to anyone who's listening. (Section 5 isn't explicit at saying that the described format is what N1MM sends.)

Then there's Section 9, which describes an AIDF-like format for data. It looks like N1MM will listen to broadcasts of datagrams in this format. When it receives one, it uses the information inside to get add a record to its log. (Section 9 is pretty clear about saying that it's what N1MM expects to receive.)

The arrangement of different incoming and outgoing protocol establishes directionality where we must know which type of listener we're broadcasting into. Or, in the case of HRD, which kind of listener we mean to simulate. It's something like the null-modem problem; a terminal has one pinout and a modem has another. If we ever want to connect a terminal to another terminal, then we've got to find a way to flip the pinouts.

In the context of N1MM, neither section advises what programs other than N1MM should do, and that's where HRD runs into trouble. Is HRD talking to N1MM, or is it a replacement for N1MM?

The HRD "QSO Forwarding" UI says that it can either listen or send.

When listening, HRD assumes that it is listening for broadcasts coming from N1MM. HRD expects these announcements to be in the Section 5 format.

When sending, HRD assumes that it taking the place of N1MM, and announcing records entered to programs which are in the position of listening to N1MM -- but actually take their data from Logbook. HRD sends these announcements in Section 5 format.

Assuming there aren't bugs the formatting or content of the messages (but there are; Chris outlines some above), HRD would work as a sender if it took the place of N1MM and sent new QSOs to other programs. And it would work as a receiver if it was listening to entries being logged by N1MM itself.

We could, then, support four different modes in HRD:


  1. Send Section 9 datagrams. This would let Logbook announce a new QSO to N1MM, or any other program that listened and simulated itself as N1MM.

  2. Receive Section 9 datagrams. In this mode, Logbook listens for datagrams in the Section 9 that come from other programs. WSJT-X is one suche example.

  3. Send Section 5 datagrams. This would let Logbook take the place of N1MM and announce a new QSO to a program that was compatible with N1MM.

  4. Receive Section 5 datagrams. This would let Logbook receive datagrams from N1MM itself, and mirror whatever N1MM was recording. It would also work to mirror any other program that announced this format.



The status quo is that the Logbook (again, modulo whatever bugs) is trying to do Mode #3 and Mode #4 from the list above.

We should start with figuring out which of these use cases is valid. #2 certainly is; it would be nice to have WSJT-X log straight into the Logbook. Maybe #1 is useful, too, but I'm not too familiar with the landscape of applications and usage patterns in this area.

WA9PIE

2018-08-25 23:42

administrator   ~0006008

Someone smarter than me will have to chime in here.

vk2byi

2018-08-26 01:33

viewer   ~0006009

I would never dare to suggest that I am smarter, but I will chime in just the same. :)

By way of disclosure I point out that I currently rely on use case #3 with my own mapping application (non-commercial, and still in development), so I would be disappointed if that use case was retired.

I routinely test the QSO Forwarding features (both use cases #3 and #4) with each round of HRD beta testing. I enable both the Send and Receive ports in Logbook in my testing environment. My test case utility sends a Contact Info packet to the Logbook receive port (12060) and then waits for up to one second for Logbook to forward the details of that contact on the send port (500). The details of the packet sent to Logbook, and the subsequent packet received from Logbook, are documented in the test results. This is how I discovered the two defects mentioned previously.

I believe these QSO Forwarding features are quite useful and deserve to have the defects fixed. I suspect that there may be other current users of use case #3, and to a lesser extent use case #4.

I completely agree that direct logging from WSJT-X to Logbook (use case #2) is desirable, especially if the details can be enriched with Callsign Lookup(s) in a similar way to a contact added manually through the ALE window.

I don’t have an opinion on use case #1 other than I could use it with my mapping application if use case #3 was removed.

Therefore, I recommend use cases #2, #3 and #4.

Some time ago I wrote a document on the Logging API and QSO Forwarding features of Logbook and I am only too happy to update it as required and attach it here as a reference if that helps.

I am also happy to continue helping any way I can with this. Feel free to contact me directly as well.

73 Chris

K7ZCZ

2018-08-26 12:29

manager   ~0006010

I've refactored the QSO Forwarded code substantially; not quite a re-write, but not much left of the original. I've added mode #2 as above, and I can now send QSOs from WSJT-X into the Logbook. I've made a few bug fixes, and I can log something in N1MM and have it appear in the Logbook, too.

The new QSO Forwarding configuration page is attached.

There's a lot of "revealed" functionality that was previously blocked by bugs here, so we have to do some testing to find out what needs to be fixed. The related issues already identify some of the more serious issues; of particular concern is the scoping of the feature across multiple logbook databases.

Here's a shelf with the code so far:
https://hrdsoftware.visualstudio.com/HRD/_versionControl/shelveset?ss=QSO%20Forwarding%20fixes%3Bmikeblas%40msn.com

NewForwardingOptions.png (22,935 bytes)
NewForwardingOptions.png (22,935 bytes)

vk2byi

2018-08-26 20:58

viewer   ~0006011

I like what I see on the new QSO Forwarding configuration page Mike B.

Can I assume a contact received and logged using either mode #2 (WSJT-X) or mode #4 (TR4W, N1MM) will trigger a mode #3 notification by Logbook on the send side if enabled? I can modify my test case runner application to send mode #2 datagrams to Logbook in addition to the existing mode #4 datagrams, and record a test pass if the mode #3 notification sent by Logbook contains the corresponding contact details.

K7ZCZ

2018-08-28 11:16

manager   ~0006014

I don't know what happens now; I don't know if receiving an external QSO from UDP also causes a send of a UDP message with that added QSO. I can see what's up in the code and figure it out.

But: should it do so? I don't think we've got any kind of use cases documented, and we don't have any specifications for the way things are meant to behave -- this feature is no exception. Should all modes send the other modes? Should that be an option to do so or not?

vk2byi

2018-08-28 19:54

viewer   ~0006016

The current release (.876) works that way - any contacts 'added' via the Logbook API (TCP:7826), DM-780 (TCP:7825), QSO Forwarding Receive port or the ALE window, all result in a datagram being sent on the QSO Forwarding Send port if enabled. The checkbox label that enables the QSO Forwarding Send port even suggests that behaviour: "Forward logbook changes using UDP to other logging programs". However, it is only 'inserts' that trigger the datagram, not 'updates' or 'deletes' of an existing Logbook entry.

I would encourage that same behaviour to continue - any contact inserted via the Logbook API, DM-780, QSO Forwarding Receive port, the ALE window and now with WSJT-X datagrams received on UDP:2333, should all trigger the QSO Forwarding Send broadcast if enabled to keep the listener up-to-date with all Logbook inserts.

I guess forwarding of updates and deletes is a whole new subject and as a third-party developer with a vested interest, I have made suggestions in the past on how that could be implemented. But seeing as this issue is about getting the WSJT-X datagram broadcast working with Logbook, maybe that discussion can be left for a later time if there is any interest.

K7ZCZ

2018-08-29 16:02

manager   ~0006017

The gist of this issue (QSO Lookup not working at all, plus an implementation of a listener for the Section 9 protocol) is implemented with this checkin.
https://hrdsoftware.visualstudio.com/HRD/_versionControl/changeset/4315

Other issues identified here should be given their own individual bugs for the ease of prioritization, work tracking, and testing.

g3ucq

2018-08-30 08:18

viewer   ~0006026

A CW contact made in N1MM+ is added to the HRD Logbook.
Fixed.

vk2byi

2018-08-30 20:09

viewer   ~0006029

Although enabling the new QSO Forwarding feature to receive notifications using the Section 9 protocol results in a contact being logged, there are a number of test case failures with wrong or missing values in the inserted row. I have documented my test results in the attachment.

QSO Forwarding WSJT-X .pdf (164,853 bytes)

k2dls

2018-08-31 13:22

viewer   ~0006034

QSO sent from WSJT-X to HRD Lobgook does not appear in Logbook. See discussion at:

https://forums.hamradiodeluxe.com/node/47353

K7ZCZ

2018-09-01 14:40

manager   ~0006040

This checkin does some refactoring and makes the database manager UI compatible with the QSO Forwarder implementation changes.
https://hrdsoftware.visualstudio.com/HRD/_versionControl/changeset/4317

K7ZCZ

2018-09-01 14:45

manager   ~0006042

Thanks for testing, Chris. Please create a separate Mantis issue for the problems you find. Doing so makes tracking, communicating, documenting, testing issuses much easier than trying to work through a single Mantis issue that contains a composite list of different bugs.

k2dls

2018-09-01 15:16

viewer   ~0006043

After correcting config in HRD Logbook per advice from Chris, VK2BYI, , I retested, attempting to log a QSO from WSJT-X to HRD without involvement of JT Alert. The action caused a minidump. Please advise on where you want the minidump uploaded.

WA9PIE

2018-09-02 23:54

administrator   ~0006064

In both the 877 and 878 builds... and I'm using an odd configuration (I'll get to that in a moment)... not using JTAlert...

When WSJT-X prompts for saving the QSO, I select "Ok" and it sends it to Logbook... without the Frequency, Band, and Mode populated.

I'm using RemoteHamRadio.com... with their "RHR Helper" app... which control the radio from Rig Control as-if it were connected to a physical radio. Using Ham Radio Deluxe as rig control in WSJT-X... with VOX as PTT... everything works fine... except no Frequency, Band, or Mode populated into the QSO.

Anyone else seeing this? (comment in beta forum)

WA9PIE

2018-09-05 01:11

administrator   ~0006076

I tested this tonight with the 881 beta build and it seemed to work correctly. It logs frequency, band, and mode.

There was mention of issues with SRX and STX. However, I think these issues are - at least in-part - related to an overall issue with SRX and STX that we need to (a) better understand and (b) fix separately.

  K7ZCZ added a selection for target database... that's good. If VK2BYI believes that all the fields are being populated, I'd say the UDP API is fixed with the 881 build (with caveat - see next sentence here).

HOWEVER... when updating the info in the QSO Forwarding dialog box, the "OK" button is grey'd out. I'm unable to change and save an option in that dialog box. We'll need to fix that.

CaptureQSOForwarding.PNG (36,203 bytes)
CaptureQSOForwarding.PNG (36,203 bytes)

vk2byi

2018-09-05 02:14

viewer   ~0006077

Mike C: Once you make selections in both of the Target Database controls, the grayed out button enables. This is the correct UI behavior in my humble opinion - force user to specify the target database(s) explicitly.

I think the N1MM QSO Forwarding Send (Mantis 2872) and still has some way to go. But the QSO Forwarding Receive (WSJT-X) side is looking good.

Mike B: Target database is excellent!

vk2byi

2018-09-05 02:16

viewer   ~0006078

CORRECTION TO PREVIOUS POST (cut n paste error):

Mike C: Once you make selections in both of the Target Database controls, the grayed out button enables. This is the correct UI behavior in my humble opinion - force user to specify the target database(s) explicitly.

I think the QSO Forwarding Send - N1MM (Mantis 2872) and Receive - N1MM (this ticket) still have some way to go. Bad dates and SRX/STX to be better understood.

But the QSO Forwarding Receive - WSJT-X (Mantis 2871) is looking real good (even if SRX/STX is fixed separately later)

Mike B: Target database is excellent!

WA9PIE

2018-09-05 12:05

administrator   ~0006081

Ah, okay. Got it! And I agree.

If we're tracking the other issues in 2871/2872, and if we change the title of this issue to make it specific to WSJT-X... then (Chris), do you agree that this is resolved?

vk2byi

2018-09-05 19:04

viewer   ~0006088

Yes, from my perspective it is good. I don't think it will matter that much if SRX/STX is left to be resolved later.

However, I did take a look at a CQ-WW-RTTY contest contact of mine where exchanges consist of CQ Zone and State abbreviation for US stations only, to see what DM-780 logged:
    Call (col_call): 'NA60'
    Rcvd Exch (col_srx): 3
    Sent Exch (col_stx): 30
    Srx # (col_srx_string): '3 CA' <--- the exchange I received
    Stx # (col_stx_string): '30' <--- the exchange I sent

You could look at the DM-780 code to confirm, but it looks to me something like:
    If the exchange is alphanumeric string - store the full string in the srx/stx string suffix column, and the numeric portion in the numeric column
else
    if the exchange is a numeric string - store null in the string suffix column, and store the integer value in the numeric column

Maybe that helps to better understand SRX/STX.

K7ZCZ

2018-09-05 19:13

manager   ~0006089

The "OK" button in the QSO Forwarding dialog will be disabled when the configuration in the dialog isn't valid. This way, we'll never save an invalid configuration. Disabling the OK button in a dialog that's got bogus settings is a common way to cue the user.

In your screenshot, the OK button is disabled because the "Receive logbook changes" box is marked, which means we want to listen to port 12060. But there's no target database selected, so we can't possibly successfully listen to port 12060 because, if we did, we'd have no idea where to store the incoming record. The OK button will enable if you choose a target database, or of you un-mark the "receive logbook changes" checkbox.

Is that not happening? Which options are you unable to change?

WA9PIE

2018-09-07 01:12

administrator   ~0006101

Yes. In my response to Chris, I acknowledged that he guided me to the right place on the "Ok" button. Understood... got it now.

All good... closing this one.

WA9PIE

2018-09-10 10:41

administrator   ~0006174

One last note here...

It would be nice if the database options were initially defaulted to "Default database".

K7ZCZ

2018-09-10 14:30

manager   ~0006188

I don't know which database is the "default database". At this point, this issue has become a collection of other smaller issues which makes it quite difficult to manage. Pelase write up the specific functionality you'd like to see and open a new Mantis issue for it, and we can start a conversation at that new issue.

WA9PIE

2018-09-10 16:41

administrator   ~0006191

I'll cede the point for the moment... but the "default database" is whatever one that was set in Database Manager as shown in this image and related to another change made recently to add an option for setting the default database from a right-click menu.

Overall, my point was this...

If the user has never configured this, then what would be the initial selection of database? None? Or "Default Database" as defined in Databases Manager (image below)?

DefaultDatabase.PNG (40,266 bytes)
DefaultDatabase.PNG (40,266 bytes)

Issue History

Date Modified Username Field Change
2018-06-30 10:49 WA9PIE New Issue
2018-06-30 10:49 WA9PIE File Added: WSJT-x dumpfile02.pcap
2018-07-01 23:30 vk2byi File Added: QSO Forwarding Timestamp.pdf
2018-07-01 23:30 vk2byi File Added: QSO Forwarding XML Header.pdf
2018-07-01 23:30 vk2byi Note Added: 0005590
2018-08-24 00:17 WA9PIE Assigned To => K7ZCZ
2018-08-24 00:17 WA9PIE Status new => feedback
2018-08-24 00:17 WA9PIE Note Added: 0006002
2018-08-25 14:24 K7ZCZ Note Added: 0006006
2018-08-25 17:33 K7ZCZ Relationship added related to 0002858
2018-08-25 23:42 WA9PIE Note Added: 0006008
2018-08-25 23:42 WA9PIE Status feedback => assigned
2018-08-26 01:33 vk2byi Note Added: 0006009
2018-08-26 12:29 K7ZCZ File Added: NewForwardingOptions.png
2018-08-26 12:29 K7ZCZ Note Added: 0006010
2018-08-26 20:58 vk2byi Note Added: 0006011
2018-08-28 11:16 K7ZCZ Note Added: 0006014
2018-08-28 19:54 vk2byi Note Added: 0006016
2018-08-29 16:02 K7ZCZ Status assigned => resolved
2018-08-29 16:02 K7ZCZ Resolution open => fixed
2018-08-29 16:02 K7ZCZ Note Added: 0006017
2018-08-29 18:38 K7ZCZ Fixed in Version => 6.4.0.877
2018-08-30 08:18 g3ucq Note Added: 0006026
2018-08-30 20:09 vk2byi File Added: QSO Forwarding WSJT-X .pdf
2018-08-30 20:09 vk2byi Note Added: 0006029
2018-08-31 13:22 k2dls Note Added: 0006034
2018-09-01 14:35 K7ZCZ Relationship added related to 0002867
2018-09-01 14:40 K7ZCZ Note Added: 0006040
2018-09-01 14:45 K7ZCZ Note Added: 0006042
2018-09-01 15:16 k2dls Note Added: 0006043
2018-09-02 23:54 WA9PIE Note Added: 0006064
2018-09-05 01:11 WA9PIE File Added: CaptureQSOForwarding.PNG
2018-09-05 01:11 WA9PIE Note Added: 0006076
2018-09-05 02:14 vk2byi Note Added: 0006077
2018-09-05 02:16 vk2byi Note Added: 0006078
2018-09-05 12:05 WA9PIE Note Added: 0006081
2018-09-05 12:06 WA9PIE Fixed in Version 6.4.0.877 => 6.4.0.881
2018-09-05 12:06 WA9PIE Summary QSO Forwarding (UDP broadcast) API in Logbook doesn't work => QSO Forwarding (UDP broadcast) API in Logbook doesn't work (for WSJT-X).
2018-09-05 12:06 WA9PIE Testing Not Started => Beta Successful
2018-09-05 19:04 vk2byi Note Added: 0006088
2018-09-05 19:13 K7ZCZ Note Added: 0006089
2018-09-07 01:12 WA9PIE Status resolved => closed
2018-09-07 01:12 WA9PIE Note Added: 0006101
2018-09-10 10:41 WA9PIE Note Added: 0006174
2018-09-10 14:30 K7ZCZ Note Added: 0006188
2018-09-10 16:41 WA9PIE File Added: DefaultDatabase.PNG
2018-09-10 16:41 WA9PIE Note Added: 0006191
2018-09-11 13:15 WA9PIE Project 3 - Current Dev List => Ham Radio Deluxe
2018-09-11 13:18 WA9PIE Fixed in Version 6.4.0.881 => 6.4.0.886