View Issue Details

IDProjectCategoryView StatusLast Update
0003495Ham Radio DeluxeBugpublic2019-11-08 02:32
ReporterWA9PIEAssigned ToK7ZCZ 
PrioritynormalSeveritymajorReproducibilityalways
Status closedResolutionfixed 
Product Version 
Target VersionFixed in Version6.7.0.244 
Summary0003495: Complete the Callsign Lookup function for the Callook.info method
DescriptionSome fields or data are missing from the Callook.info callsign lookup method.

Once the Callook.info data is obtained for a given callsign, the following ALE fields should be populated from the data found as follows:

- COL_QSO_STATE should be parsed from the data in the "line2" tag in the response; everything between the first ", " (comma space, without quotes) and the following " " (space)
- COL_QSO_QTH should be parsed from the data in the "line2" tag in the response; everything before the first ", " (comma space, without quotes)

None of these fields are being populated.

This Mantis issue is to fix only those fields that were not being populated. All fields that were previously populated correctly by this method should continue to be populated.
Steps To ReproduceI repro this in the 6.7.0.232 beta build (6.7.0.234 can't be tested because it crashes 0003484).

- Open up Logbook
- Setup Callsign Lookup by Tools > Configure > Callsign Lookup and make sure that Callook.info is the only Enabled method listed above Country List in Enabled Methods
- Click "Ok"
- Open an ALE
- Enter the callsign "WA9PIE" (without the quotes)
- Press the "Lookup" button or hit tab

Result: These fields are populated in the respective fields in the ALE
Additional InformationThe correct result for these two fields should be:

- COL_QSO_STATE should be "TX" (without quotes)
- COL_QSO_QTH should be "Prosper" (without quotes)
TagsNo tags attached.
ModuleLogbook
Sub-ModuleCall lookup
Testing Beta Successful

Relationships

related to 0003464 closedWA9PIE Some fields missing from QRZ.com call sign lookup 
related to 0003546 closedK7ZCZ Callook.info, the Callsign Lookup does not correctly assign DXCC, Country, Continent, and HRDCountry 
child of 0003001 closedWA9PIE Callsign lookup function does not appear to be working as designed 

Activities

WA9PIE

2019-10-13 07:25

administrator  

Callook info (WA9PIE).xml (855 bytes)

K7ZCZ

2019-10-17 09:10

administrator   ~0008839

This checkin gets the lookup of the country and continent with the HRD country list working:
https://hrdsoftware.visualstudio.com/HRD/_versionControl/changeset/5215

The repro steps say that the QTH should be "Prosper", but the CalLook source doesn't provide a QTH field. Looks like the spreadsheet part of the spec says "parse city from address/line2", but no advice is given on exactly how this parsing occurrs. What grammar should be used? What if the parsing fails? Seems to counter to the "no guessing" design goal. The CalLook reference for their data doesn't seem to indicate that Address2 contains anything specific, and indicates no specific format -- just a string. Cam this please be clarified?

WA9PIE

2019-10-17 09:18

administrator   ~0008841

In the example file, we find the following:

<line2>PROSPER, TX 75078</line2>

I suppose... I'd parse the QTH by taking everything from the left up to (but not including) the comma.

Callook.info is a US only database. So it follows that the State is between ", " (comma space) and the next space.

If I were a developer, I'd do that.

K7ZCZ

2019-10-17 09:28

administrator   ~0008842

It's not about being a developer or not -- nobody's has asked you to wear that hat.

What I'm asking is for an indication of the functionality you want. That way, I don't have to guess and we can finish this up a lot faster and with fewer iterations.

It's not hard to use what's in the field up to the first comma, if that's what you want. But what if the field is empty, or doesn't contain a comma?

Further, since parsing one field to create another is opposed to the "no guessing' goal for this feature, are we sure we want to make an exception in this case? If so, shouldn't that be clearly documented in the specification?

WA9PIE

2019-10-17 09:35

administrator   ~0008843

If the data could reasonably contain "city, state zip"... then I would parse everything up to the first comma as city and pull the state out of the middle.

If <line2> is empty - I'd leave the corresponding fields (QTH and State) in the ALE empty from Callook.info.
If it doesn't contain a comma, I'd be tempted to ignore the whole entire <line2> data and again leave it empty.

This data comes directly from the FCC's database. I would think it's rare that the data would not contain "city, state zip". This is the structure of this data (<line2>). Anything that doesn't fit this structure should be ignored.

K7ZCZ

2019-10-17 10:11

administrator   ~0008844

Rare or not, if it happens, we must write code to that situation.

The documentation for the data source doesn't say what the field contains; it's described as a string and given the label "Address2". If we don't know what the field contains, we don't have any business parsing it to try to get data we want out of it as that data might or might not be there.

I think that, given our "no guessing" tenet, we shouldn't be using this field at all except as the second line of whatever we assemble into COL_QSL_ADDRESS. If you don't agree, then let me know and I'll make it work the way you want.

WA9PIE

2019-10-17 16:31

administrator   ~0008847

The website for the API is described here: https://callook.info/api_reference.php#xml

The format for <line2> is shown there. The format is:

<line2>CITY, ST ZIP<line2>

Let's parse it based on the comma and spaces to populate QTH and STATE.

In order to avoid guessing, if there are cases where <line2> is either empty or does not match this format, ignore the data leave the QTH and STATE fields blank.

K7ZCZ

2019-10-17 17:27

administrator   ~0008848

While the document provides an example with data, it does not describe a specific format or precisely describe any content. If we want to assume that any address is formatted to match that example, we can do so.

But there are further problems. The spreadsheet part of the spec says that the DXCC should be used from this data, but the data source doesn't return a DXCC value. (The documentation page you've linked doesn't mention DXCC at all.) The implication is that the data is all US, but the United States includes its territories and possessions, and places like Alaska and Guam and Pueto Rico have their own DXCC numbers (and HRD Country Numbers), apart from 291 for the rest of the US.

Should we just assume all addresses returned from this service are 291 for DXCC and HRDCountryNumber, then?

WA9PIE

2019-10-17 19:56

administrator   ~0008853

Rather than considering it to be an assumption, we should consider it a requirement that <line2> is formatted as shown on the Callook.info API description page and in the WA9PIE example attached.

Given this as a requirement - when <line2> is in this format, we should parse it and populate QTH and STATE with it. When <line2> is not formatted in this way, or when it is not populated, we should leave QTH and STATE empty.

Further... having reviewed this... I inadvertently changed the spec in error with regards to DXCC, Country, and HRDCountryNumber. With regards to these fields, I would leave them blank as a result of Callook.info and allow them to be populated by the "source of last resort" - the Country List method (as a subsequent part of the overall search flow that follows).

K7ZCZ

2019-10-17 21:50

administrator   ~0008854

Seems like we're moving away from clarity rather than towards it. Please update this issue with the results that you'd like to see; then, I can re-evaluate them and see how we can get there.

K7ZCZ

2019-10-17 21:50

administrator   ~0008855

Seems like we're moving away from clarity rather than towards it. Please update this issue with the results that you'd like to see; then, I can re-evaluate them and see how we can get there.

WA9PIE

2019-10-17 22:03

administrator   ~0008857

The approach taken was to describe what does not work in this lookup method. The reason being... it didn't seem to make sense to clutter this with things that DO work. Questions I anticipated in either event:
1. Why did you put content into this issue for cases that do work? Can't we just concentrate on those things that need fixed? (which was my approach)
2. Do you no longer want fields populated that aren't listed in this Mantis issue? (of course we want the fields that were working to continue working)

In summary and for clarity...
- if the field was working when I tested it, I did not list it here; we should not make any changes that would prevent fields that did get populated from being populated
- if <line2> is formatted with "CITY, ST ZIP", parse it and put CITY in COL_QSO_QTH and ST in COL_QSO_STATE
- if <line2> is not formatted with "CITY, ST ZIP", ignore that data and leave COL_QSO_QTH and COL_QSO_STATE unpopulated by THIS method
- for this method, leave DXCC, Country, and HRDCountryNumber empty (we'll let other enabled methods or Country List populate those fields)

When I test this, I'll test ALL fields - the ones that previously worked and these that weren't. I'll either close/update this one for the items referenced here and/or I'll open a new one for new things that change

K7ZCZ

2019-10-19 10:45

administrator   ~0008867

For the this data source, you've decided the DXCC, Country, Continent, and HRDCountryNumber fields aren't useful. Since we consider some fields from some sources to be not useful, it's thus not obvious that "of course we want the fields that were working to continue working".

These issues are the first time that sample data was used to express the desired behaviour of the product. The exercise of building the partial sets of desired results has identified further issues in the specifications. I think it would've been beneficial to have done this work as part of finishing the specification.

This change set removes the DXCC, Country, Continent, and HRDCountryNumber fields from the data produced by this source. It also parses a QTH out of the "Line2" field given the rules identified above. I don't think there's much more progress I can make with this issue without more information about the desired behaviour.

https://hrdsoftware.visualstudio.com/HRD/_versionControl/changeset/5219

WA9PIE

2019-10-19 18:00

administrator  

Callook info NoDXCC.png (62,732 bytes)
Callook info NoDXCC.png (62,732 bytes)

WA9PIE

2019-10-19 18:00

administrator   ~0008873

Not only have I decided that DXCC, Country, Continent, and HRDCountryNumber fields "aren't useful" for this data source, these fields are in-fact non-existent in this data source. (I inadvertently pasted values in the field mapping spec from one of the other sources and created confusion probably.)

Because data from Callook.info does not have a DXCC field, then the DXCC, Country, Continent, and HRDCountryNumber fields should populate naturally from the Country List.

WA9PIE

2019-10-21 10:16

administrator   ~0008890

Last edited: 2019-10-21 10:17

View 2 revisions

This note reflects testing of the 6.7.0.235 build.

Using WA9PIE as the example with only Callook.info added as an "Enabled Method" (aside from the two UCSDB and Country List):

- COL_ADDRESS field is not being populated. It should populate as follows:

Thus, the COL_ADDRESS would be populated as:
MICHAEL G CARPER, PHD
PO BOX 995
PROSPER, TX 75078
United States

- if <line2> is formatted with "CITY, ST ZIP", parse it and put CITY in COL_QSO_QTH and ST in COL_STATE

- if <line2> is not formatted with "CITY, ST ZIP", ignore that data and leave COL_QSO_QTH and COL_STATE unpopulated by THIS method

K7ZCZ

2019-10-21 15:53

administrator   ~0008899

I just inadvertently removed the QTH and ADDRESS assignments when cleaning up the other code.

Fixed with this checkin:
https://hrdsoftware.visualstudio.com/HRD/_versionControl/changeset/5229

WA9PIE

2019-10-29 05:25

administrator   ~0009062

Validated

Issue History

Date Modified Username Field Change
2019-10-13 07:25 WA9PIE New Issue
2019-10-13 07:25 WA9PIE Status new => assigned
2019-10-13 07:25 WA9PIE Assigned To => K7ZCZ
2019-10-13 07:25 WA9PIE File Added: Callook info (WA9PIE).xml
2019-10-13 07:25 WA9PIE Relationship added child of 0003001
2019-10-13 07:25 WA9PIE Relationship added related to 0003464
2019-10-13 07:29 WA9PIE Description Updated View Revisions
2019-10-13 07:29 WA9PIE Steps to Reproduce Updated View Revisions
2019-10-13 07:29 WA9PIE Additional Information Updated View Revisions
2019-10-13 07:51 WA9PIE Steps to Reproduce Updated View Revisions
2019-10-13 09:04 WA9PIE Description Updated View Revisions
2019-10-17 09:10 K7ZCZ Assigned To K7ZCZ => WA9PIE
2019-10-17 09:10 K7ZCZ Status assigned => feedback
2019-10-17 09:10 K7ZCZ Note Added: 0008839
2019-10-17 09:18 WA9PIE Note Added: 0008841
2019-10-17 09:18 WA9PIE Assigned To WA9PIE => K7ZCZ
2019-10-17 09:28 K7ZCZ Note Added: 0008842
2019-10-17 09:28 K7ZCZ Assigned To K7ZCZ => WA9PIE
2019-10-17 09:35 WA9PIE Note Added: 0008843
2019-10-17 09:35 WA9PIE Assigned To WA9PIE => K7ZCZ
2019-10-17 10:11 K7ZCZ Note Added: 0008844
2019-10-17 10:11 K7ZCZ Assigned To K7ZCZ => WA9PIE
2019-10-17 16:31 WA9PIE Note Added: 0008847
2019-10-17 16:32 WA9PIE Assigned To WA9PIE => K7ZCZ
2019-10-17 17:27 K7ZCZ Assigned To K7ZCZ => WA9PIE
2019-10-17 17:27 K7ZCZ Note Added: 0008848
2019-10-17 19:56 WA9PIE Note Added: 0008853
2019-10-17 19:56 WA9PIE Assigned To WA9PIE => K7ZCZ
2019-10-17 21:50 K7ZCZ Note Added: 0008854
2019-10-17 21:50 K7ZCZ Note Added: 0008855
2019-10-17 21:50 K7ZCZ Assigned To K7ZCZ => WA9PIE
2019-10-17 22:03 WA9PIE Note Added: 0008857
2019-10-17 22:05 WA9PIE Assigned To WA9PIE => K7ZCZ
2019-10-17 22:05 WA9PIE Description Updated View Revisions
2019-10-17 22:05 WA9PIE Additional Information Updated View Revisions
2019-10-19 10:45 K7ZCZ Note Added: 0008867
2019-10-19 10:46 K7ZCZ Status feedback => resolved
2019-10-19 10:46 K7ZCZ Resolution open => fixed
2019-10-19 10:46 K7ZCZ Assigned To K7ZCZ => WA9PIE
2019-10-19 18:00 WA9PIE File Added: Callook info NoDXCC.png
2019-10-19 18:00 WA9PIE Note Added: 0008873
2019-10-21 10:16 WA9PIE Note Added: 0008890
2019-10-21 10:17 WA9PIE Assigned To WA9PIE => K7ZCZ
2019-10-21 10:17 WA9PIE Status resolved => assigned
2019-10-21 10:17 WA9PIE Note Edited: 0008890 View Revisions
2019-10-21 15:53 K7ZCZ Status assigned => resolved
2019-10-21 15:53 K7ZCZ Note Added: 0008899
2019-10-29 05:25 WA9PIE Status resolved => closed
2019-10-29 05:25 WA9PIE Fixed in Version => 6.7.0.239
2019-10-29 05:25 WA9PIE Testing Not Started => Beta Successful
2019-10-29 05:25 WA9PIE Note Added: 0009062
2019-11-04 21:58 K7ZCZ Relationship added related to 0003546
2019-11-08 02:10 WA9PIE Fixed in Version 6.7.0.239 => 6.7.0.244
2019-11-08 02:32 WA9PIE Project 3 - Current Dev List => Ham Radio Deluxe