View Issue Details

IDProjectCategoryView StatusLast Update
00026963 - Current Dev ListMaintenancepublic2019-02-21 16:02
ReporterPD9FER 
Assigned ToWA9PIE 
PrioritynormalSeveritycrashReproducibilityrandom
Status closedResolutionreopened 
Product Version 
Target VersionFixed in Version6.5.0.194 
Summary0002696: Importing Logfile 155,000 + QSO's crashes LB
DescriptionImport log of 155000 QSO's. It starts to save then crashes in the the middle or beginning giving me a minidump error.

Found out when disconnecting from the DX Cluster the problem has gone
Steps To ReproduceOpen Logbook
Have a large Adif log ready (+ 155000 QSO's)
Have DX Cluster connected
Do a database import (ADIF)
It randomly crashes in the beginning or somewhere along the saving process.

Retry above steps with the Cluster disconnected
Additional InformationTicket #590779
Minidump added to the Google Drive Dump folder
TagsNo tags attached.
ModuleLogbook
Sub-ModuleImport Export
Testing Beta Successful

Relationships

related to 0002222 assignedK7ZCZ 3 - Current Dev List Logbook: restoring large XML backup causes out of memory crash 
related to 0002219 new 2 - Next Dev List (Holding Area) Logbook: Loading an XML backup is very slow due to the use of a DOM parser 

Activities

K7ZCZ

2018-04-30 18:18

manager   ~0004911

I think you forgot to attach the ADIF file and the minidump. Assigned to you for those files; you can assign this issue back to me when you've provided them.

PD9FER

2018-05-01 04:26

updater   ~0004914

see Dump folder (probably did not Sync)
Mantis_ID-0002696.rar

K7ZCZ

2018-05-01 08:25

manager  

Mantis_ID-0002696.rar (9,816,074 bytes)

K7ZCZ

2018-05-01 08:25

manager   ~0004916

Attaching the file here directly; I've removed the file Google Drive.

K7ZCZ

2018-05-01 08:26

manager   ~0004917

The call stack shows that the Logbook application is running out of memory.

0:075> .ecxr
eax=00000000 ebx=2b342208 ecx=00450000 edx=2b342208 esi=70c507ea edi=2b342200
eip=7778e486 esp=1933f8b0 ebp=1933f8e4 iopl=0         nv up ei pl nz na po nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010202
ntdll!RtlpLowFragHeapFree+0x31:
7778e486 8b4604          mov     eax,dword ptr [esi+4] ds:002b:70c507ee=????????
0:075> kb
  *** Stack trace for last set context - .thread/.cxr resets it
 # ChildEBP RetAddr  Args to Child              
00 1933f8e4 7778e0e3 1933fb88 00000000 1933fb88 ntdll!RtlpLowFragHeapFree+0x31
01 1933f8fc 75286e2a 00450000 00000000 2b342208 ntdll!RtlFreeHeap+0x105
02 1933f910 75286f14 753776bc 2b342208 1933f930 ole32!CRetailMalloc_Free+0x1c [d:\w7rtm\com\ole32\com\class\memapi.cxx @ 687] 
03 1933f920 5cf359a7 2b342208 1933fb70 1933fb58 ole32!CoTaskMemFree+0x13 [d:\w7rtm\com\ole32\com\class\memapi.cxx @ 475] 
04 1933f930 5cf398f0 1933f94c 75286f01 00000000 fundisc!CCoMemString::Assign+0x15
05 1933fb58 5cf39540 00000000 001ad0bc 001ad068 fundisc!CCategoryManagerEnum::AdvanceSubcategory+0x75
06 1933fba4 5cf43b69 2d0e5ee8 00000000 00000001 fundisc!CCategoryManager::LoadCategoryMetadata+0xf8
07 1933fbd4 5cf375aa 00000001 001ad068 8007000e fundisc!CQueryWorker::InitializeCategory+0x53
08 1933fbec 5cf31ce9 5ffe1590 00000000 00000001 fundisc!CQueryWorker::Initialize+0x1bd
09 1933fc1c 5cf31c39 5ffe1590 00000000 00000001 fundisc!CFunctionDiscoveryWorker::CreateInstanceCollectionQuery+0x73
0a 1933fc4c 5ffe2f84 001ad2c0 5ffe1590 00000000 fundisc!CFunctionDiscovery::CreateInstanceCollectionQuery+0xfb
0b 1933fc7c 5ffe30a5 135174f8 001ad2c0 5ffe1590 NetworkItemFactory!CNetworkItemFactory::_StartFDQuery+0x29
0c 1933fcac 5ffe3144 135174f8 1933fd40 757943c0 NetworkItemFactory!CNetworkItemFactory::s_StartFDInMTA+0x73
0d 1933fcb8 757943c0 135174f8 00000000 00000000 NetworkItemFactory!FDBackgroundThreadHandler+0x1b
0e 1933fd40 76a0343d 0a40f8b8 1933fd8c 77799832 shlwapi!WrapperThreadProc+0x1b5
0f 1933fd4c 77799832 0a40f8b8 61ab48ad 00000000 kernel32!BaseThreadInitThunk+0xe
10 1933fd8c 77799805 757942ed 0a40f8b8 00000000 ntdll!__RtlUserThreadStart+0x70
11 1933fda4 00000000 757942ed 0a40f8b8 00000000 ntdll!_RtlUserThreadStart+0x1b

WA9PIE

2018-06-14 00:57

administrator   ~0005316

Just a guess... but it seems likely that the DX cluster WSIs are trying to be updated during the import. It might be a good approach to suspend updating the WSIs until after the import is complete.

K7ZCZ

2018-06-14 19:02

manager   ~0005327

I think the provided call stack makes it pretty clear that the problem is that the logbook runs out of memory.

Memory pressure is caused by the inefficiency of the logbooks internal record data structure.

Further memory pressure is due to the design decision of using a DOM-based XML parser for reading and writing XML data. The parser is obliged to read the whole XML document and build the DOM object structure for it. The DOM structure isn't accessed randomly by the logbook's backup and restore code, so the DOM is completely unnecessary. The logbook would be better served by a streaming XML parser, which would be faster for run-time performance and also use less memory.

PD9FER

2018-06-28 07:34

updater   ~0005512

Do you need any more information?

K7ZCZ

2018-06-29 11:16

manager   ~0005542

I don't think so, but a copy of the backup would be nice. It's about 30% larger than the largest I have access to at the moment.

PD9FER

2018-06-29 11:43

updater   ~0005543

Am I missing something?
That Adif is included in the Dump archive

K7ZCZ

2018-06-29 12:05

manager   ~0005544

Got it. Thanks!

WA9PIE

2018-12-06 16:09

administrator   ~0006543

Retest. If this repeats, give us the log.

WA9PIE

2019-02-10 22:17

administrator   ~0007351

I did this myself at Hamcation with a log of 181k QSOs. It did run the machine out of memory (16GB RAM) and it crashed Logbook.

We'll need to solve for this.

K7ZCZ

2019-02-11 20:37

manager   ~0007355

It would be helpful to share the large backup file that you used.

The notes in 5327 identify the problem, so all we have to do is decide that this has a higher priority that the other things that I've been told have a higher priority.

I've done some research on parsers; looks like LibXML2 is a good, open source solution. It would also be helpful to review the license: http://xmlsoft.org/FAQ.html

If it's acceptable, then:

1) Get LibXML2 sources and make them part of the build.
2) Re-write the XML import code to use LibXML2 to get data.
3) Test.

I figure the biggest risk is testing ... we're just not good at it. The potholes are around character encoding support, I think. LibXML2 does what we need, but handles UTF-8 in an odd way in an attempt to make portability of the library good and easy.

WA9PIE

2019-02-13 12:33

administrator   ~0007374

Last edited: 2019-02-13 13:00

View 2 revisions

Yep... it was my plan to get the user's file. I had to get out of the Hamcation show and pull it from one of the machines.

I'm attaching it here.

Basically, if you run the ADIF import and use this file to pull it into a new or existing log, it will eventually run the machine out of memory and crash Logbook.

I'm not sure how far back this goes, but we do have users with larger logs than this one and there was a time when we did not see this happen.

NOTE: I placed the user's log in ADIF format here: "\Team Drives\HRD Software\Logs (customers)"

K7ZCZ

2019-02-13 13:45

manager   ~0007379

Your help is still needed in reviewing the LibXML2 license.

It would also be helpful to understand how the size of the log is being measured in the comparison you make.

WA9PIE

2019-02-13 15:02

administrator   ~0007382

Where is the information I need to review regarding the LibXML2 license?

The comparisons we make are related to the number of QSOs in the file.

K7ZCZ

2019-02-15 10:46

manager   ~0007399

Note number 0007355 in this issue includes a link to the licensing information for an open-source library that I think we can use to relieve some memory pressure during XML imports. Using that library would help address Mantis 2219 and 2222, which are related to this issue. We need to know if the terms of the license are acceptable, and if they require any changes to the documentation or our own EULA.

I don't think number of records is a good metric for evaluating expecting memory use of the import feature, or the Logbook in general. That's because practically all of the fields in the logbook are optional, and almost all of them are variable-width. A Logbook record, then doesn't take a fixed amount of memory. Instead, a record might take only a few dozen bytes to represent, or might take a couple of kilobytes to store.

Because of the variable length, it's alway possible that a file with fewer records by count requires more memory than a file with more records.

Note that the provided WB2EM backup file is an ADI file, not an XML file, and won't be impacted by switching to a streaming XML parser. I haven't been able to repro the problem with an ADI file, so I don't know what the memory budget is. It seems likely that the problem involves the Logbook's inefficient internal data structures, an issue documented at Mantis 2197.

WA9PIE

2019-02-15 14:42

administrator   ~0007403

It’s hard to conceive of another metric, given that we are unable to import it. That said - I understand and agree with your point.

If we need a library, great.

Regarding the ADI file, this is related to reproducing the problem. I know of no relationship to this in XML format (which is related to backup and restore, not import of ADIF. This problem is specific to the import of an ADIF file).

K7ZCZ

2019-02-16 12:56

manager   ~0007409

A more viable metric is the size of the file, on bytes, on disk. Because the file is loaded into memory entirely (and not processed line-by-line), the amount of memory used will be very closely related to the file size on disk.

As Ferry indicates in his description and notes in this issue, the out-of-memory issue isn't specific to ADIF file imports. Please see Mantis 2219 and 2222 to see specific XML-related scenarios.

What would help me proceed is your review of the license terms for the library I'd like to use. Taking it as a dependency means committing to those licensing terms. I need to know if they are acceptable to the business.

K7ZCZ

2019-02-17 14:28

manager   ~0007419

The ADIF import dialog builds a raw copy of the file in memory, then builds a parsed copy of the ADIF records, then adds those to the database, then reloads the database. Only after the database is re-loaded is any memory freed.

The headroom for large databases can be increased by freeing intermediate representations of the imported data before reloading the database. The capacity of the logbook is still not infinite (and is artificially low due to the design of the data structures used; see Mantis ), but this checkin improves matters:

https://hrdsoftware.visualstudio.com/HRD/_versionControl/changeset/4832

While the situation for ADIF imports is improved, this issue is about both ADIF and XML imports. The XML side of the house remains dependent on finding a better parsing library. Such a library has been identified, and we need to review the license terms of the code before incorporating it into the product.

WA9PIE

2019-02-17 19:15

administrator   ~0007422

I have read the licensing information and I approve the use of this library under the terms of the published license information provided here: http://xmlsoft.org/FAQ.html

K7ZCZ

2019-02-19 19:11

manager   ~0007434

Great, thanks. I can work on integrating the LibXML2 code into our product.

K7ZCZ

2019-02-20 11:41

manager   ~0007457

This issue is hard to manage because it is discussing both the XML restore feature and the ADIF import feature. Leaving the issue as-is would mean that we couldn't mark the issue resolved until both the XML restore and the ADIF import feature were fixed. Instead, I'd rather use this issue to track the ADIF issue and use the related issues (Mantis 2219 and 2222) to track the XML issues.

Since the ADIF issue is fixed, I'm resolving this issue. Work related to the XML restore OOM can be tracked at 2219 and 2222.

WA9PIE

2019-02-20 23:38

administrator   ~0007464

The mention of the XML is unfortunate. I have not had any repro on that. I do know that the ADIF does repro using the ADIF provided by the customer.

WA9PIE

2019-02-21 09:44

administrator   ~0007476

Last edited: 2019-02-21 09:52

View 2 revisions

I got feedback on the 194 build from the user who reported this problem that the program is still crashing on ADIF import.

Given that there is another Mantis issue open for the XML thing, I removed the topic related to XML here. In this issue, we'll focus on the ADIF import.

I'm posting the customer's new minidumps in the following folder: "Team Drives\HRD Software\Dumps\Mantis 2696"

K7ZCZ

2019-02-21 12:36

manager   ~0007478

I don't think there's any rason to edit the content of an issue. In my experience, it's quite enough to just make note that the issue is meant to only track a part of the original report, and indicate that the balance of the problem is tracked elswhere.

I'll have a look at the new minidumps. What were the result when you tested it yourself?

K7ZCZ

2019-02-21 13:43

manager   ~0007480

These dump files from Google drive are HRDLogbook_20190221_104427.mdmp and HRDLogbook_20190221_125256.mdmp. They are from build 195, not build 194 as indicated above.

0:010> lmDvmHRDLogbook
Browse full module list
start    end        module name
011d0000 02a11000   HRDLogbook   (deferred)             
    Image path: C:\Program Files (x86)\HRD Software LLC\Ham Radio Deluxe\HRDLogbook.exe
    Image name: HRDLogbook.exe
    Browse all global symbols  functions  data
    Timestamp:        Wed Feb 20 17:53:48 2019 (5C6E04AC)
    CheckSum:         01613AE8
    ImageSize:        01841000
    File version:     6.5.0.195
    Product version:  6.5.0.195
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        1.0 App
    File date:        00000000.00000000
    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4


With the right symbols and binaries set up, we see this call stack in both dumps:

0:009> kp
  *** Stack trace for last set context - .thread/.cxr resets it
 # ChildEBP RetAddr  
00 0deff1c0 5e52f396 msxml6!Model::Model(struct TLSDATA * ptlsdata = 0x3078c930, class Object * pObject = 0x00000000)+0xb [d:\w7rtm\sql\xml\msxml6\core\base\tls.cxx @ 355] 
01 0deff1d8 5e5a16b1 msxml6!OMWriteLock::OMWriteLock(struct TLSDATA * ptlsdata = 0x3078c930, class DOMNode * pNode = 0x07c1b370)+0x1e [d:\w7rtm\sql\xml\msxml6\xml\om\omlock.cxx @ 276] 
02 0deff238 5e5a1a9e msxml6!DOMNode::insertBefore(struct IXMLDOMNode * pNewChild = 0x00c95e08, struct tagVARIANT refChild = NULL, struct IXMLDOMNode ** ppNewChild = 0x0deff2cc)+0x39 [d:\w7rtm\sql\xml\msxml6\xml\om\domnode.cxx @ 1377] 
03 0deff274 5e5a235b msxml6!DOMNode::appendChild(struct IXMLDOMNode * newChild = 0x00c95e08, struct IXMLDOMNode ** ppNewChild = 0x0deff2cc)+0x2c [d:\w7rtm\sql\xml\msxml6\xml\om\domnode.cxx @ 1430] 
04 0deff288 013fd1b1 msxml6!W3CDOMWrapper::appendChild(struct IXMLDOMNode * newChild = 0x00c95e08, struct IXMLDOMNode ** ppNewChild = 0x0deff2cc)+0x17 [d:\w7rtm\sql\xml\msxml6\xml\om\w3cdom.cxx @ 232] 
05 (Inline) -------- HRDLogbook!CXMLMgr::AppendChildToParent+0x12 [c:\hrd65\hrdcommon\xmlmgr.cpp @ 1343] 
06 0deff2f8 013f38e7 HRDLogbook!CXMLMgr::CreateSubNode(struct IXMLDOMElement * pParentNode = 0x07c27dd0, wchar_t * lpcstrNodeName = 0x01afbd6c "Record", wchar_t * lpcstrNodeText = 0x01a78f80 "", class CStringArray * astrItemNames = 0x0deff54c, class CStringArray * astrItemValues = 0x0deff560)+0xa1 [c:\hrd65\hrdcommon\xmlmgr.cpp @ 1280] 
07 (Inline) -------- HRDLogbook!CXMLMgr::CreateSubNode+0x51 [c:\hrd65\hrdcommon\xmlmgr.cpp @ 1222] 
08 0deff5c0 013f2973 HRDLogbook!CXMLSaveToFile::Save(void)+0xe07 [c:\hrd65\logbook\hrdlogbook\xmlsavetofile.cpp @ 640] 
09 0deff5e8 011cdf09 HRDLogbook!CXMLSaveToFile::CXMLSaveToFile(class CWnd * pParent = 0x00000000, struct HWND__ * hWndFrame = 0x00040928, class ATL::CStringT<wchar_t,StrTraitMFC<wchar_t,ATL::ChTraitsCRT<wchar_t> > > * strFilename = 0x0deff718 "C:\Users\GW-DX4860\Dropbox\WB2REM-HRDLOG\Test backup 2019-02-21 0749.xml", class CWnd * pCounter = 0x00000000, class CProgressCtrl * pProgress = 0x00000000, class ATL::CStringT<wchar_t,StrTraitMFC<wchar_t,ATL::ChTraitsCRT<wchar_t> > > * strDatabase = 0x0deff700 "Test", class ATL::CStringT<wchar_t,StrTraitMFC<wchar_t,ATL::ChTraitsCRT<wchar_t> > > * strDSN = 0x0deff71c "Test", class ATL::CStringT<wchar_t,StrTraitMFC<wchar_t,ATL::ChTraitsCRT<wchar_t> > > * strConnectString = 0x0deff6f8 "DSN=MS Access Database;DBQ=C:\Users\GW-DX4860\AppData\Roaming\HRDLLC\HRD Logbook\Test.mdb;DriverId=25;FIL=MS Access", eDBMSProductNames eDatabase = DBMS_ACCESS (0n1), int bXML = 0n304604656)+0x283 [c:\hrd65\logbook\hrdlogbook\xmlsavetofile.cpp @ 423] 
0a 0deff868 011cb8b1 HRDLogbook!CBackgroundProcessingThread::XMLExport(struct HWND__ * hWnd = 0x00040928, class ATL::CStringT<wchar_t,StrTraitMFC<wchar_t,ATL::ChTraitsCRT<wchar_t> > > * strXML = 0x1eaab208)+0x9d9 [c:\hrd65\logbook\hrdlogbook\backgroundprocessingthread.cpp @ 1028] 
0b 0deff8c4 011cb7b8 HRDLogbook!CBackgroundProcessingThread::ProcessData(class CBkgPacket * pPkt = 0x1eaab1e0, int nCount = 0n0)+0x81 [c:\hrd65\logbook\hrdlogbook\backgroundprocessingthread.cpp @ 429] 
0c 0deff8fc 013d8b56 HRDLogbook!CBackgroundProcessingThread::DoWork(void)+0xa8 [c:\hrd65\logbook\hrdlogbook\backgroundprocessingthread.cpp @ 225] 
0d (Inline) -------- HRDLogbook!CThinThread::Run+0x55 [c:\hrd65\logbook\hrdlogbook\thinthread.cpp @ 189] 
0e 0deff93c 01912640 HRDLogbook!CThinThread::Start(void * pv = 0x01bf90a8)+0x96 [c:\hrd65\logbook\hrdlogbook\thinthread.cpp @ 228] 
0f (Inline) -------- HRDLogbook!invoke_thread_procedure+0xd [d:\th\minkernel\crts\ucrt\src\appcrt\startup\thread.cpp @ 91] 
10 0deff978 769f343d HRDLogbook!thread_start<unsigned int (void * parameter = 0x07a11d40)+0x57 [d:\th\minkernel\crts\ucrt\src\appcrt\startup\thread.cpp @ 115] 
11 0deff984 76f29802 kernel32!BaseThreadInitThunk+0xe
12 0deff9c4 76f297d5 ntdll!__RtlUserThreadStart+0x70
13 0deff9dc 00000000 ntdll!_RtlUserThreadStart+0x1b



The call stack indicates a hard crash while performing an XML Backup, so this doesn't seem related to the problem here, or the repro steps that are provided. Is correct information avialble from this report? If so, we should open another issue for those steps and leave this issue specifically for the OOM issue surrounding ADIF restores.

PD9FER

2019-02-21 15:30

updater   ~0007483

Check in Logbook, Tools > Configure > Backup
Test with the option Backup on every x changes

WA9PIE

2019-02-21 15:46

administrator   ~0007484

Let's test this with the backups turned off. I'll gather more information about what the customer was doing and how his backups were configured. I'm testing the import with the backups turned off. I can then try it with the same backup settings the customer has and we can verify whether these contributed to the crash.

WA9PIE

2019-02-21 16:02

administrator   ~0007485

Validated (this was tested by importing the WB2REM log... with backup options disabled; the file imported in a reasonable amount of time without fail)

We DO need to now gather more information about the backup process and treat it as a separate issue.

Issue History

Date Modified Username Field Change
2018-04-27 02:43 PD9FER New Issue
2018-04-30 18:17 K7ZCZ Project 1 - Backlog => 2 - Next Dev List (Holding Area)
2018-04-30 18:18 K7ZCZ Assigned To => PD9FER
2018-04-30 18:18 K7ZCZ Status new => feedback
2018-04-30 18:18 K7ZCZ Note Added: 0004911
2018-04-30 18:19 K7ZCZ Relationship added related to 0002222
2018-05-01 04:26 PD9FER Note Added: 0004914
2018-05-01 08:25 K7ZCZ File Added: Mantis_ID-0002696.rar
2018-05-01 08:25 K7ZCZ Note Added: 0004916
2018-05-01 08:26 K7ZCZ Note Added: 0004917
2018-06-14 00:57 WA9PIE Note Added: 0005316
2018-06-14 18:59 K7ZCZ Relationship added related to 0002219
2018-06-14 19:02 K7ZCZ Note Added: 0005327
2018-06-23 16:49 WA9PIE Project 2 - Next Dev List (Holding Area) => 3 - Current Dev List
2018-06-28 07:34 PD9FER Note Added: 0005512
2018-06-29 11:16 K7ZCZ Note Added: 0005542
2018-06-29 11:43 PD9FER Note Added: 0005543
2018-06-29 12:05 K7ZCZ Note Added: 0005544
2018-12-06 16:09 WA9PIE Status feedback => assigned
2018-12-06 16:09 WA9PIE Note Added: 0006543
2019-02-10 22:17 WA9PIE Severity minor => crash
2019-02-10 22:17 WA9PIE Testing => Not Started
2019-02-10 22:17 WA9PIE Note Added: 0007351
2019-02-11 20:37 K7ZCZ Note Added: 0007355
2019-02-13 08:02 K7ZCZ Assigned To PD9FER => WA9PIE
2019-02-13 12:33 WA9PIE Note Added: 0007374
2019-02-13 13:00 WA9PIE Note Edited: 0007374 View Revisions
2019-02-13 13:00 WA9PIE Assigned To WA9PIE => K7ZCZ
2019-02-13 13:45 K7ZCZ Note Added: 0007379
2019-02-13 13:45 K7ZCZ Assigned To K7ZCZ => WA9PIE
2019-02-13 15:02 WA9PIE Note Added: 0007382
2019-02-13 15:02 WA9PIE Assigned To WA9PIE => K7ZCZ
2019-02-15 10:46 K7ZCZ Note Added: 0007399
2019-02-15 10:46 K7ZCZ Assigned To K7ZCZ => WA9PIE
2019-02-15 14:42 WA9PIE Note Added: 0007403
2019-02-15 14:43 WA9PIE Assigned To WA9PIE => K7ZCZ
2019-02-16 12:55 K7ZCZ Assigned To K7ZCZ => WA9PIE
2019-02-16 12:56 K7ZCZ Note Added: 0007409
2019-02-17 14:28 K7ZCZ Note Added: 0007419
2019-02-17 19:15 WA9PIE Assigned To WA9PIE => K7ZCZ
2019-02-17 19:15 WA9PIE Note Added: 0007422
2019-02-19 19:11 K7ZCZ Note Added: 0007434
2019-02-20 11:41 K7ZCZ Note Added: 0007457
2019-02-20 11:41 K7ZCZ Status assigned => resolved
2019-02-20 11:41 K7ZCZ Resolution open => fixed
2019-02-20 11:41 K7ZCZ Fixed in Version => 6.5.0.194
2019-02-20 23:38 WA9PIE Note Added: 0007464
2019-02-21 09:42 WA9PIE Steps to Reproduce Updated View Revisions
2019-02-21 09:44 WA9PIE Status resolved => feedback
2019-02-21 09:44 WA9PIE Resolution fixed => reopened
2019-02-21 09:44 WA9PIE Testing Not Started => Beta Failed
2019-02-21 09:44 WA9PIE Note Added: 0007476
2019-02-21 09:52 WA9PIE Note Edited: 0007476 View Revisions
2019-02-21 12:36 K7ZCZ Note Added: 0007478
2019-02-21 13:43 K7ZCZ Assigned To K7ZCZ => WA9PIE
2019-02-21 13:43 K7ZCZ Note Added: 0007480
2019-02-21 15:30 PD9FER Note Added: 0007483
2019-02-21 15:30 PD9FER Status feedback => assigned
2019-02-21 15:46 WA9PIE Note Added: 0007484
2019-02-21 16:02 WA9PIE Status assigned => closed
2019-02-21 16:02 WA9PIE Testing Beta Failed => Beta Successful
2019-02-21 16:02 WA9PIE Note Added: 0007485