View Issue Details

IDProjectCategoryView StatusLast Update
0002219Ham Radio DeluxeBugpublic2019-03-31 11:13
ReporterK7ZCZAssigned ToK7ZCZ 
PrioritynormalSeverityminorReproducibilityhave not tried
Status closedResolutionfixed 
Product Version6.4.0.780 
Target VersionFixed in Version6.5.0.207 
Summary0002219: Logbook: Loading an XML backup is very slow due to the use of a DOM parser
DescriptionWhen restoring an XML logbook backup, the backup is slow for a few different reasons. One reason in particular is that the application uses a pull-mode DOM parser.

If we consider an XML backup file, we might think of its size. Large files might be more than 100 megs. When restoring a backup, the logbook application uses MSXML to open the file. The file is read in, and a data structure of the document is built -- called the document object model (DOM).

This structure is built completely, so the in-memory representation is at least as large as the on-disk content of the file ... probably larger.

Once built, the application traverses the structure and enumerates each record. A single record is inserted into the database, and the enumeration continues.

When the database work is done, the in-memory data structure is torn down. That takes a very long time, since millions of small-sized data items must be individual freed. The user perceives this as a pause after the progress bar shows 100% complete.

Instead of using a DOM parser, the code should use a streaming parser. The restoration code never needs more information from the XML document than it can see in one record. It could process the file sequentially, allocating little more than one record's worth of memory and handling each record in turn. At the end, there's no large composite data structure to be freed. Memory usage is significantly lower, and the user can see an accurate representation of their progress through the requested work.


Steps To Reproduce1) Open Logbook
2) create or select a new, empty database
3) Restore an XML backup of a very large database
BUG#1) Progress after 100% stops and the application is not responsive. Windwos might mark the application as hanging. Really, it's freeing memory used by the XML DOM, as described above.
TagsNo tags attached.
ModuleLogbook
Sub-ModuleData
Testing Beta Successful

Relationships

related to 0002222 closedK7ZCZ Ham Radio Deluxe Logbook: restoring large XML backup causes out of memory crash 
related to 0002696 closedWA9PIE Ham Radio Deluxe Importing Logfile 155,000 + QSO's crashes LB 
child of 0002062 new 1 - Backlog Logbook performance is poor with a large database 

Activities

K7ZCZ

2019-03-07 10:48

administrator   ~0007631

fixed with this checkin:
https://hrdsoftware.visualstudio.com/HRD/_versionControl/changeset/4887

g3ucq

2019-03-09 05:52

viewer   ~0007643

Much faster restore for me and Deleting over 10,000 QSOs was almost instantaneous.
It used to take as long as the Restore if not longer.

WA9PIE

2019-03-10 17:39

administrator   ~0007656

I've validated this and one user has validated it.

Issue History

Date Modified Username Field Change
2017-08-18 00:04 K7ZCZ New Issue
2017-08-18 00:06 K7ZCZ Relationship added child of 0002062
2017-08-19 16:29 K7ZCZ Relationship added related to 0002222
2017-09-18 00:14 WA9PIE Project 3 - Current Dev List => 2 - Next Dev List (Holding Area)
2018-06-14 18:59 K7ZCZ Relationship added related to 0002696
2019-03-07 10:48 K7ZCZ Assigned To => K7ZCZ
2019-03-07 10:48 K7ZCZ Status new => resolved
2019-03-07 10:48 K7ZCZ Resolution open => fixed
2019-03-07 10:48 K7ZCZ Note Added: 0007631
2019-03-08 12:40 K7ZCZ Project 2 - Next Dev List (Holding Area) => 3 - Current Dev List
2019-03-08 12:40 K7ZCZ Fixed in Version => 6.5.0.202
2019-03-08 12:40 K7ZCZ Steps to Reproduce Updated View Revisions
2019-03-09 05:52 g3ucq Note Added: 0007643
2019-03-10 17:39 WA9PIE Status resolved => closed
2019-03-10 17:39 WA9PIE Testing Not Started => Beta Successful
2019-03-10 17:39 WA9PIE Note Added: 0007656
2019-03-31 11:13 WA9PIE Fixed in Version 6.5.0.202 => 6.5.0.207
2019-03-31 11:13 WA9PIE Project 3 - Current Dev List => Ham Radio Deluxe