View Issue Details
|ID||Project||Category||View Status||Date Submitted||Last Update|
|0002219||Ham Radio Deluxe||Bug||public||2017-08-18 00:04||2019-03-31 11:13|
|Priority||normal||Severity||minor||Reproducibility||have not tried|
|Target Version||Fixed in Version||188.8.131.52|
|Summary||0002219: Logbook: Loading an XML backup is very slow due to the use of a DOM parser|
|Description||When restoring an XML logbook backup, the backup is slow for a few different reasons. One reason in particular is that the application uses a pull-mode DOM parser.|
If we consider an XML backup file, we might think of its size. Large files might be more than 100 megs. When restoring a backup, the logbook application uses MSXML to open the file. The file is read in, and a data structure of the document is built -- called the document object model (DOM).
This structure is built completely, so the in-memory representation is at least as large as the on-disk content of the file ... probably larger.
Once built, the application traverses the structure and enumerates each record. A single record is inserted into the database, and the enumeration continues.
When the database work is done, the in-memory data structure is torn down. That takes a very long time, since millions of small-sized data items must be individual freed. The user perceives this as a pause after the progress bar shows 100% complete.
Instead of using a DOM parser, the code should use a streaming parser. The restoration code never needs more information from the XML document than it can see in one record. It could process the file sequentially, allocating little more than one record's worth of memory and handling each record in turn. At the end, there's no large composite data structure to be freed. Memory usage is significantly lower, and the user can see an accurate representation of their progress through the requested work.
|Steps To Reproduce||1) Open Logbook|
2) create or select a new, empty database
3) Restore an XML backup of a very large database
BUG#1) Progress after 100% stops and the application is not responsive. Windwos might mark the application as hanging. Really, it's freeing memory used by the XML DOM, as described above.
|Tags||No tags attached.|
|related to||0002222||closed||K7ZCZ||Ham Radio Deluxe||Logbook: restoring large XML backup causes out of memory crash|
|related to||0002696||closed||WA9PIE||Ham Radio Deluxe||Importing Logfile 155,000 + QSO's crashes LB|
|child of||0002062||new||2 - Next Dev List (Holding Area)||Logbook performance is poor with a large database|
fixed with this checkin:
Much faster restore for me and Deleting over 10,000 QSOs was almost instantaneous.
It used to take as long as the Restore if not longer.
||I've validated this and one user has validated it.|
|2017-08-18 00:04||K7ZCZ||New Issue|
|2017-08-18 00:06||K7ZCZ||Relationship added||child of 0002062|
|2017-08-19 16:29||K7ZCZ||Relationship added||related to 0002222|
|2017-09-18 00:14||WA9PIE||Project||3 - Current Dev List => 2 - Next Dev List (Holding Area)|
|2018-06-14 18:59||K7ZCZ||Relationship added||related to 0002696|
|2019-03-07 10:48||K7ZCZ||Assigned To||=> K7ZCZ|
|2019-03-07 10:48||K7ZCZ||Status||new => resolved|
|2019-03-07 10:48||K7ZCZ||Resolution||open => fixed|
|2019-03-07 10:48||K7ZCZ||Note Added: 0007631|
|2019-03-08 12:40||K7ZCZ||Project||2 - Next Dev List (Holding Area) => 3 - Current Dev List|
|2019-03-08 12:40||K7ZCZ||Fixed in Version||=> 184.108.40.206|
|2019-03-08 12:40||K7ZCZ||Steps to Reproduce Updated||View Revisions|
|2019-03-09 05:52||g3ucq||Note Added: 0007643|
|2019-03-10 17:39||WA9PIE||Status||resolved => closed|
|2019-03-10 17:39||WA9PIE||Testing||Not Started => Beta Successful|
|2019-03-10 17:39||WA9PIE||Note Added: 0007656|
|2019-03-31 11:13||WA9PIE||Fixed in Version||220.127.116.11 => 18.104.22.168|
|2019-03-31 11:13||WA9PIE||Project||3 - Current Dev List => Ham Radio Deluxe|