0002197: Logbook open and close performance is slow

ID	Project	Category	View Status	Date Submitted	Last Update

0002197	1 - Backlog	Bug	public	2017-08-11 17:24	2019-11-01 10:29

Reporter	K7ZCZ	Assigned To	K7ZCZ
Priority	normal	Severity	minor	Reproducibility	always
Status	assigned	Resolution	open

Summary	0002197: Logbook open and close performance is slow
Description	The loogbook application stores records in memory, after reading them from the database. It does so with an egregiously bad data structure. If we imagine each record as a set of values for each column, we can think of each of those values as a “cell”. The value of the “call sign” cell in record #6 might be “K7ZCZ”, while the value of “Frequency” in record 102 might be “14.313.000”. The logbook has about 120 columns that are visible to users; it maintains about 225 columns internally. Thus, each row has 225 cells. If you’ve got a logbook with 60,000 QSOs in it, the logbook is managing values for 225 * 60,000 == 13,500,000 cells. The problem is that these cells are each managed as individually allocated or freed bits of memory. Some work was done to eliminate null, empty, or zero values; but in my testing, a 60,000-row logbook still allocates more than 4 million individual blocks of memory.
Steps To Reproduce	1) We have a 60,000-entry logbook for testing. Import it. 2) Close it. Takes about 50 seconds 3) Open it. Takes about 35 seconds.
Tags	No tags attached.

Module	Logbook
Sub-Module	General
Testing	Not Started

Date Modified	Username	Field	Change
2017-08-11 17:24	K7ZCZ	New Issue
2017-08-11 17:28	K7ZCZ	Relationship added	related to 0002062
2017-09-07 15:59	K7ZCZ	Relationship added	related to 0001332
2017-09-07 22:31	K7ZCZ	Note Added: 0004134
2017-09-07 22:31	K7ZCZ	Assigned To	=> K7ZCZ
2017-09-07 22:31	K7ZCZ	Status	new => assigned
2017-09-18 00:14	WA9PIE	Project	3 - Current Dev List => 2 - Next Dev List (Holding Area)
2019-06-15 11:30	WA9PIE	Project	2 - Next Dev List (Holding Area) => 1 - Backlog
2019-11-01 10:29	K7ZCZ	Relationship added	related to 0003535

related to	0003535	new		3 - Current Dev List	Rig Control polls radio continuously
related to	0002062	new		1 - Backlog	Logbook performance is poor with a large database
related to	0001332	closed	K7ZCZ	Ham Radio Deluxe	Logbook Filter Pane Should Be Static Rather Than Dynamic

K7ZCZ 2017-09-07 22:31 administrator ~0004134	I've got a prototype fix for this. It needs to be integrated to the mainline, and tested further (particularly with Awards and Filter)