View Issue Details

IDProjectCategoryView StatusLast Update
00021092 - Next Dev List (Holding Area)Bugpublic2020-07-02 02:13
ReporterK7ZCZAssigned To 
Status newResolutionopen 
PlatformIntel i7-5960XOSWindows 10 Professional x64OS Version1703
Summary0002109: Canada.Country lookup is too large to load in memory
Data in the Canada.Country module is stored as an XML resource. The resource is about 125 megs in its ASCII representation. It must be converted to Unicdoe to be fed to the XML parser, then shredded into an in-memory data structure.

The Unicode representation would be approximately 250 megs. The in-memory representation is probably 25% of that size, but the Unicode representational remains in memory intil it is completely digested -- the peak memory usage is about 250 * (1 + 0.25) == 312 megs, then. This should fit in a 32-bit process, but is substantial ... and since the XML data is monolithic, it is hard to manage.

A good solution would be to not use XML; then also not use the XML parser. Combined, that means the resource will be substantially smaller and won't need to be converted to Unicode monolithically. The smaller representation can be kept as a static resource where it won't cause a memory allocation, and the data structure built from that representation.
Steps To Reproduce
1) Manually build the Canada.Country DLL
2) Make sure it's next to the HRDLogbook.EXE (same directory)
3) Run HRDLogbook.EXE
4) Place a break point on the parser thread function; it's called "LoadCitiesProc"
5) Connect to a cluster.
6) When the break point is hit, you can step through the code to find the code loads the resource, and fails to convert it to Unicode because memory can't be allocated for the conversion (a 250-meg request). The code tries to recover from this error by allocating a CStringW that holds about 125 milllion characters -- another 250-meg memory allocation request. That fails with an unhanded exception from CString and the app crashes.
TagsNo tags attached.
Sub-ModuleDX Cluster


There are no notes attached to this issue.

Issue History

Date Modified Username Field Change
2017-07-07 13:50 K7ZCZ New Issue
2018-04-13 10:22 WA9PIE Severity minor => crash
2018-04-13 10:23 WA9PIE Project 2 - Next Dev List (Holding Area) => 3 - Current Dev List
2020-07-02 02:13 WA9PIE Project 3 - Current Dev List => 2 - Next Dev List (Holding Area)