Motivated by a user comment on excessive memory consumption of fpspreadsheet during reading of large xlsx files
(http://forum.lazarus.freepascal.org/index.php/topic,33292.msg231598.html#msg231598) I began to investigate an alternative approach to read xlsx files based on SAX instead of DOM which is currently used.

However, I found that SAX is considerably slower than DOM - I always thought it would be the other way round because SAX avoids building the tree of DOM nodes.

Probably, I am doing something wrong.

If somebody wants to look into this issue here's a little demo. It consists of three projects:

* *create_xml* creates an xml file similar to the sharedstrings.xml used by xlsx files internally. The file consists of 500,000 nodes with random strings, and is about 20 MB in size.
* *read_dom* reads this file using the dom routines. On my system this is accomplished within about 1.2 seconds.
* *read_sax* reads the same file using the sax routines of fpc. On my system this takes 4.3 seconds.

So, why is the sax project slower than the dom project?

Werner

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20161225/e038b067/attachment.html> -------------- next part --------------
A non-text attachment was scrubbed...
Name: sax.zip
Type: application/x-zip-compressed
Size: 4921 bytes
Desc: not available
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20161225/e038b067/attachment.bin>

Werner Pamler wrote

Motivated by a user comment on excessive memory consumption of fpspreadsheet

Sorry to not answer your question, but, fpspreadsheet looks very intereresting and first time I really looked.. So it is a spreadsheet without using any MS COM or activex to utilize ms excel? A native spreadsheet for lazarus? Cool! Something I'll probably need some time. Any crashes or major issues, other than slowness being reported in this thread now?

Yes, pure Pascal. No major issues (at least if you don't want it to do anything it was not designed for - like the poster in the forum). See http://wiki.lazarus.freepascal.org/FPSpreadsheet, or http://wiki.lazarus.freepascal.org/FPSpreadsheet_tutorial:_Writing_a_mini_spreadsheet_application

I have been using fpSpreadsheet for years in production in various products, and it works really well.

Michael.

The sax one uses FCL XML routines that are widestring based, and you force them into an ansistring stringlist.

The dom one uses the lazarus XML units, which are 1-byte.

So it is apple and oranges, benchmarking two different XML libs, and on top of that, the Sax example has to do a lot of string conversion, and the dom one not or less.

To make things more comparable you could try to use the FCL XML dom units, or change/port the sax units to use the laz_* versions.