This morning I did a bit research after the help files, I hope it sheds some light on the WHY for the recent CHM efforts:

html html
help package size on disk(k) real size(k) chm(k) rtl 38816 22096 1865 lcl 107404 72499 5227
fcl 8548 5148 445 ------------------------------------------------------- All 154768 99743 7537

The disk is NTFS, so probably 4kb clustersize.

Total html bzip2'ed : 2.9MB or so
Total html zipped : 25MB
total chm bzip2'ed : 6,7MB

Extraction time html: 1 minute+ (Core 2 6600 (2.4GHz), winXP's unzip) Extraction time chm : not measurable.

Note that while the bzip2 archive is very small, it must be unpacked entirely before use, contrary to either zip (in theory you could open the zip and extract only the one file) and chm (which has several indexes internally too, the unpacked chm is possibly larger than the html due to this)

Don't know what's your question (perhaps you posted on wrong list) but chm uses LZX compression algorithm internally. You can see that you can't compress too much a chm (from 7,5 MB to 6,7). Bzip2 is block oriented and you can decompress each block separately. You can choose between 100K to 900K block sizes that corresponds to 1-9 bzip2 switch. Bzip2 has the same API than zlib and (if my memory don't fails) allows you to extract a single file.

HTH

No virus found in this outgoing message.
Checked by AVG - http://www.avg.com
Version: 8.0.175 / Virus Database: 270.8.4/1749 - Release Date: 27/10/2008 7:57

Eduardo Morras wrote

Don't know what's your question (perhaps you posted on wrong list) but chm uses LZX compression algorithm internally.

I was about to mention that as well. :-)

I have always had the idea of using the HTML help in a zip file and access the data directly from it. For use with Lazarus IDE and for my own personal projects. TZipFile component created by Darius can do this and allows you to access files in a zip archive as if you are accessing then from a normal hard drive. Only compression still needs to be implement, but that shouldn't be to hard.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

In fact deflate/zip is 18-19 years old and there are lot of better compression algorithm, like LZX. I think there is one implemented in Pascal (ABC if memory don't fails).

Eduardo Morras wrote

In fact deflate/zip is 18-19 years old and there are lot of better compression algorithm, like LZX. I think there is one implemented in Pascal (ABC if memory don't fails).

I'm still trying to find a compression algorithm that beats whatever 7-zip uses. The results are by magnitudes smaller than any other compression algorithm I have seen.

The important thing for TZipFile component is that the archive format must compresses every file separately. Otherwise you can't extract a specific file without unpacking everything first.

The other thing is the algorithms need to be free and supporting Unicode. 7-zip's LZMA does pass both requirements. I'm just not sure if it compresses filed separately - I would imagine it can/does.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Graeme Geldenhuys wrote

The important thing for TZipFile component is that the archive format must compresses every file separately. Otherwise you can't extract a specific file without unpacking everything first.

And at the same time, that is its weak point for packing a large number of similar documents, like the LCL documentation. It only compresses one document at a time, so it cannot use the information of previous documents for compressing. So it is nice for on disk compression, but less valuable for distribution purposes, see also Marco's numbers.

Vincent

In our previous episode, Eduardo Morras said:

No question. Follow up on earlier discussions that dismissed chm, and went for zipped html. I wanted to quantify that a bit. The zip is 30M.

These discussions were partially on maillists (if not this, then fpc-devel) and IRC, so you might have missed it.

The background is a bit that Vincent and I are messing with CHM generation (Vincent for the lhelp package of lazarus, I for the textmode IDE)

but chm uses LZX compression algorithm internally.

Correct. Slightly modified, it resets the window more often. Note that FPC has a portable native CHM package with read/write support.

Hmm. Then that could actually work. There is a native bzip2 decompressor, and the dependancy on libbz2 for crafting the helpfile would not be bad.

But the problem is more that that is also again an own invention with tricks and special crafting, and then it is easier to use an established, "well defined" (well, at least better than own invention) format as CHM, for which KDE and gnome etc come with viewers. (though they are fairly slow)

If I have a bit of time, I'll write a wiki page about helpsystems.

In our previous episode, Graeme Geldenhuys said:

But ZIP is 5-6 times larger than CHM, which can do all this too, and we have the whole software shebang without deps.

I was somewhat surprised that bz2 was another 2 times smaller, and according to Eduardo it is possible to extract blocks separately, without changing compression parameters. Then you could index the tar+bz2 (which files in which block + offset in block) by decompressing fully once, and then extract single files.

Still, since that adds another index and handling, and a lot of work, chm is working and not too bad.

The other thing is the algorithms need to be free and supporting Unicode.

A compression algorithm is not related to unicode. That's the job of the archive component.

7-zip's LZMA does pass both requirements. I'm just not sure if it compresses filed separately - I would imagine it can/does.

A portable, not overly complex implementation in Pascal is also a requirement IMHO. Not an hard one, but the fact that it is already there for CHM makes it one for an alternative.

rtl with 7zip = 762KB
fc with 7zip = 117KB

I don't have the LCL help local. Like I said, I have not seen anything beat 7-zip yet. ;-)

I also double checked. With a "solid archive", one stream of all files, it is still possible to read a specific file from the archive without decompressing the whole bunch.

I used Total Command 6.55 with the 7-zip plugin v0.4.6. Total Commander and the 7-zip plugin is written in Object Pascal (delphi). No dependencies on external libraries. I ran Total Commander under Linux by the way.

7-zip plugin settings:
Compression level: Ultra
Compression method: LZMA
Dictionary size: 32Mb
Word size: 64
Solid Archive (one stream for all files): check

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

See my previous reply. Clearly there is a 7-zip implementation for Object Pascal (delphi) that we could possibly use. Correct - "solid archive" enabled (one stream for all files) makes a huge difference with the final archive size. Good news is that even with solid archive enabled, 7-zip allows reading a specific file without decompressing the whole archive. Total Commander's 7-zip plugin allows for this. And 7-zip beat the CHM compression numbers by far (see numbers in my previous post).

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

I used Total Command 6.55 with the 7-zip plugin v0.4.6. Total Commander and the 7-zip plugin is written in Object Pascal (delphi). No dependencies on external libraries. I ran Total Commander under Linux by the way.

For the latest version of the 7-zip plugin with source code. http://www.totalcmd.net/plugring/7zip_plugin.html

I see it's under LGPL license. I'm pretty sure if we contact the author (Adam Strzelecki) he would grant us the Modified-LGL exception for static linking. Most authors I spoke to before are okay with that.

Okay, I downloaded the source for the latests 7-zip plugin v0.5.8 and noticed it's written in C++ (not Object Pascal - my mistake). Now lets see how good source conversion programs we can find (C++ --> Object Pascal)... ;-) Or create obj file and link them into FPC code (though I have never done this myself).

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

In our previous episode, Graeme Geldenhuys said:

But that is not a help system. It is a solid archive. If I have to depack first it lasts a minute and contains 150MB on disk (first column)

I don't have the LCL help local. Like I said, I have not seen anything beat 7-zip yet. ;-)

Great, I can't wait till you submit a patch that turns it into a helpsystem.

It is not a comparison of archivers, but what to do with the helpsystem. And size is not the only parameter. (native compressor/decompressors are too, so that it can be used in installers that don't need external dlls/.so's that are potentially not installed or distro specific)

I archived the RTL and FCL help generated by fpdoc (lots and lots of HTML files). The RTL HTML help files were also 38MB in size before I archived it. All that is missing from that would be a index file (maybe) and a TOC file.

Great, I can't wait till you submit a patch that turns it into a helpsystem.

We are starting on a custom help system in Q2 next year - if nothing decent exists yet. We need something cross-platform and that can work with fpGUI but will be designed to work for LCL based apps as well. I'm already working on a HTML viewer component for fpGUI - seeing that I can't figure out how to embed Mozilla Firefox in a Free Pascal based application.

So far we are looking at the TZipFile component, but with zlib for a start (and 7zip later) compression. Help file will be similar to OpenOffice or CHM help. A compressed file with HTML and image content and some index and toc files.

It is not a comparison of archivers, but what to do with the helpsystem. And size is not the only parameter. (native compressor/decompressors are too, so that it can be used in installers that don't need external dlls/.so's that are potentially not installed or distro specific)

I understand that, and that's a requirement for us as well. A self-contained help viewer and help files with no external dependencies. But if the help system is designed correctly, you can abstract the compression algorithm and make it swappable. So if some better algorithm comes out in the future, the help system is ready for it. The tiOPF project has done this already if you want sample code (and it doesn't rely on the base tiOPF units - it's stand-alone), and that is what I showed Darius when I spoke to him about the TZipFile component.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

And like I said, 7zip allows a single file to be unpacked from a solid archive. No need to unpack everything. I have verified that with Total Commander (TC) and the 7-zip plugin, otherwise 7-zip will not be an option for a help system. With TC I can browse a 7z solid archive and view a single file and all that lies in the temp directory is the single file I am viewing. For archives like .tar.gz the TC temp directory contains the whole archive uncompressed (not desired).

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

You do have to decompress everything in the archive coming before it (but nothing forces you, or TC, to store the decompressed but undesired data to disk, obviously).

Jonas

In our previous episode, Graeme Geldenhuys said:

So far we are looking at the TZipFile component, but with zlib for a start (and 7zip later) compression. Help file will be similar to OpenOffice or CHM help. A compressed file with HTML and image content and some index and toc files.

CHM has all that, and working code is in the tree now, so why bother? The indexes and toc are several MB each too. (for CHM, they are XML, though you could cook something up binary that is tighter. OTOH it will also compress worse)

Well that is the fun of the CHM stuff. It does all that, but at the same time Windows, Gnome and KDE have viewers, and you can even extract them on cmdline linux (package chmlib comes with the appopriate cmdline tools).

Moreover you can get heaps of authoring tools for it, and the fpdoc process is validated for it. And FPC comes with native encoders and decoders units without dependancies, which means even the textmode IDE uses it.

Note that a good custom html viewer would be great for CHM too. Make sure you abstract where it gets its data.

So if some better algorithm comes out in the future, the help system is ready for it.

Probably with any own invention is that you are your own island again. Usually I'm the one argueing that, but a few percent compression is not worth the trouble for me.

The tiOPF project has done this already if you want sample code (and it doesn't rely on the base tiOPF units - it's stand-alone), and that is what I showed Darius when I spoke to him about the TZipFile component.

If you have more data about what they have working that would be nice.

Well, there are some ones. There is the PAQ family that are the best now. Of course they are research compressors mostly written in C++ but they exist. There is a freepascal/lazarus app that implements a GUI for this and other algorithms, at
http://peazip.sourceforge.net/ (i'm not the developer).

HTH

Jonas Maebe wrote

You do have to decompress everything in the archive coming before it (but nothing forces you, or TC, to store the decompressed but undesired data to disk, obviously).

That would normally make sense and how I understand it with .tar.gz files for example, but when using TC to browse the RTL HTML archive (original 38MB of data) and view a single file, it's instantly open. Compare that time to extracting the whole archive which takes much longer or when viewing a browsing a .tar.gz file. I know you are never supposed to assume, but I assumed that because there is such a *huge* difference in speed when only viewing a single file, then 7-zip must support extracting a single file without unpacking what comes before it for solid archives (or it's just very very efficient).

Anyway, I'll investigate this further so I don't have to make assumptions... ;-)

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

There's a pascal port of LZMA at http://www.birtles.org.uk/programming/. Be aware that to compress/decompress 7-zip (.7z) files requires more than the compression/decompression algorithm. Is necessary to handle the file format.

Luiz

Like I said, I'll investigate all the options available to me when we start the help files and help system in Q2 2009. I just quoted what we had it mind and decided some 1.5 years ago. At the time, not much was available.

Well that is the fun of the CHM stuff. It does all that, but at the same time Windows, Gnome and KDE have viewers, and you can even extract them on cmdline linux (package chmlib comes with the appopriate cmdline tools).

And for Linux (Gnome and KDE) you are relying on 3rd party packages that as far as I have found doesn't come standard with any Linux distro I tried. Plus the Gnome and KDE ones differ in performance and features. Also the whole point of fpGUI is to give the end-user a consistent UI even if they switch between Linux and Windows. Our franchisees are such a case in point. They have centres that have mixed OS's next to each other. Learners come for lessons and grab the first available PC. One day they can be on Linux the next on Windows.

Moreover you can get heaps of authoring tools for it, and the fpdoc process is validated for it. And FPC comes with native encoders and decoders units

True, that is one benefit of CHM. Though fpdoc already creates a nice structure and some index files. So it should be much trouble working with what fpdoc already does with HTML output. TOC and improved Indexing can always be added to fpdoc in required.

Well considering that the RTL CHM size example has been reduced by more than half with 7-zip, that's a nice push for supporting alternative compression. We ship updates of our product on a single CD every six months (500+ centres). We are already pressed for space, so every little bit helps. Plus having to ship 2 CD's to every centre will hugely impact on printing and postage costs.

If you have more data about what they have working that would be nice.

tiOPF has abstract classes for Compression and Encryption in the 'Core' directory.
Actual implementations are in the 'Options' directory.

Core/
tiCompress.pas
tiEncrypt.pas

The following implementations are available in Options/:

Compression:
tiCompressNone.pas // <-- very handy for testing
tiCompressZLib.pas

Encription:
tiEncryptNone.pas // <-- again very handy for testing tiEncryptSimple.pas
tiEncryptDES.pas
tiEncryptBlowfish.pas

All code has been extensively unit tested and example projects are available. All code is available on SourceForge.
http://tiopf.sourceforge.net

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

The do mention peazip in the following page, but they do not have values for it in the "Comparison of efficiency" table.

http://en.wikipedia.org/wiki/Comparison_of_file_archivers

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

An archiver is not a compressor and viceversa. Tar is an archiver but not compress v.gr. You can look at the compressor efficiency at this page http://cs.fit.edu/~mmahoney/compression/text.html, there is Peazip using deflate/zip compression but now it uses lpaq9 and paq8 (the tow of the top 5 algs) and at the
http://www.maximumcompression.com/index.html

Regards, - Graeme -

Again HTH

L

------------------------------------------------------------------------------------------------ If debugging is the art of eliminate bugs in source code, is programming the art of create them?

In our previous episode, Graeme Geldenhuys said:

And for Linux (Gnome and KDE) you are relying on 3rd party packages that as far as I have found doesn't come standard with any Linux distro I tried.

Fedora had both in YUM iirc, though kchmviewer was flaky. But I'm not relying on them, they are additional.

Plus the Gnome and KDE ones differ in performance and features.

While implementing the native Lazarus viewer, it could be easily seen, if you naievely loaded the topics/index into the treeviews, the lazarus version was as slow as gnome/kde. The textmode IDE was a lot faster.

Ergo: the problem is a simplistic treeview widget approach, not the CHM itself. Vincent was already looking into it what could be done to speed it up.

Well, I lean towards using native windows help for chm. But that can of course be the same lazarus viewer too, there is no reason why it would be needed. One could even give the user the choice.

It's already in there.

Not really, the 7zip value is not final. First, test with something that 100% sure can extract single files, and a pascal version. Then we'll compare again.

Moreover, I'm not that interest in one mb.

We ship updates of our product on a single CD every six months (500+ centres). We are already pressed for space, so every little bit helps. Plus having to ship 2 CD's to every centre will hugely impact on printing and postage costs.

That's possible, but maybe there are ways to save space that don't hurt the customer with a insular help system.

Ah, I thought they had a helpsystem. Nevermind.