Hi,

I'm using TChmWriter to pack some html files into a chm file. The index works. I don't have a TOC.
Now I want a full text search.

I set Writer.FullTextSearch to true, it takes a long time to process and the resulting chm is 30% bigger than the total of all files. So I guess it has created some index.

But in lhelp I don't see a search, only the index.

chmmaker can show a search for the example.chm, but can not find anything.

Has someone an example how to create a chm file with full text search?

Mattias

As far as I know setting FullTextSearch, before you call execute, should create the search index. Also the files should match "*.ht" to be included in the search index.

use chmls and see if the file $FIftiMain exists in it. That is the file that contains the search index.

In fpc/packages/chm/src there is chmfilewriter.pas which chmcmd uses to create chm's.

Andrew

I added an lcl program to the ccr in applications/chmmaker that I wrote when I wrote the chm writing code. I just compiled it and it created a chm that was searchable.

It's aimed at a help project approach. The help projects it creates can be compiled with chmcmd as well. (Last I knew)

Regards,

Andrew

Andrew Haines wrote

> I added an lcl program to the ccr in applications/chmmaker that I wrote

Nevermind I had forgotten it's in lazarus/tools anyway.

:)

Andrew

30% bigger than the size of the uncompressed html files?!

But in lhelp I don't see a search, only the index.

There was some bug somewhere a while ago where $FIftiMain was incorrectly searched for as $FiftiMain (i vs I) so maybe make sure lhelp is a recent build.

chmmaker can show a search for the example.chm, but can not find anything.

I attached the example.chm file that I made a couple of minutes ago using chmmaker with the project in the example subfolder. Can you see if it works for you? I opened it in lhelp and the toc, index and search were all working.

Can you PM me the chm(s) that are not working for you? Also what version of fpc are you using for which platform?

Thanks,

Andrew Haines

P.S. Sorry for the storm of emails

Yes.

I updated once more. No change.

Yes, that has a search. And it works.
I wonder, what you are doing different.

Can you PM me the chm(s) that are not working for you? Also what version of fpc are you using for which platform?

linux, fpc 2.7.1

Mattias

I think you should set chmPrj.MakeSearchable := true; (where chmPrj: TChmProject;).
At least I set it (with an ~1 year old lazarus+fpc and lhelp build) and I can search for (complete) words in generated chm using lhelp. (I think I do not set FullTextSearch to true though).

Regards,
Žilvinas

"Žilvinas Ledas" hat am 21. Februar 2012 um 10:38 geschrieben:

TChmProject is just a wrapper that makes a few common things easy. It sets:

  Writer.FullTextSearch := MakeSearchable;

I can not use TChmProject, because my HTML files are not on disk, only in memory.

But thanks. You confirmed that full text search is not broken and I just have to find out what I'm doing different. I simply copied some code from TChmProject.

Mattias

In our previous episode, Mattias Gaertner said:

fulltext search is overriden if there is nothing to write. IOW if indexing found nothing (uppercase extensions maybe, \ vs / ?)

does lazarus/docs/html/build_lcl_docs --outfmt chm work for you? I get a 8mb lcl.chm file. if --outfmt html then the lcl folder has 74mb of html files.

Your chm file should not be bigger than the the uncompressed files unless you are writing only
a couple of tiny html files.

You said in another mail that you are writing the html files from memory. (You must have a good memory! ;) I guess then you are using AddStreamToArchive. FullTextSearch must be set before this is called because the files are scanned as you add them. Also AddStreamToArchive has a default parameter Compress: Boolean, are you setting this to false?

PostAddStreamToArchive should only be used inside code called from the OnLastFile callback.

Can you send me the non working example.chm?

Thanks

Andrew

Andrew Haines hat am 21. Februar 2012 um 15:24
geschrieben:

Bingo. It does not:

Generating Table of contents...
Generating Index...
Finishing compressing...
Exception at 000000000050057F: EAccessViolation:
Access violation.

docs/html/lcl/lcl.chm is 0 bytes.

I have a few thousand html files, about 10k on average.

[...] You said in another mail that you are writing the html files from memory. (You must have a good memory! ;)

Yes. It generates the html on the fly.

I guess then you are using AddStreamToArchive.

No. I set

    Writer.OnGetFileData ;

And in OnWriterGetFileData I write the html into the stream.

FullTextSearch must be set before this is called because the files are

scanned as you add them. Also AddStreamToArchive

I will do

Which FPC version are you using? Those errors occured for me in FPC 2.4.* but disappeared in FPC 2.6.0

FPC 2.7.1 introduced new errors =)

Mattias Gaertner wrote

What do you mean with "Those errors"?

AVs in fpdoc

I'm pretty sure 2.6.0 can create AVs too.

oh ... I haven't yet had any AV in fpdoc from 2.6.0 while I had them all the time in 2.4.x
I guess your mileage can vary then.

In our previous episode, Felipe Monteiro de Carvalho said:

> What do you mean with "Those errors"? AVs in fpdoc

Cough r19903, cough :-)
(not yet merged to 2.6.x)

I haven't seen any since.

Ok, found it. The file extension was wrong.
Fixing that and testing with 3 pages I get a Search. \O/

But it only finds whole words. :-
And clicking on a page gives a black page in lhelp. :(

Processing all files required 12 minutes and terrifying 4GB ram. :( Then comes some final part and it needed 9GB. I only have 8 so it became very slow. :(
Then it went down to 5GB.
Finally it crashed with an AV, just like with the LCL chm. And I have no chm.

Maybe some 64bit issue?

Mattias

:)

But it only finds whole words. :- And clicking on a page gives a black page in lhelp. :(

The whole words is how the words are indexed. It would be fairly easy to match a partial word against the beginning of an indexed word. Beyond that if you want to find "here" in "there" then you would have to dump the search index and create a second search index -> ugly

Processing all files required 12 minutes and terrifying 4GB ram. :( Then comes some final part and it needed 9GB. I only have 8 so it became very slow. :(

"terrifying 4GB ram." :) That made me laugh :)

The memory usage is significantly changed by generating a search index?

I had no crash .

I made an artificial chm file that contained the same file with a different name 4000 times.

the html file was 13k bytes x 4000 (around 52 mb)

the chm was 2.9 mb

(I enabled LZX_USE_THREADS in chmwriter.pas)

time project1

real 10m50.497s
user 36m9.276s
sys 0m3.459s

According to top I used ~320mb of memory

I guess my chm does not have enough unique words and this is why the memory usage is so low.

Yes sir.
Without search: 8 minutes and 600MB.

Should I try 2.6.1?

Ehm, 10 Minutes for 52MB of text is still a lot, isn't it?

According to top I used ~320mb of memory I guess my chm does not have enough unique words and this is why the memory usage is so low.

Mattias

Ok I found a bug here which may affect you or not.

in OnWriterGetFileData make sure you set Stream.Size = 0 before you write your file to it. It will keep the size of the biggest stream you have given it to compress otherwise.

Alternatively you could free the stream and then just assign your opened stream to to the variable.

I really noticed it when it compressed the table of contents of the lcl (it's > 1mb) and suddenly all the files started taking forever to compress. This doesn't effect fpdoc since it uses AddStreamToArchive.

Regards,

Andrew

Yes, this helped.
Now the files are done in under a minute and with only 500MB. But then mem consumption goes up again. Then it goes down to 5GB and seems to be stuck in an endless loop. I cancelled it.

I tried with only 500 files and it worked. That means I get a help and it finds files. But choosing a page just shows black. And after that any page is black.
Note: If I don't use the search but the Index, then I see the pages.

I compiled the chm units with -Criot and found various range check errors and uninitialized variables which I can fix myself. But then I came to a point where don't know what to do:

chmfiftimain.pas(361,49) Warning: Constructing a class "TLeafNode" with abstract method "ChildIsFull"
chmfiftimain.pas(72,15) Hint: Found abstract method: TFIftiNode.ChildIsFull(<TFIftiNode>,AnsiString,LongWord);

And I get an exception in:
#5 0x00000000005552aa in CHILDISFULL (this=0x415906, AWORD=0x409d0c "\311\303f\220H\203\354(H\211\\$\bL\211d$\020L\211l$\030L\211t$ I\211\376I\211\365f\211\323L\211\350H\203", <incomplete sequence \370>, ANODEOFFSET=8234056) at chmfiftimain.pas:688 #6 0x00000000005552aa in CHILDISFULL (this=0x7fffdd9a0b40, AWORD=0x0, ANODEOFFSET=3579960) at chmfiftimain.pas:688

Mattias

In lhelp try commenting chmcontentprovider.pas:1239
TIpChmDataProvider(DataProvider);

This procedure tries to modify the loaded html page to highlight search terms in red.

A parent node is always a TIndexNode so Parent.ChildIsFull =
TIndexNode(Parent).ChildIsFull;

I guess to fix the warning add TLeafNode.ChildIsFull and raise an exception if it is called, since it shouldn't be.

I notice in #6 that AWord = nil. Afaik that shouldn't be the case.

The entries in TIndexNodes are the last word added to it's child node.

The basic overview is each word is written to a TLeafNode. When the leafnode is full then it tells it's parent node the last word it wrote then writes it's data to the final stream and is ready to be filled again.

The index nodes work the same. Every time it is full it tells it's parent the last word it wrote and writes it's data to the final stream.

there is always one root index node that contains the last word of each child node below it.

The index nodes only have information to find leaf nodes. The leaf node has the actual data in it.

Each tier grows exponentially.

So 10000 leaf nodes only requires 3 tiers probably (2 index levels and a leaf level)

IndexNode
IndexNode IndexNode ...
IndexNode IndexNode IndexNode IndexNode...
Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf ....

Anyway I need to think about this more...

Regards,

Andrew

Not sure about this exception but I found an error in TLeafNode.AddWord. It could possibly cause memory corruption and cause your error....

Replace the line

FBlockStream.Write(NewWord[1], Length(Trim(NewWord)));

with

if Length(NewWord) > 0 then
    FBlockStream.Write(NewWord[1], Length(Trim(NewWord)));

The same word can exists twice, once for title results and once for body results. NewWord is the difference between the last word and the new word. So it's length could be 0.

I compiled a chm using the lcl html docs in a folder. Compiled with -Criot I am able to build it with a search index with no exceptions.

Andrew

I commited some changes in r20412

I fixed a couple range check errors.
Searching now will match the beginning of a word as well.

Andrew

The Length(Trim(NewWord)) looks suspicious too me. Is this correct?

Now it stops earlier on my files:

Exception at 0000000000473837: ERangeError:
Range check error.
Backtrace does not help much:

#0 0x00000000004120d0 in fpc_raiseexception ()
#1 0x000000000045fc38 in SYSUTILS_$$_RUNERRORTOEXCEPT$LONGINT$POINTER$POINTER () #2 0x0000000000000000 in ?? ()

Mattias

I've just discovered "info symbol x"

Exception at 000000000053BA25: ERangeError:
Range check error.
[Inferior 1 (process 15556) exited normally]
(gdb) bt
No stack.
(gdb) info symbol 0x53ba25
CHMWRITER$_$TCHMWRITER_$_APPENDBINARYINDEXFROMSITEMAP$TCHMSITEMAP$BOOLEAN_$$_PREPARECURRENTBLOCK
+ 197 in section .text of /usr/local/bin/fpdoc
(gdb)

This at least give me a hint

Regards,

Andrew