Ok Guys.
I already know how convert PDF files in Text files. To search strings in Text files, which lib in FPC do you recommends? I need speed and the program is a console program.

Thanks,
Marcos Douglas

Somebody can help me please?
I need to search strings in Text files using just FPC. I'm using the xpdf (pdftotext.exe) to convert PDF files to Text files and searching some words in these files.
If is possible to search in PDF files is the better for me. But text files is ok too.

Thanks
Marcos Douglas

Somebody can help me please? I need to search strings in Text files using just FPC.

how about reading every line and then using Pos() to see if some string is there?

HTH ;-)

Cheers, A.

I don't think this way is the fast way :(
I have many PDF files with several pages each.

Marcos Douglas

MSEgui ttextstream has search functions (lib/common/kernel/msestream.pas), I don't know if it is fast enough for your purpose.

Martin

Thanks Martin, I will see this file.

don't know if it is fast enough for your purpose.

I suppose your implementation always is fast enough ;)

Marcos Douglas

In our previous episode, Marcos Douglas said:

You'll be surprised. I've done multi million line logfiles that way. A pdf2txt is infinitely slow compared with such processing.

Ok, but I need to work with pure text. The PDF files are "compiled".

Marcos Douglas

Well, there at least two gotchas there. First, it's better to use a reasonable (= large enough) buffer size. Second, the simplest approach implying reading line by line and searching using Pos() obviously isn't sufficient for searching across line breaks, i.e. you either need to handle that yourself, or use some unit providing such functionality.

Tomas

Which unit do you recommends?

Marcos Douglas

Sorry, I'm confused now. Do you want to search for text strings within PDF files or within regular text files?

Tomas

I need to search for text strings in PDF files... but I don´t know do that, so I use the pdf2txt[1] to convert in pure text. If there is a unit that can search for text within a PDF file, would be the best solution for me.

[1] http://www.foolabs.com/xpdf/home.html

Marcos Douglas

I see, but that means that Marco's comment applies - conversion of PDF files to pure text will probably require more time than finding your string within the (converted) text file.

Tomas

I agree. But as I search for text within PDF files?

Marcos Douglas

I don't have any specific one in mind, I just wanted to point out that you need to take care about that. Personally, I'd probably use my unit Buffered (http://www.volny.cz/xhajt03/buffered.zip) and handle the part related to spanning of text across lines myself if relevant (obviously, that depends on your use case), but I'm sure that there are more complete solutions for your needs readily available too; my comment was just to signalize that "Pos" mentioned by Marco may not be sufficient by itself.

BTW, to answer your other e-mail too - I don't know about solutions for searching within PDF files directly, I'd also use the external converter myself.

Tomas

Sorry,

I agree. But as I search for text within PDF files?

I assumed true the following statement of yours...

[Somebody can help me please?
I need to search strings in Text files using just FPC.]

so I suggested the fast-enough (in my opinion) use of the Pos function. If _now_ you want to search directly in the PDFs, then I cannot help you, as I myself would go for the conversion to text files on first instance.

TTYTT, I usually search for text in PDFs (using Acrobat Reader, just searching, when I need to convert some e-books) and I see it's blooody slow, compared to a search in a plain text file.
I doubt (but also would like to be discredited here...) there is a faster library than the Adobe Acrobat Reader one itself, so I think the combination of the two methods (conversion, and then search) would bring to the same overall speed; if not higher...

Just my last 2c on the thing.

P.S.
Why don't you give it a try, cronometer at hand, first?

Ok Tomas. Thanks for informations and file (I'll see it).

Marcos Douglas

Yes, I changed my first mail because nobody answered me about search in PDF files!

so I suggested the fast-enough (in my opinion) use of the Pos function. If _now_ you want to search directly in the PDFs, then I cannot help you, as I myself would go for the conversion to text files on first instance.

I'll try this method too.
Now, I'm trying to convert the PDF files as best I can... I don't know if pdf2txt is the best way...

Yes, that's what I thought too ;-)

Just my last 2c on the thing. P.S. Why don't you give it a try, cronometer at hand, first?

Because I have not gotten in this stage yet :)

Thanks,
Marcos Douglas

Hello FPC-Pascal,

Monday, November 1, 2010, 8:40:45 PM, you wrote:

MD> On Mon, Nov 1, 2010 at 4:27 PM, Alberto Narduzzi

MD> Yes, I changed my first mail because nobody answered me about search MD> in PDF files!

Search an string in a pdf directly is far away from trivial, you need at least a parse.r and a decompressor. The decompressor is not a problem usually as most files have streams compressed with zlib, but the parser is very complex, locate the text zones is easy (more or less) but if the strings have international characters, it implies the use o CMAPs which is a nighmare.

So if you are looking for ASCII words, use PDF2Text and use the POS function over the result:

function HaveString(Filename: String; TheString: string): Boolean;
var
  F: TFileStream;
  S: String;
  AtPos: integer;
begin
  Result:=false;
  F:=TFileStream.Create(Filename,fmOpenReadOnly);
  SetLength(S,F.Size);
  if F.Size>0 then
  begin
    F.ReadBuffer(S[1],F.Size);
    AtPos:=Pos(UpperCase(TheString),S);
    if AtPos>0 then
      Result:=true;
  end;
  F.Free;
end;

Hi,

I don't know the PDF file format at all, but I would guess if you look at the many pdf-to-text conversion tools, you will see what they have done. In all likelihood, those conversion tools will be written in C, which you will have to translate to Object Pascal - but that is normally not a difficult task, just a boring task.

Regards,
- Graeme -

On Mon, November 1, 2010 21:52, José Mejuto wrote: .
.

If I understand it correctly, this assumes reading the whole file into on the size of that file and other conditions,
this may or may not be advisable...

Tomas

Hello FPC-Pascal,

Tuesday, November 2, 2010, 11:02:18 AM, you wrote:

TH> If I understand it correctly, this assumes reading the whole file into TH> on the size of that file and other conditions, TH> this may or may not be advisable...

Yes, and a pdf2text conversion will reduce the PDF file to a 1% of its original size, so unless you handle 10 gigabyte PDFs should be not problem in loading the whole file in memory.

I doubt that there are memory problems as running pdf2text will for sure consume more memory that the result file size.

Of course if you will end up with 300 megabytes txt files then a different approach would be needed using a buffer with a window over the size of the searched text.

Also logic will be different if you would like to match one word, several words, large sentences, sequeces of chars, etc.

I need to search several words. So, I can't use Pos function to search each word.
My algorithm need to read each word (token) just one time to be fast. I'll define each separator for each Token like <space>, comma, "/", "\", <enter>, etc. For each token found, I'll search a combination in my lists of words. If I found a match, I need to know which page the token was found...

Marcos Douglas

Marcos Douglas wrote

If I found a match, I need to know which page the token was found...

That may prove most difficult, since I doubt your pdf to text will preserve that very well.

Jeff.

Yeah... that is a big problem...
Maybe, if I have only one number in all line so this is the page number.

Marcos Douglas