Hi,

I'm writing a little personal program in Lazarus that manages pdf files. One of the things I want to do is search for text/phrases within the pdfs. Has anybody tried to do this before and if so what is the best (easiest) way you've come across?

I've detailed what I've been doing below, but this is for background information, as after messing about with it for a couple of days I am not so sure this is the most sensible way to go about this even if I can get it work. The awk count command detailed below was just me trying out a proof of concept and for the real search I was planning on it being slightly more sophisticated, but failed at the first hurdle!

I will appreciate your opinions and experiences please. Many thanks.

Martin Collins

Free Pascal Compiler version 2.6.2-5 [2013/07/25] for x86_64 Lazurus SVN 1.3
Awk - GNU Awk 4.0.1

I'm using Linux and have access to all the opensource goodies that offers. I Googled for a pure pascal solution and did not find anything. So over the last couple of days I been experimenting with pdftotext and then awk on the text files, both executed through TProcess.

Working on the bash command line awk is fine but it seems to play up when executed through TProcess. I think it's an awk (or stupid me) problem rather than a TProcess (note: I am an awk novice and not an experienced programmer in general!).

The bash command line awk instruction (to count the number of search string instances) -

awk '$1 ~ /searchstring/ {++c} END {print c}' FS=: textfile.txt

In a simple pascal program to replicate the above, this works;
     ...
     aString := 'awk ''$1 ~ /searchstring/ {++c} END {print c}'' FS=:
textfile.txt';
     AProcess.CommandLine := aString;
     AProcess.Execute;
     ...

but requires return to be pressed before the program exits and 'CommandLine' is depreciated!

This fails;
     AProcess.Executable := 'awk';
     aString := '''$1 ~ /searchstring/ {++c} END {print c}'' FS=:
textfile.txt';
     AProcess.Parameters.Text := aString; // same with Parameters.Add
     AProcess.Execute;
     ...

with the error, awk: cmd. line:1: ^ invalid char ''' in expression

One minor warning for background. On a given distro, the PDF-related utilities usually use a single underlying library. So if you come across a situation where you're having problems extracting content, it can be more useful to look at a system upgrade than spending time trying to hack in updated versions of utilities such as pdftotext.

I'd suggest reading the file back into a stringlist, and manipulating it in Pascal. There might be efficiency problems if you're dealing with /really/ big files, but that way you'll be able to move forward and backward in the file if you want context, have a chance at handling UTF-8 properly and so on.

AWK was all very well when it was the only tool available, and I'm generally defensive of Perl. But if the main program is already written in Pascal you might as well use it for the text handling as well.

Thanks for your advice & help Mark,

Typical, after spending ages searching, I finally found a couple of pascal solutions this morning once I googled with the word Delphi instead of freepascal/lazarus. Then checked and of course freepascal has the same functionality.

The first is reading the file into a string list and then using pos to see if the search term exists. This is fairly simple and may meet my requirements.

The second is searchbuf which appears to require an understanding of pointers - I've only just got my head around creating my own classes so I may need to study a little more for this one!

However, both still require converting the pdf file to text first using pdftotext. So I'll keep on looking for a pure pascal solution.

Thanks & best regards,

Martin Collins

One minor warning for background. On a given distro, the PDF-related utilities usually use a single underlying library. So if you come across a situation where you're having problems extracting content, it can be more useful to look at a system upgrade than spending time trying to hack in updated versions of utilities such as pdftotext.

I'd suggest reading the file back into a stringlist, and manipulating it in Pascal. There might be efficiency problems if you're dealing with /really/ big files, but that way you'll be able to move forward and backward in the file if you want context, have a chance at handling UTF-8 properly and so on.

AWK was all very well when it was the only tool available, and I'm generally defensive of Perl. But if the main program is already written in Pascal you might as well use it for the text handling as well.

Martin Collins wrote

One of the things I want to do is search for text/phrases within the pdfs.

Do you already know in what way text is coded in PDFs ?

I suppose this can be very complicated.

- finding the areas in the file that in fact contain text
- using the appropriate decompression (AFAIK, PDFs can )
- finding out the code (such as ANSIxxx, UTF-8, UTF16, or whatever) the text is stored in.

-Michael