Hi all,

I had never used Perl before. Until someone showed me Perl is very fast for text processing (using its powerful regex), despite it's an interpreted language. It even beat Delphi and FPC though both are compiled language. A few lines Perl program almost two times faster than a few pages pascal program.

How can an interpreted language is faster than a compiled language (I assume both has been optimized well and using the best algorithm)? Can FPC improve its text processing related classes/units, so they can be faster than Perl (as it should be, logically)? Especially the TStrings class.

TIA.

-Bee-

has Bee.ography at:
http://beeography.wordpress.com

Give us a test case (some example source code) and I will beat the living crap out of any perl script. Perl is built using Cee, so anything Perl can do Cee can do better.. which means Pascal can do better or similar. Perl is not written in Perl. In other words, perl is just a wrapper around the Cee language.. a similar effect can be done by making terse procedural wrappers around verbose pascal units.

For the number of lines it takes to write perl code: the same nonsense goes on with ruby. You can write a lot of code on one line but doesn't matter that much if you can't go in and extend the code. Take a look at the Mediawiki parser, and compare it to the parser I have built in Simple Wiki. PHP offers lots of syntax shortcuts too but you cannot maintain the mediawiki parser like you can a beautiful case statement. Take a look at smarty source code.. lots of syntax shortcuts, horrible crap. I'm serious when I say take a look.. you have to look to know.

Build wrappers in Pascal and you'll have shorter/terser code.. for example It takes a lot of code to get a file into a string in Pascal.. lots of readlns or tstreams and create/free noise.. but if you use StrLoadFile() from my units it loads it instantly without any class instantation or without any Readln. Using the stack without any free/create can also tersen code a lot.

It depends for what you are doing and what source code was used.. many times ansistrings are very slow if the person did not use a buffer and had lots of concats.

Without any code showing what it is that is 'slow' in pascal, your post is meaningless.. ;-)

Give me/us a test case that you want to process/parse and I'll beat the crap out of it with real benchmarks showing the results. I parse over 15,000 HTML files all the time and I highly doubt some of my parsers could be beaten by perl... and I build wrappers around more wrappers around more wrappers if I want to tersen the code more and more. Most of the units out there for pascal are not wrapped enough.. this is why it takes so many lines of code in Pascal.

Example: synapse. With my wrapper?

s:= GetHtm('http://somesite');
parse(s);

With regular synapse and regular typical VCL style code (no offence)?

var
  buf: string
begin
 blah:= TSynapse.Create;
 blah.setupthis;
 blah.setthis:= 'blah';
 blah.method:= 'GET';
 buf:= blah.strings.text;
 blah.execute;
 blah.free;
 blah:= nil;
 SomeParser.create;
 SomeParser.setupthis;
 SomeParser.string:= buf;
 SomeParser.Exec;
 SomeParser.free;
 SomeParser:= nil;
end;

It's all about wrappers. Perl is just a wrapper. Pascal can do wrappers too. That's why webwrite() is so easy to use, for example. Encapsulation even works well even with procedures, not just objects. Perl contains a lot of quick and dirty procedures and syntax that are mapped to Cee procedures. That's all.

L505

Quite expected response. :)

Here is the Perl code:

--->8--- begin perl code --->8---

open(FH, "Koleksi.dat") or die("Koleksi.dat: $!");
$/ = "</DOC>\n";
while (<FH>) {
   while (m/<(TITLE|TEXT)>(.+)<\/\1>/gcs) {
     $str = lc $2;
     while ($str =~ m/\b([a-z]+)\b/gcs) {
       $arr_kata{$1}++;
       $jum_kata++;
     }
   }
}
print "Word count: $jum_kata\n";
print "Unique word count: ", scalar keys %arr_kata, "\n";

--->8--- end perl code --->8---

The compressed "Koleksi.dat" file can be obtained from http://wawan.web.id/text/Koleksi.zip (957 KB). It's actually an HTML document taken from somewhere. Using time command as simple profiler, resulting:

time perl koleksi.perl

Word count: 126944
Unique word count: 11793

real 0m0.203s
user 0m0.196s
sys 0m0.004s

Tested on Ubuntu (gutsy) with on IBM x60 (1.8 GHz processor and 1 GB memory).

The pascal counter-part resulting almost twice slower. Though not as simple as Perl, the pascal code is quite simple and only using standar fpc's units. But, I won't post the code here to not influence your logic. ;)

Without any code showing what it is that is 'slow' in pascal, your post is meaningless.. ;-)

Just try to beat it. I expect you'll only use basic or default fpc's units. :)

It's all about wrappers. Perl is just a wrapper. Pascal can do wrappers too. That's why webwrite() is so easy to use, for example. Encapsulation even works well even with procedures, not just objects. Perl contains a lot of quick and dirty procedures and syntax that are mapped to Cee procedures. That's all.

Understood. But newbies or newcomers won't write the wrappers by themselves in the first place unless it's already provided by the tool (fpc).

-Bee-

has Bee.ography at:
http://beeography.wordpress.com

Bee wrote

The pascal counter-part resulting almost twice slower. Though not as simple as Perl, the pascal code is quite simple and only using standar fpc's units. But, I won't post the code here to not influence your logic. ;)

For me it would be better if you posted the pascal program, because I don't understand the perl program. Or even better, give a clear problem description.

Vincent

Perl is about regex. They use regex even for stuff where regexes aren't necessary. Since Perl is so fixated on regex, it is likely the case where most time is spent. Moreover, regexes are relative tight statemachines, and most of the time of regex benchmarks is spent in it, and thus outside perl.

TStrings is meant for GUI purposes. It's design and implementation are not optimized.

Or even better, give a clear problem description.

TASKS:

First, is to count number of words inside the document. Second, is to count number of unique words inside the document.

INPUT:

Document format is using HTML-like format for storing articles. Here's the format:

<DOC> (contains an article)
|- <DOCID> (contains article's ID)
|- <TITLE> (contains article's title text)
|- <TEXT> (contains article's content text)

CRITERIA:

A "word" criteria are:
- alphabetic (a..z) character sequence separated by whitespaces or hyphenation characters (space, tab, return, minus).
- character sequence that contains non alphabetic character is NOT considered as a word, ignored it.
- inside <TITLE> and <TEXT> tag, ignore anything inside <DOCID>.

Unique word criteria is case-insensitive. So "word" and "Word" is considered a same word.

EXAMPLE:

<DOC>
<DOCID>TEMPO-022904-111</DOCID>
<TITLE>This is the article title.</TITLE>
<TEXT>
This is The article-content 123abc.
</TEXT>
</DOC>

would give result:

Number of words = 10
Number of unique words = 6

10 words are 5 words from inside TITLE tag and 5 words from inside TEXT tag. The unique words are: "this", "is", "the", "article", "title.", and "content". "123abc." is ignored since it contains numbers.

The document may contain more than one article. Counting is done through out all articles found.

My program (using TStrings) requires about 0.4s to process the given document. :( Tested on the same machine using FPC v.2.0.4.

-Bee-

has Bee.ography at:
http://beeography.wordpress.com

TStrings is meant for GUI purposes. It's design and implementation are not optimized.

Then its our task to optimize it more (and more) so it could as fast as Perl. What I meant here is using standard or default FPC's classes/units. ;) I don't count any third parties class/units/library or non standard attempt here since it'd require extra effort, especially for newbies or newcomers (to FPC).

Of course, I know I can beat Perl using my own optimized TStrings written in assembler by hand mixed by pascal syntax. But, that's not the point. ;)

-Bee-

has Bee.ography at:
http://beeography.wordpress.com

The point is that you are probably using the wrong class (I have not seen your code). To do unique word counts, a hash table is more suitable that a string list.

Vincent

You can't separate performance from requirements. First define a problem, then a solution. Not the other way around.

If you just want to show off, the easiest is simply making a FPC header to pcre, it might be useful even.

Of course, I know I can beat Perl using my own optimized TStrings written in assembler by hand mixed by pascal syntax. But, that's not the point. ;)

TString returns strings. A regex returns a boolean. This means that tstrings must always copy strings.

If you just want to show off, the easiest is simply making a FPC header to pcre, it might be useful even.

Making a header on top of other libraries written by other languages is showing that FPC is lack of powerful units/libraries. For some particular cases maybe it's alright. For example header over OS API, or over Apache, or over widgetsets (gtk, qt, wince, etc), or other common and huge libraries. But for basic or simple task like text processing, IMO we should provide it by ourselves (fpc). Maybe we should write our own pascal regex library that is more powerful than pcre, or at least equal. That way we could be more "independent". ;)

-Bee-

has Bee.ography at:
http://beeography.wordpress.com

To give you a head start, check out:
http://svn.freepascal.org/svn/fpc/trunk/packages/base/regexpr/

It needs testing and fixing.

Vincent

It merely means you don't want to waste time on rewriting a perfectly good library simply because of abstract "language purity" reasons. It is completely irrelevant in what language a library is written if it works well with your program and does what you want.

Jonas

To give you a head start, check out: http://svn.freepascal.org/svn/fpc/trunk/packages/base/regexpr/

You read my mind! I've just been thinking about to use fpc's regex unit. :D

It needs testing and fixing.

How is it compare with regex that comes from FPC 2.0.4? I still can't upgrade to fpc 2.2.0 for some particular reasons. ;)

-Bee-

has Bee.ography at:
http://beeography.wordpress.com

It merely means you don't want to waste time on rewriting a perfectly good library simply because of abstract "language purity" reasons. It is completely irrelevant in what language a library is written if it works well with your program and does what you want.

Yes. If C is very good, I won't use pascal (especially fpc) at the first place. :-P

-Bee-

has Bee.ography at:
http://beeography.wordpress.com

Heck, I'm not even a programmer, and this kludge is about 25% faster than your perl script on my machine....

program koleski;
{$MODE OBJFPC} {$H+}
uses classes, strings;
var
  f: text;
  s:ansistring;
  wc:longint=0;
  wl, ul:TStringList;
  i,n:LongInt;
begin
  assign(f, 'Koleksi.dat');
  reset(f);
  wl:=TStringList.Create();
  ul:=TStringList.Create();
  ul.Sorted:=true;
  ul.Duplicates:=dupIgnore;
  while not eof(f) do begin
    readln(f,s);
    n:=length(s);
    if (n>0) then begin
    StrLower(@s[1]);
      if (s[1]='<') then begin
        if StrLComp(@s[1], '<title>',7) = 0 then begin
          delete(s,1,7);
        end else continue;
      end;
      for i:=1 to n do if not (s[i] in ['a'..'z','0'..'9']) then begin
        if ( s[i]  '<' ) then begin
          s[i]:=#10
        end else begin
          s[i]:=#0;
          SetLength(s,StrLen(@s[1]));
          break;
        end;
      end;
      wl.Text:=s;
      for i:=0 to wl.Count-1 do begin
        s:=wl[i];
        for n:=1 to length(s) do if (s[n] in ['0'..'9']) then begin
          s:='';
          break;
        end;
        if (s'') then begin
          inc(wc);
          ul.Add(s);
        end;
      end;
    end;
  end;
  close(f);
  WriteLn('Word count:',wc, #10'Unique word count:', ul.Count);
end.

- Jeff

Do you really not see the difference between
a) code already written and tested by others
b) code written by yourself

The regexp library is case 1. A program using that library is case 2. Case 1 can be written in C because the original author prefers C. Case 2 can be written in Pascal because you prefer Pascal. "Language superiority" does not enter the picture.

Jonas

Heck, I'm not even a programmer, and this kludge is about 25% faster than your perl script on my machine....

Nope. It's still more or less twice slower. :-D

time perl koleksi.perl

Word count: 126944
Unique word count: 11793

real 0m0.208s
user 0m0.204s
sys 0m0.004s

time ./koleksi2

Word count:126944
Unique word count:11793

real 0m0.452s
user 0m0.432s
sys 0m0.016s

-Bee-

has Bee.ography at:
http://beeography.wordpress.com

Jonas Maebe wrote

The regexp library is case 1. A program using that library is case 2. Case 1 can be written in C because the original author prefers C. Case 2 can be written in Pascal because you prefer Pascal. "Language superiority" does not enter the picture.

It well enters the picture once you start changing the library code.

Vinzent.

> this kludge is about 25% faster than your perl script > on my machine.... Nope. It's still more or less twice slower. :-D

I guess it depends on the hardware:

% time koleksi.pl # perl
Word count: 126944
Unique word count: 11793

real 0m1.019s
user 0m0.992s
sys 0m0.028s

% time koleksi # fpc
Word count:126944
Unique word count:11793

real 0m0.817s
user 0m0.784s
sys 0m0.020s

AMD-K6-700 / SuSE-10.3 / Linux-2.6.22 / perl-5.8.8 / fpc-2.2.0

- Jeff

If you just want to show off, the easiest is simply making a FPC header to pcre, it might be useful even.

Someone had worked on it using Delphi7 and PCRE. A little optimization is done on the TStringList, it uses CompareText instead of CompareStr for text comparison. It does faster than my program (using fpc and the default TStringList), but still about 40% slower than Perl. Plus, the program depends on PCRE library. :(

-Bee-

has Bee.ography at:
http://beeography.wordpress.com

Thanks Jeff, for writing that parser code, I am not good in doing that.

I made it three times as fast on my computer (windows 2000, fpc 2.3.1, P4 1.5 Ghz) using a hashlist for the unique word count. Using a larger textbuf gave an additional 10% speed up:

program project1;
{$MODE OBJFPC} {$H+}
uses classes, strings, contnrs;
const
   bufsize = $1FFF;
var
   f: text;
   s:ansistring;
   wc:longint=0;
   wl:TStringList;
   uhl: TFPStringHashTable;
   i,n:LongInt;
   textbuf: array[0..bufsize-1] of byte;
begin
   assign(f, 'Koleksi.dat');
   reset(f);
   SetTextBuf(f, textbuf, sizeof(textbuf));
   wl:=TStringList.Create();
   uhl:=TFPStringHashTable.Create;
   while not eof(f) do begin
     readln(f,s);
     n:=length(s);
     if (n>0) then begin
     StrLower(@s[1]);
       if (s[1]='<') then begin
         if StrLComp(@s[1], '<title>',7) = 0 then begin
           delete(s,1,7);
         end else continue;
       end;
       for i:=1 to n do if not (s[i] in ['a'..'z','0'..'9']) then begin
         if ( s[i]  '<' ) then begin
           s[i]:=#10
         end else begin
           s[i]:=#0;
           SetLength(s,StrLen(@s[1]));
           break;
         end;
       end;
       wl.Text:=s;
       for i:=0 to wl.Count-1 do begin
         s:=wl[i];
         for n:=1 to length(s) do if (s[n] in ['0'..'9']) then begin
           s:='';
           break;
         end;
         if (s'') then begin
           inc(wc);
           if uhl.Find(s) = nil then
             uhl.Add(s,'');
         end;
       end;
     end;
   end;
   close(f);
   WriteLn('Word count:',wc, #10'Unique word count:', uhl.Count);
end.

AMD-K6-700 / SuSE-10.3 / Linux-2.6.22 / perl-5.8.8 / fpc-2.2.0

Probably because the different fpc version, no? I'm using fpc 2.0.4. However, this is a good news. :)

-Bee-

has Bee.ography at:
http://beeography.wordpress.com

(Probably the other way around, but that breaks MBCS (Delphi) and UTF-8(FPC) compability, and the internationalization of your GUI. So that is not something safe to do)

You are not the first to optimize TStringList in this way, and will not the last. There are reasons why TStringList is as it is.

the easiest is simply making a FPC header to pcre, it might be useful even.

I did that, once upon a time...

http://www.hotlinkfiles.com/files/526004_qpvr0/pcre-fpc.tar.gz

-Jeff

I made it three times as fast on my computer (windows 2000, fpc 2.3.1, P4 1.5 Ghz) using a hashlist for the unique word count. Using a larger textbuf gave an additional 10% speed up:

Arrrggghhhh, I hate myself for not able to upgrade to fpc v.2.2.0! I can't find TFPStringHashTable on fpc v.2.0.4! :((

-Bee-

has Bee.ography at:
http://beeography.wordpress.com

Well done Vincent!! :-) I can confirm your results...

time ./project1
Word count:126944
Unique word count:11793

real 0m0.185s
user 0m0.140s
sys 0m0.000s

time perl project1.perl
Word count: 126944
Unique word count: 11793

real 0m0.281s
user 0m0.244s
sys 0m0.016s

Hardware: Intel P4 CPU 2.40GHz with 1Gig RAM
FPC Compiler: v2.2.0

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Vincent said it was 3 times faster. I expected the result would be about 0.10s. Or am I wrong?

-Bee-

has Bee.ography at:
http://beeography.wordpress.com

Bee wrote

Vincent said it was 3 times faster. I expected the result would be about 0.10s. Or am I wrong?

Maybe that's machine dependent.... I'll try the one without the hash table to see the difference. Otherwise, lets just compare the sys time and say it's 16x faster. ;-)

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Maybe I have a relatively slow computer, so I get more speedup. Keep in mind, that disk time is constant.

So for example, total time is disk time + processing time For me:
StringList: 10 + 50 = 60
Hashtable: 10 + 10 = 20
Speedup: 3 times

For Graeme with his faster computer (2x processor time): StringList: 10 + 25 = 35
Hashtable: 10 + 5 = 15
Speedup: 2.3

Please choose appropiate number to get to the result :-)

Vincent

It was three times faster than the string list version of Jeff. I don't have a perl interpreter :-).

Vincent

Vincent Snijders wrote

Maybe I have a relatively slow computer, so I get more speedup. Keep in mind, that disk time is constant.

I'm also not sure if FPC compiler parameters where used. I did. I compiled as: fpc project1.pas

Anyway, here is the Hash Table vs No Hash Table results. Quite a difference in speed when using the hash table.

time ./project1_nohashtable
Word count:126944
Unique word count:11793

real 0m0.291s
user 0m0.276s
sys 0m0.004s

time ./project1
Word count:126944
Unique word count:11793

real 0m0.196s
user 0m0.132s
sys 0m0.008s

time perl ./project1.perl
Word count: 126944
Unique word count: 11793

real 0m0.292s
user 0m0.268s
sys 0m0.000s

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

It could be wise to add -O3 for anything considered a benchmark :-)

Marco van de Voort wrote

It could be wise to add -O3 for anything considered a benchmark :-)

It squeezed another 0.015s out of the time making it even faster. :-)

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Alright everyone. The answers are enough. Now I can say confidently that pascal (fpc v.2.2) is FASTER than Perl, including in text processing. ;)

Thank you very much to everyone who involves in this thread. :)

-Bee-

has Bee.ography at:
http://beeography.wordpress.com

That was fun!!! :-0 So what's the task for tomorrow? ;-)

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

[ Charset ISO-8859-1 unsupported, converting... ]

I don't think that everybody that saw:

http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=all&calc=Calculate&xfullcpu=1&xmem=0.5&xloc=0&binarytrees=1&chameneos=0&message=0&fannkuch=1&fasta=1&knucleotide=1&mandelbrot=1&meteor=0&nbody=1&nsieve=1&nsievebits=1&partialsums=1&pidigits=1&recursive=1&regexdna=1&revcomp=1&spectralnorm=1&hello=0&sumcol=1

ever doubted that.

But people who have seen
http://shootout.alioth.debian.org/gp4/benchmark.php?test=regexdna&lang=all may have doubted that.

I wonder where fpc would end up in that list,
A: if it uses its own regexpr parser (atm not good enough) B: if it uses Jeff's header translations for pcre

Vincent

But people who have seen http://shootout.alioth.debian.org/gp4/benchmark.php?test=regexdna&lang=all may have doubted that.

Vincent, are we connected or what?! I was about to post the very exact URL! :-D

I wonder where fpc would end up in that list, A: if it uses its own regexpr parser (atm not good enough) B: if it uses Jeff's header translations for pcre

I prefer it would use fpc own regexpr unit instead of using other language libraries. IMO, it'd show the power of pascal (fpc) by itself. ;)

-Bee-

has Bee.ography at:
http://beeography.wordpress.com

Sure, but as Jonas pointed out it is better to use a good library than the write a bad library yourself. Feel free to improve the regexpr unit, if you want it to be used.

Vincent

Graeme Geldenhuys wrote

That was fun!!! :-0 So what's the task for tomorrow? ;-)

for tomorrow the homework is: improve fpc regexp capability and get fpc in the top 5 on the regex-dna shootout ratings ;)

Regards, - Graeme -

henry

Sure, but as Jonas pointed out it is better to use a good library than the write a bad library yourself.

And someone would claim that the speed comes from the library (c?), not from pascal. :P It's a LANGUAGE shootout btw, not LIBRARY shootout. Maybe you had forgotten that. ;)

-Bee-

has Bee.ography at:
http://beeography.wordpress.com

A. That is not a problem. A couple of other, non-c participants use pcre. B. The same is done with pidigits.
C. It shows that fpc has good interoperability.

Well, considering that perl's or Python's speed greatly rely on the underlying C-implementation of (at least) particular functionality, all those comparisons basically boil down to C vs. C, anyway. Or don't they?

;)

Vinzent.

A. That is not a problem. A couple of other, non-c participants use pcre.

I myself would consider them lacks of the language advantage on their own.

B. The same is done with pidigits. C. It shows that fpc has good interoperability.

We could submit more than one programs, one using fpc own regepxr unit, another using pcre. This way we would also implicitly/indirectly compares regepxr performance between fpc own regexpr against other libraries, besides the good interoperability of fpc. ;)

-Bee-

has Bee.ography at:
http://beeography.wordpress.com

Well, considering that perl's or Python's speed greatly rely on the underlying C-implementation of (at least) particular functionality, all those comparisons basically boil down to C vs. C, anyway. Or don't they?

Exactly! That's why I prefer to use fpc own regexpr unit. :)

-Bee-

has Bee.ography at:
http://beeography.wordpress.com

Why not SetLength(s,i)? StrLen is _very_ expensive. I don't see a way how another #0 can be before.

That is right, I am working on a version which does not do that anymore.

Vincent

What about the TRegExpr library?
http://www.regexpstudio.com/TRegExpr/TRegExpr.html The license is MIT-like...

Cheers,
Flávio

Does someone know, why java 6 -server is that fast?

Mattias

Or you can use TRegexp or regexp2.pas units..

But from my testing, TRegexp is a bit slow on big files.

Not sure if the latest TRegexp is faster:

http://www.regexpstudio.com/TRegExpr/TRegExpr.html

L505

Can someone do a test for 5 minutes of parsing and see if things slow down or speed up for one of the programs?

That takes away process load time too..

example: the time it takes to fork the process.

Not sure if perl scripts are initially faster since you don't have to fork a process, assuming perl is already in memory waiting for the script.

Rather I mean can someone do a more realistic test such as parsing 1500 files instead of one single file. At least in my line of work, I never parse one single file.. it is usually about 1500 or more at a time.

Recent versions of JCL include a pcre header, too. Even a .obj is provided for linking statically :-)

-Flávio

Florian Klaempfl wrote

Vincent Snijders schrieb: Why not SetLength(s,i)? StrLen is _very_ expensive. I don't see a way how another #0 can be before.

No more strlen:
http://www.hu.freepascal.org/fpcircbot/cgipastebin?msgid=1432

Vincent

Op Wed, 31 Oct 2007, schreef Vincent Snijders:

One more possible speedup: Why are you using strlower and strlcomp instead of lowercase/pos? The latter ones are probably faster and they don't care of code pages which are not relevant in this example.

Daniël

Vincent Snijders wrote

No more strlen: http://www.hu.freepascal.org/fpcircbot/cgipastebin?msgid=1432

Wow, that version improved quite a bit from the previous one!!

time ./project1_fast
Word count:126944
Unique word count:11793

real 0m0.107s
user 0m0.100s
sys 0m0.000s

time perl ./project1.perl
Word count: 126944
Unique word count: 11793

real 0m0.271s
user 0m0.248s
sys 0m0.008s

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

> > No more strlen: > http://www.hu.freepascal.org/fpcircbot/cgipastebin?msgid=1432

This doesn't work if you have spaces in front of the < tags >

<sometag>
<sometag>

I'm not sure if the Perl one fails too though.
I don't have perl installed and can't test it ;-)

A real parser doesn't care about whitespace in front. And will be a bit slower.. because of that check.

L505

{$MODE OBJFPC} {$H+}
uses sysutils, strings, contnrs;
const
  chars : set of char = ['a'..'z','0'..'9'];
var
  f: text;
  line : ansistring;
  p, pword : pchar;
  saved: char;
  wc : longint;
  counting, good : boolean;
  unique: TFPStringHashTable;
  textbuf: array[1..4096] of byte;
  when : tDateTime;
function do_tag( var s: ansistring; var p: pchar):boolean;
var
  pword: pchar;
begin
  pword := p;
  while p^  '>' do
    inc(p);
  p^ := #0;
  result := ('<title'=pword) or ('<text'=pword);
end;
begin
 when := time;
  assign(f, 'Koleksi.dat');
  reset(f);
  SetTextBuf(f, textbuf, sizeof(textbuf));
  wc := 0;  counting := false;
  unique := TFPStringHashTable.Create;
  while not eof(f) do
  begin
    readln(f, line );
    if '' = line then continue;
    line := lowercase( line );
    p := pchar( line );
    repeat
      // Skip junk.
      while (p^  #0) and (not (p^ in chars)) do
      begin
        if '<' = p^ then
          counting := do_tag( line, p );
        inc(p);
      end;
      // Build word.
      pword := p;
      good := true;
      while p^ in chars do
      begin
        if not (p^ in ['a'..'z']) then good := false;
        inc(p);
      end;
      if counting and good then
        if pword  p then
        begin
          saved := p^;
          p^ := #0;
          inc( wc );
          if unique.Find( pword) = nil then
            unique.Add( pword,'');
          p^ := saved;
        end
    until #0 = p^;
  end;
  close(f);
  writeln( ((time-when)*secsPerDay):0:3 );
  WriteLn('Word count:',wc, #10'Unique word count:', unique.Count);
end.
{
Word count: 126944
Unique word count: 11793
}

__________________________________________________ Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

I think strlower is faster than lowercase, because it can do the change in place. Lowercase needs to allocate a new ansistring for the result.

Vincent