Splitting a string with a sentence into words shouldn't be hard, should it?

I've adapted the UTF8 examples on the wiki article to this: [1]

I'm assuming this must have been done countless times before.
1) Would somebody know a good solution which I can use? E.g. in the syntaxhighlighter etc, but I'm a bit lazy/afraid to start looking ;)
2) There are probably things I've missed or done incorrectly below. For my edifications, hints and tips more than welcome.

Thanks,
Reinier

[1]
uses
...
LazUTF8
...
procedure Sentence2Words(const Sentence: UTF8String; Words: TStringList);
// Splits words on spaces and lower ASCII characters including #10,#13
into strings
// It will collapse/ignore multiple spaces
// Expects UTF8 sentences; generates UTF8 words
var
  p: PChar;
  CharacterIndex: integer;
  CharLen: integer;
  FirstCharacter: integer;
  FoundSpace: boolean;
begin
  Words.Clear;
  // Indicate we start on character 1:
  CharacterIndex:=1;
  FirstCharacter:=1;
  FoundSpace:=false;
  p:=PChar(Sentence);
  repeat
    CharLen := UTF8CharacterLength(p);
    //todo: find out other UTF8 word delimiters!
    case Ord(P[0]) of
      0..32:
        // Skip double spaces...
        if not(FoundSpace) then
        begin
          // Add what we have up to now
          FoundSpace:=true;
Words.Add(UTF8Copy(Sentence,FirstCharacter,CharacterIndex-FirstCharacter));
          FirstCharacter:=CharacterIndex+1; //skip this space character
        end;
      else FoundSpace:=false;
    end;
    inc(p,CharLen); //Take # bytes into account
    CharacterIndex:=CharacterIndex+1;
  until (CharLen=0) or (p^ = #0);
  // Last word if it didn't end with a space:
  if not(FoundSpace) then
    Words.Add(UTF8Copy(Sentence,FirstCharacter,CharacterIndex));
end;

Hy, is it possible to create hook function on lazarus. And y search function implement the api SAPI under linux

Thank's
Fred

TStringList.Delimiter (with careful use of .StrictDelimiter) is your friend :)

L.

Reinier Olislagers wrote

Splitting a string with a sentence into words shouldn't be hard, should it?

See the strutils unit.

WordCount, ExtractWord:

http://www.freepascal.org/docs-html/rtl/strutils/extractword.html http://www.freepascal.org/docs-html/rtl/strutils/wordcount.html

They can be used to create the functions you want.

Michael.

Thanks, Lukasz & Michael.

The wordcount/extractword route seems a bit more elegant.

I was a bit leary of trying the stringlist route (e.g. bug 19610 has not filled me with confidence, but perhaps that is misplaced). I could also try that.

Thanks again,
Reinier