UTF8 Tools: what changed
Revision 0 to 1. Removed lines have a minus sign; added lines have a plus sign.
Title: UTF8 Tools
| Old | New | Line | |
|---|---|---|---|
| 1 | + | ## About | |
| 2 | + | ||
| 3 | + | This code allows to process Unicode text and determine for unicode chars: | |
| 4 | + | ||
| 5 | + | - if char "letter" | |
| 6 | + | - if char "digit" | |
| 7 | + | - if char upper-case, lower-case | |
| 8 | + | - if char "white space" | |
| 9 | + | - if char "punctuation" | |
| 10 | + | - etc. | |
| 11 | + | ||
| 12 | + | Also it has class to read/write Unicode from/to TStream. | |
| 13 | + | ||
| 14 | + | # Units | |
| 15 | + | ||
| 16 | + | ## Using streams | |
| 17 | + | ||
| 18 | + | Unit "charencstreams": load/save data from/to almost any text source: | |
| 19 | + | ||
| 20 | + | - ANSI, UTF8, UTF16, UTF32 | |
| 21 | + | - big-endian, little-endian | |
| 22 | + | - with/without BOM | |
| 23 | + | ||
| 24 | + | Demo: | |
| 25 | + | ||
| 26 | + | ``` pascal | |
| 27 | + | f := TCharEncStream.Create; | |
| 28 | + | f.LoadFromFile(OpenDialog1.FileName); | |
| 29 | + | Memo1.Text := f.UTF8Text; | |
| 30 | + | f.Free; | |
| 31 | + | ``` | |
| 32 | + | ||
| 33 | + | ## Character info | |
| 34 | + | ||
| 35 | + | Unit "character": get information about code points using the TCharacter class. Demo: | |
| 36 | + | ||
| 37 | + | ``` pascal | |
| 38 | + | if TCharacter.IsLetter(s[i]) then | |
| 39 | + | s[i] := TCharacter.toLower(s[i]); | |
| 40 | + | ``` | |
| 41 | + | ||
| 42 | + | ## Access UTF-8 by code index | |
| 43 | + | ||
| 44 | + | Unit "utf8scanner": access UTF-8 strings by code index, use case statements on UTF-8 strings and more. Demo: | |
| 45 | + | ||
| 46 | + | ``` pascal | |
| 47 | + | s := TUTF8Scanner.Create(Memo1.Text); | |
| 48 | + | for i := 1 to s.Length do | |
| 49 | + | if TCharacter.IsLetter(s[i]) then | |
| 50 | + | s[i] := TCharacter.toLower(s[i]); | |
| 51 | + | Memo1.Text := s.UTF8String; | |
| 52 | + | s.Free; | |
| 53 | + | ``` | |
| 54 | + | ||
| 55 | + | Case demo: | |
| 56 | + | ||
| 57 | + | ``` pascal | |
| 58 | + | s := TUTF8Scanner.Create(Memo1.Text); | |
| 59 | + | s.FindChars := 'öäü'; | |
| 60 | + | repeat | |
| 61 | + | case s.FindIndex(s.Next) of | |
| 62 | + | {ö} 0: s.Replace('oe'); | |
| 63 | + | {ä} 1: s.Replace('ae'); | |
| 64 | + | {ü} 2: s.Replace('ue'); | |
| 65 | + | end; | |
| 66 | + | until s.Done; | |
| 67 | + | Memo1.Text := s.UTF8String; | |
| 68 | + | s.Free; | |
| 69 | + | ``` | |
| 70 | + | ||
| 71 | + | # Download | |
| 72 | + | ||
| 73 | + | [Download utf8tools.zip](http://www.theo.ch/lazarus/utf8tools.zip) | |
| 74 | + | ||
| 75 | + | --- | |
| 76 | + | *From the Free Pascal wiki, page "UTF8 Tools", as it stood on 2018-12-12 after a change by Zoltanleo, out of the wiki's dump of 2024-05-26. Copyright stays with the author who wrote it, according to the old wiki's [[Licensing]] page. If you have concerns about this material, [message the admin](/wiki-contact).* | |
| 77 | + | Category: Unicode |