UTF8 strings and characters: what changed

Revision 0 to 1. Removed lines have a minus sign; added lines have a plus sign.

Title: UTF8 strings and characters

New		Line
1	+	## The beauty of UTF-8
2	+
3	+	Bytes starting with '0' (0xxxxxxx) are reserved for [[ASCII]]-compatible single byte characters. With multi-byte codepoints the number of 1’s in the leading byte determines the number of bytes the codepoint occupies. Like this :
4	+
5	+	- 1 byte : 0xxxxxxx
6	+	- 2 bytes : 110xxxxx 10xxxxxx
7	+	- 3 bytes : 1110xxxx 10xxxxxx 10xxxxxx
8	+	- 4 bytes : 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
9	+
10	+	The design of [[UTF-8]] has some benefits over other encodings :
11	+
12	+	- It is backwards compatible with ASCII and produces compact data for western languages. ASCII is also used in markup language tags and other metadata which gives UTF-8 an advantage with any language. However that backwards compatibility does not extend to code, since code has to be recrafted to avoid mangling utf8 strings.
13	+	- The integrity of multi-byte data can be verified from the number of '1'-bits at the beginning of each byte.
14	+	- You can always find the start of a multi-byte codepoint even if you jumped to a random byte position.
15	+	- A byte at a certain position in a multi-byte sequence can never be confused with the other bytes. This allows using the old fast string functions like Pos() and Copy() in many situations. See examples below.
16	+
17	+	Note that similar integrity features are also exists in UTF-16. (D800 range signals first surrogate, DC00 range signals second part of surrogate)
18	+
19	+	- Robust code. Code that deals with codepoints must always be done right with UTF-8 because multi-byte codepoints are common. For UTF-16 there is plenty of sloppy code which assumes codepoints to be fixed width.
20	+	- Most textual data moving in internet is encoded as UTF-8. Processing the data directly as UTF-8 eliminates useless conversions. Many (Unix-related) operating systems use UTF-8 natively.
21	+
22	+	## Examples
23	+
24	+	Simply iterating over characters as if the string was an array of equal sized elements does not work with Unicode. This is not something specific to UTF-8, the Unicode standard is complex and the word "character" is ambiguous. If you want to iterate over codepoints of a UTF-8 string, there are basically two ways:
25	+
26	+	- iterate over bytes - useful for searching a substring or when looking only at the ASCII characters in the UTF8 string, for example when parsing XML files.
27	+	- iterate over codepoints or characters - useful for graphical components like [[SynEdit]], for example when you want to know the third printed character on the screen.
28	+
29	+	### Searching a substring
30	+
31	+	Due to the special nature of UTF8 you can simply use the normal string functions for searching a sub-string. Searching for a valid UTF-8 string with Pos will always return a valid UTF-8 byte position:
32	+
33	+	``` pascal
34	+	uses LazUTF8;
35	+	...
36	+	procedure Where(SearchFor, aText: string);
37	+	var
38	+	BytePos: LongInt;
39	+	CodepointPos: LongInt;
40	+	begin
41	+	BytePos:=Pos(SearchFor,aText);
42	+	CodepointPos:=UTF8Pos(SearchFor,aText);
43	+	writeln('The substring "',SearchFor,'" is in the text "',aText,'"',
44	+	' at byte position ',BytePos,' and at codepoint position ',CodepointPos);
45	+	end;
46	+	```
47	+
48	+	### Search and copy
49	+
50	+	Another example of how Pos(), Copy() and Length() work with UTF-8. This function has no code to deal with UTF-8 encoding, yet it works with any valid UTF-8 text always.
51	+
52	+	``` pascal
53	+	function SplitInHalf(Txt, Separator: string; out Half1, Half2: string): Boolean;
54	+	var
55	+	i: Integer;
56	+	begin
57	+	i := Pos(Separator, Txt);
58	+	Result := i > 0;
59	+	if Result then
60	+	begin
61	+	Half1 := Copy(Txt, 1, i-1);
62	+	Half2 := Copy(Txt, i+Length(Separator), Length(Txt));
63	+	end;
64	+	end;
65	+	```
66	+
67	+	### Iterating over string looking for ASCII characters
68	+
69	+	If you only want to find characters in ASCII-area, you can use Char type and compare with Txt\[i\] just like in old times. Most parsers do that and they continue working.
70	+
71	+	``` pascal
72	+	procedure ParseAscii(Txt: string);
73	+	var
74	+	i: Integer;
75	+	begin
76	+	for i:=1 to Length(Txt) do
77	+	case Txt[i] of
78	+	'(': PushOpenBracketPos(i);
79	+	')': HandleBracketText(i);
80	+	end;
81	+	end;
82	+	```
83	+
84	+	### Iterating over string looking for Unicode characters or text
85	+
86	+	If you want to find all occurrances of a certain character or substring in a string, you can call PosEx() repeatedly.
87	+
88	+	If you want to test for different text inside a loop, you can still use the fast Copy() and Length(). UTF-8 specific functions could be used but they are not needed.
89	+
90	+	``` pascal
91	+	procedure ParseUnicode(Txt: string);
92	+	var
93	+	Ch1, Ch2, Ch3: String;
94	+	i: Integer;
95	+	begin
96	+	Ch1 := 'Й'; // Characters to search for. They can also
97	+	Ch2 := 'ﯚ'; // be combined codepoints or longer text.
98	+	Ch3 := 'Å';
99	+	for i:=1 to Length(Txt) do
100	+	begin
101	+	if Copy(Txt, i, Length(Ch1)) = Ch1 then
102	+	DoCh1(...)
103	+	else if Copy(Txt, i, Length(Ch2)) = Ch2 then
104	+	DoCh2(...)
105	+	else if Copy(Txt, i, Length(Ch3)) = Ch3 then
106	+	DoCh3(...)
107	+	end;
108	+	end;
109	+	```
110	+
111	+	The loop could be optimized by jumping over the already handled parts.
112	+
113	+	### Iterating over string analysing individual codepoints
114	+
115	+	This code copies each codepoint into a variable of type String which can then be processed further.
116	+
117	+	``` pascal
118	+	procedure IterateUTF8(S: String);
119	+	var
120	+	CurP, EndP: PChar;
121	+	Len: Integer;
122	+	ACodePoint: String;
123	+	begin
124	+	CurP := PChar(S); // if S='' then PChar(S) returns a pointer to #0
125	+	EndP := CurP + length(S);
126	+	while CurP < EndP do
127	+	begin
128	+	Len := UTF8CodepointSize(CurP);
129	+	SetLength(ACodePoint, Len);
130	+	Move(CurP^, ACodePoint[1], Len);
131	+	// A single codepoint is copied from the string. Do your thing with it.
132	+	ShowMessageFmt('CodePoint=%s, Len=%d', [ACodePoint, Len]);
133	+	// ...
134	+	inc(CurP, Len);
135	+	end;
136	+	end;
137	+	```
138	+
139	+	### Accessing bytes inside one UTF8 codepoint
140	+
141	+	UTF-8 encoded codepoints can vary in length, so the best solution for accessing them is to use an iteration. To iterate through codepoints use this code:
142	+
143	+	``` pascal
144	+	uses LazUTF8;
145	+	...
146	+	procedure DoSomethingWithString(AnUTF8String: string);
147	+	var
148	+	p: PChar;
149	+	CPLen: integer;
150	+	FirstByte, SecondByte, ThirdByte, FourthByte: Char;
151	+	begin
152	+	p:=PChar(AnUTF8String);
153	+	repeat
154	+	CPLen := UTF8CodepointSize(p);
155	+
156	+	// Here you have a pointer to the char and its length
157	+	// You can access the bytes of the UTF-8 Char like this:
158	+	if CPLen >= 1 then FirstByte := P[0];
159	+	if CPLen >= 2 then SecondByte := P[1];
160	+	if CPLen >= 3 then ThirdByte := P[2];
161	+	if CPLen = 4 then FourthByte := P[3];
162	+
163	+	inc(p,CPLen);
164	+	until (CPLen=0) or (p^ = #0);
165	+	end;
166	+	```
167	+
168	+	### Accessing the Nth UTF8 codepoint
169	+
170	+	Besides iterating one might also want to have random access to UTF-8 codepoints.
171	+
172	+	``` pascal
173	+	uses LazUTF8;
174	+	...
175	+	var
176	+	AnUTF8String, NthCodepoint: string;
177	+	begin
178	+	NthCodepoint := UTF8Copy(AnUTF8String, N, 1);
179	+	```
180	+
181	+	### Showing codepoints with UTF8CharacterToUnicode
182	+
183	+	The following demonstrates how to show the 32bit code point value of each codepoint in an UTF8 string:
184	+
185	+	``` pascal
186	+	uses LazUTF8;
187	+	...
188	+	procedure IterateUTF8Codepoints(const AnUTF8String: string);
189	+	var
190	+	p: PChar;
191	+	unicode: Cardinal;
192	+	CPLen: integer;
193	+	begin
194	+	p:=PChar(AnUTF8String);
195	+	repeat
196	+	unicode:=UTF8CodepointToUnicode(p,CPLen);
197	+	writeln('Unicode=',unicode);
198	+	inc(p,CPLen);
199	+	until (CPLen=0) or (unicode=0);
200	+	end;
201	+	```
202	+
203	+	## Decomposed characters
204	+
205	+	Due to the ambiguity of Unicode, compare functions and Pos() might show unexpected behavior when e.g. one of the string contains decomposed characters, while the other uses the direct codes for the same letter. This is not automatically handled by the RTL. It is not specific to any encoding but Unicode in general.
206	+
207	+	### macOS
208	+
209	+	The file functions of the FileUtil unit also take care of macOS specific behaviour: macOS normalizes filenames. For example the filename 'ä.txt' can be encoded in Unicode with two different sequences (#\$C3#\$A4 and 'a'#\$CC#\$88). Under Linux and BSD you can create a filename with both encodings. macOS automatically converts the a umlaut to the three byte sequence. This means:
210	+
211	+	``` pascal
212	+	if Filename1 = Filename2 then ... // is not sufficient under macOS
213	+	if AnsiCompareFileName(Filename1, Filename2) = 0 then ... // not sufficient under fpc 2.2.2, not even with cwstring
214	+	if CompareFilenames(Filename1, Filename2) = 0 then ... // this always works (unit FileUtil or FileProcs
215	+	```
216	+
217	+	## See also
218	+
219	+	- [[Character and string types]]
220	+
221	+	---
222	+	From the Free Pascal wiki, page "UTF8 strings and characters", as it stood on 2019-12-22 after a change by Trev, out of the wiki's dump of 2024-05-26. Copyright stays with the author who wrote it, according to the old wiki's [[Licensing]] page. If you have concerns about this material, [message the admin](/wiki-contact).

Back to history