Sign in Join
Wiki copytemporary

Unicode Support in Lazarus: what changed

Revision 2 to 3. Removed lines have a minus sign; added lines have a plus sign.

OldNewLine
11## Introduction
22
3-This page covers Unicode support in Lazarus **programs** (console or server, no GUI) and **applications** (GUI with LCL) using features of FPC 3.0+.
3+This page covers Unicode support in Lazarus **programs** (console or server, no GUI) and **applications** (GUI with LCL) using features of FPC 3.0+.
44
5-The solution is cross-platform and uses UTF-8 encoding which is different from Delphi's UTF-16, but you can write code that is fully compatible with Delphi at source code level by remembering just a few rules.
5+The solution is cross-platform and uses UTF-8 encoding which is different from Delphi's UTF-16, but you can write code that is fully compatible with Delphi at source code level by remembering just a few rules.
66
7-The Unicode support is enabled automatically for LCL applications since Lazarus 1.6.0 when compiling with FPC 3.0+.
7+The Unicode support is enabled automatically for LCL applications since Lazarus 1.6.0 when compiling with FPC 3.0+.
88
99### RTL with default codepage UTF-8
1010
11-By default RTL uses the system codepage for AnsiStrings (e.g. FileExists and TStringList.LoadFromFile). On Windows this is a non Unicode encoding, so you can only use characters from your language group (at most 256 characters). On Linux and macOS UTF-8 is typically the system codepage, so the RTL uses here by default CP_UTF8.
11+By default RTL uses the system codepage for AnsiStrings (e.g. FileExists and TStringList.LoadFromFile). On Windows this is a non Unicode encoding, so you can only use characters from your language group (at most 256 characters). On Linux and macOS UTF-8 is typically the system codepage, so the RTL uses here by default CP_UTF8.
1212
13-FPC since version 3.0 provides an API to change the default system codepage of RTL to something else. Lazarus (actually its LazUtils package) takes advantage of that API and changes it to UTF-8 (CP_UTF8). It means also Windows users now use UTF-8 strings in the RTL.
13+FPC since version 3.0 provides an API to change the default system codepage of RTL to something else. Lazarus (actually its LazUtils package) takes advantage of that API and changes it to UTF-8 (CP_UTF8). It means also Windows users now use UTF-8 strings in the RTL.
1414
1515## Usage
1616
17-Simple rules to follow:
17+Simple rules to follow:
1818
19-- Normally use type "String" instead of UTF8String or UnicodeString.
20-- Assign a constant always to a type String variable.
21-- Use type UnicodeString explicitly for API calls that need it.
19+ * Normally use type "String" instead of UTF8String or UnicodeString.
20+ * Assign a constant always to a type String variable.
21+ * Use type UnicodeString explicitly for API calls that need it.
2222
23-These rules make most code already compatible with Delphi when using default project settings.
23+These rules make most code already compatible with Delphi when using default project settings.
2424
2525### Using UTF-8 in non-LCL programs
2626
27-In a non-LCL project add a dependency for **LazUtils** package. Then add **LazUTF8** unit in the uses section of main program file. It must be near the beginning, just after the critical memory managers and threading stuff (e.g. cmem, heaptrc, cthreads).
27+In a non-LCL project add a dependency for **LazUtils** package. Then add **LazUTF8** unit in the uses section of main program file. It must be near the beginning, just after the critical memory managers and threading stuff (e.g. cmem, heaptrc, cthreads). If **CRT** unit is used, it must be specified before **LazUTF8** , otherwise the console output will be garbled.
2828
2929## Calling API functions that use WideString or UnicodeString
3030
31-When a parameter type is WideString or UnicodeString, you can just pass a String to it. The compiler converts data automatically. There will be a warning about converting from AnsiString to UnicodeString which can be either ignored or suppressed by typecasting the String to UnicodeString.
32-
33-Variable S in examples below is defined as String which here means AnsiString.
31+When a parameter type is WideString or UnicodeString, you can just pass a String to it. The compiler converts data automatically. There will be a warning about converting from AnsiString to UnicodeString which can be either ignored or suppressed by typecasting the String to UnicodeString.
3432
35-``` pascal
33+Variable S in examples below is defined as String which here means AnsiString.
34+```
3635procedure ApiCall(aParam: UnicodeString); // Definition.
37- ...
38-ApiCall(S); // Call with String S, ignore warning.
39-ApiCall(UnicodeString(S)); // Call with String S, suppress warning.
36+ ...
37+ ApiCall(S); // Call with String S, ignore warning.
38+ ApiCall(UnicodeString(S)); // Call with String S, suppress warning.
4039```
4140
42-When a parameter type is a pointer PWideChar, you need a temporary UnicodeString variable. Assign your String to it. The compiler then converts its data. Then typecast the temporary variable to PWideChar.
43-
44-``` pascal
41+When a parameter type is a pointer PWideChar, you need a temporary UnicodeString variable. Assign your String to it. The compiler then converts its data. Then typecast the temporary variable to PWideChar.
42+```
4543procedure ApiCallP(aParamP: PWideChar); // Definition.
46- ...
47-var Tmp: UnicodeString; // Temporary variable.
48- ...
49-Tmp := S; // Assign String -> UnicodeString.
50-ApiCallP(PWideChar(Tmp)); // Call with temp variable, typecast to pointer.
44+ ...
45+ var Tmp: UnicodeString; // Temporary variable.
46+ ...
47+ Tmp := S; // Assign String -> UnicodeString.
48+ ApiCallP(PWideChar(Tmp)); // Call with temp variable, typecast to pointer.
5149```
5250
53-Note, in both cases the code is compatible with Delphi. It means you can copy/paste it to Delphi and it works 100% correctly. In Delphi String maps to UnicodeString.
51+Note, in both cases the code is compatible with Delphi. It means you can copy/paste it to Delphi and it works 100% correctly. In Delphi String maps to UnicodeString.
5452
55-A typical case is a Windows API call. Only the "W" versions of them should be called because they support Unicode. Normally use UnicodeString also with Windows API. WideString is only needed with COM/OLE programming where the OS takes care of memory management.
53+A typical case is a Windows API call. Only the "W" versions of them should be called because they support Unicode. Normally use UnicodeString also with Windows API. WideString is only needed with COM/OLE programming where the OS takes care of memory management.
5654
5755## Reading / writing text file with Windows codepage
5856
59-This is not compatible with Delphi nor with former Lazarus code. In practice you must encapsulate the code dealing with system codepage and convert the data to UTF-8 as quickly as possible.
57+This is not compatible with Delphi nor with former Lazarus code. In practice you must encapsulate the code dealing with system codepage and convert the data to UTF-8 as quickly as possible.
6058
6159**Either use RawByteString and do an explicit conversion**
62-
63-``` pascal
60+```
6461uses ... , LConvEncoding;
65- ...
66-var
67- StrIn: RawByteString;
68- StrOut: String;
69- ...
70-StrOut := CP1252ToUTF8(StrIn,true); // Use fixed codepage 1252
71-// or
72-StrOut := WinCPToUTF8(StrIn,true); // Use system codepage in this particular computer
62+ ...
63+ var
64+ StrIn: RawByteString;
65+ StrOut: String;
66+ ...
67+ StrOut := CP1252ToUTF8(StrIn,true); // Use fixed codepage 1252
68+ // or
69+ StrOut := WinCPToUTF8(StrIn,true); // Use system codepage in this particular computer
7370```
7471
7572**... or set the right codepage for an existing string**
76-
77-``` pascal
73+```
7874var
79- StrIn: String;
80- ...
81-SetCodePage(RawByteString(StrIn), 1252, false); // Codepage 1252 (or Windows.GetACP())
75+ StrIn: String;
76+ ...
77+ SetCodePage(RawByteString(StrIn), 1252, false); // Codepage 1252 (or Windows.GetACP())
8278```
8379
84-Note: there must be some text in the string variable. An empty string is actually a Nil pointer and you cannot set its codepage.
80+Note: there must be some text in the string variable. An empty string is actually a Nil pointer and you cannot set its codepage.
8581
86-Windows.GetACP() returns the Windows system codepage.
82+Windows.GetACP() returns the Windows system codepage.
8783
8884## Code that depends very much on Windows codepage
8985
90-There is a "plan B" for code that depends very much on Windows system codepage, or must write to Windows console beyond the console codepage.\
91-See: [[Lazarus with FPC3.0 without UTF-8 mode]].\
92-Fortunately it is not needed often. In most cases it is easier to convert data to UTF-8.
86+There is a "plan B" for code that depends very much on Windows system codepage, or must write to Windows console beyond the console codepage.
87+See: [[Lazarus with FPC3.0 without UTF-8 mode]].
88+Fortunately it is not needed often. In most cases it is easier to convert data to UTF-8.
9389
9490### Writing to console
9591
96-Windows console output works **if** your characters belong to the console codepage. For example a box-drawing characters '╩' in codepage CP437 is one byte **\#202** and is often used like this:
97-
98-``` pascal
92+Windows console output works **if** your characters belong to the console codepage. For example a box-drawing characters '╩' in codepage CP437 is one byte **#202** and is often used like this:
93+```
9994write('╩');
10095```
10196
102-When you convert the code to UTF-8, for example by using Lazarus' source editor popup menu item **File Settings / Encoding / UTF-8** and clicking on the dialog button "Change file on disk", the *╩* becomes 3 bytes (#226#149#169), so the literal becomes a *string*. The procedures *write* and *writeln* convert the UTF-8 string to the current console codepage. So your console program now outputs the '╩' on Windows with any codepage (i.e. not only CP437) and it even works on Linux and macOS. You can also use the '╩' in LCL strings, for instance Memo1.Lines.Add('╩');
97+When you convert the code to UTF-8, for example by using Lazarus' source editor popup menu item **File Settings / Encoding / UTF-8** and clicking on the dialog button "Change file on disk", the _╩_ becomes 3 bytes (#226#149#169), so the literal becomes a _string_. The procedures _write_ and _writeln_ convert the UTF-8 string to the current console codepage. So your console program now outputs the '╩' on Windows with any codepage (i.e. not only CP437) and it even works on Linux and macOS. You can also use the '╩' in LCL strings, for instance Memo1.Lines.Add('╩');
10398
104-If your characters do **not** belong to the console codepage, things get trickier. Then you cannot use the new Unicode system at all.\
99+If your characters do **not** belong to the console codepage, things get trickier. Then you cannot use the new Unicode system at all.
105100See: [[Lazarus with FPC3.0 without UTF-8 mode|Lazarus with FPC3.0 without UTF-8 mode#Problem System encoding and Console encoding .28Windows.29]]
106101
107102## Unicode characters and codepoints in code
108103
109-See details for UTF-8 in [[UTF8 strings and characters]].
104+See details for UTF-8 in [[UTF8 strings and characters]].
110105
111106### CodePoint functions for encoding agnostic code
112107
113-LazUtils package has **unit LazUnicode** with special functions for dealing with codepoints, regardless of encoding. They use the UTF8...() functions from LazUTF8 when used in the UTF-8 mode, and UTF16...() functions from LazUTF16 when used in FPC's {\$ModeSwitch UnicodeStrings} or in Delphi (yes, Delphi is supported!).
114-
115-Currently the {\$ModeSwitch UnicodeStrings} can be tested by defining "UseUTF16". There is also a test program LazUnicodeTest in components/lazutils/test directory. It has 2 build modes, UTF8 and UTF16, for easy testing. The test program also supports Delphi, then the UTF-16 mode is used obviously.
108+LazUtils package has **unit LazUnicode** with special functions for dealing with codepoints, regardless of encoding. They use the UTF8...() functions from LazUTF8 when used in the UTF-8 mode, and UTF16...() functions from LazUTF16 when used in FPC's {$ModeSwitch UnicodeStrings} or in Delphi (yes, Delphi is supported!).
116109
117-LazUnicode allows one source code to work between :
110+Currently the {$ModeSwitch UnicodeStrings} can be tested by defining "UseUTF16". There is also a test program LazUnicodeTest in components/lazutils/test directory. It has 2 build modes, UTF8 and UTF16, for easy testing. The test program also supports Delphi, then the UTF-16 mode is used obviously.
118111
119-- Lazarus with its UTF-8 solution.
120-- Future FPC and Lazarus with Delphi compatible UTF-16 solution.
121-- Delphi, where String = UnicodeString.
112+LazUnicode allows one source code to work between :
122113
123-It provides these encoding agnostic functions:
114+ * Lazarus with its UTF-8 solution.
115+ * Future FPC and Lazarus with Delphi compatible UTF-16 solution.
116+ * Delphi, where String = UnicodeString.
124117
125-- CodePointCopy() - Like UTF8Copy()
126-- CodePointLength() - Like UTF8Length()
127-- CodePointPos() - Like UTF8Pos()
128-- CodePointSize() - Like UTF8CharacterLength() (Deprecated. Use UTF8CodepointSize instead. [See more](https://lazarus-ccr.sourceforge.io/docs/lazutils/lazutf8/utf8characterlength.html))
129-- UnicodeToWinCP() - Like UTF8ToWinCP()
130-- WinCPToUnicode() - Like WinCPToUTF8()
118+It provides these encoding agnostic functions:
131119
132-It also provides an enumerator for CodePoints which the compiler uses for its for-in loop. As a result, regardless of encoding, this code works:
120+ * CodePointCopy() - Like UTF8Copy()
121+ * CodePointLength() - Like UTF8Length()
122+ * CodePointPos() - Like UTF8Pos()
123+ * CodePointSize() - Like UTF8CharacterLength() (**Deprecated**. Use UTF8CodepointSize instead. [See more](<https://lazarus-ccr.sourceforge.io/docs/lazutils/lazutf8/utf8characterlength.html>))
124+ * UnicodeToWinCP() - Like UTF8ToWinCP()
125+ * WinCPToUnicode() - Like WinCPToUTF8()
133126
134-``` pascal
135- var s, ch: String;
136- ...
137- for ch in s do
138- writeln('ch=',ch);
127+It also provides an enumerator for CodePoints which the compiler uses for its for-in loop. As a result, regardless of encoding, this code works:
139128```
129+var s, ch: String;
130+ ...
131+ for ch in s do
132+ writeln('ch=',ch);
133+```
140134
141-Delphi does not provide similar functions for CodePoints for its UTF-16 solution. Practically most Delphi code treats UTF-16 as a fixed-width encoding which has lead to lots of broken UTF-16 code out there. It means using LazUnicode also for Delphi will improve code quality!
135+Delphi does not provide similar functions for CodePoints for its UTF-16 solution. Practically most Delphi code treats UTF-16 as a fixed-width encoding which has lead to lots of broken UTF-16 code out there. It means using LazUnicode also for Delphi will improve code quality!
142136
143-Both units LazUnicode and LazUTF16 are needed for Delphi usage.
137+Both units LazUnicode and LazUTF16 are needed for Delphi usage.
144138
145139## String Literals
146140
147-Sources should be saved in UTF-8 encoding. Lazarus creates such files by default. You can change the encoding of imported files via right click in source editor / File Settings / Encoding.
141+Sources should be saved in UTF-8 encoding. Lazarus creates such files by default. You can change the encoding of imported files via right click in source editor / File Settings / Encoding.
148142
149-Usually {\$codepage utf8} / -FcUTF8 is not needed. This is rather counter-intuitive because the meaning of that flag is to treat string literals as UTF-8. However the new UTF-8 mode switches the encoding at run-time, yet constants are evaluated at compile-time.
143+Usually {$codepage utf8} / -FcUTF8 is not needed. This is rather counter-intuitive because the meaning of that flag is to treat string literals as UTF-8. However the new UTF-8 mode switches the encoding at run-time, yet constants are evaluated at compile-time.
150144
151-So, without -FcUTF8 the compiler (wrongly) thinks the constant string is encoded with system code page. Then it sees a String variable with default encoding (which will be changed to UTF-8 at run-time but the compiler does not know it). Thus, same default encodings, no conversion needed, the compiler happily copies the characters and everything goes right, while actually it was fooled twice during the process.
145+So, without -FcUTF8 the compiler (wrongly) thinks the constant string is encoded with system code page. Then it sees a String variable with default encoding (which will be changed to UTF-8 at run-time but the compiler does not know it). Thus, same default encodings, no conversion needed, the compiler happily copies the characters and everything goes right, while actually it was fooled twice during the process.
152146
153147**Example:**
154148
155-As a rule of thumb, use "String" type and assigning literals works.
156-
157-``` pascal
149+As a rule of thumb, use "String" type and assigning literals works.
150+```
158151const s1 = 'äй';
159-const s2: string = #$C3#$A4; // ä
160-var s3;
161-...
162- s3 := 'äöü';
152+ const s2: string = #$C3#$A4; // ä
153+ var s3;
154+ ...
155+ s3 := 'äöü';
163156```
164157
165-Note: UTF-8 string can be composed from numbers, as done for s2.
158+Note: UTF-8 string can be composed from numbers, as done for s2.
166159
167-Assigning a string literal to other string types than plain "String" is more tricky. See the tables for what works and what doesn't.
160+Assigning a string literal to other string types than plain "String" is more tricky. See the tables for what works and what doesn't.
168161
169162### Assign string literals to different string types
170163
171-Here *working* means correct codepage and correct codepoints. Codepage 0 or codepage 65001 are both correct, they mean UTF-8.\
172-
173-#### Without {\$codepage utf8} or compilerswitch -FcUTF8
174-
175-| String Type, UTF-8 Source | Example | Const (in Source) | Assigned to String | Assigned to UTF8String | Assigned to UnicodeString | Assigned to CP1252String | Assigned to RawByteString | Assigned to ShortString | Assigned to PChar |
176-|----|----|----|----|----|----|----|----|----|----|
177-| const | const s = 'äöü'; | working | working | wrong | wrong | wrong | working | working | working |
178-| String | const s: String = 'äöü'; | working | working | working | working | working | working | working | working |
179-| ShortString | const s: String\[15\] = 'äöü'; | working | working | working | working | wrong encoded | working | working | not available |
180-| UTF8String | const s: UTF8String = 'äöü'; | wrong | wrong | wrong | wrong | wrong | wrong | wrong | wrong |
181-| UnicodeString | const s: UnicodeString = 'äöü'; | wrong | wrong | wrong | wrong | wrong | wrong | wrong | wrong |
182-| String with declared code page | type CP1252String = type AnsiString(1252); | wrong | wrong | wrong | wrong | wrong | wrong | wrong | wrong |
183-| RawbyteString | const s: RawbyteString = 'äöü'; | working | working | working | working | to codepage 0 changed | working | working | working |
184-| PChar | const c: PChar = 'äöü'; | working | working | working | working | wrong | working | working | working |
164+Here _working_ means correct codepage and correct codepoints. Codepage 0 or codepage 65001 are both correct, they mean UTF-8.
185165
186-#### With {\$codepage utf8} or compilerswitch -FcUTF8
166+#### Without {$codepage utf8} or compilerswitch -FcUTF8
187167
188-| String Type, UTF-8 Source | Example | Const (in Source) | Assigned to String | Assigned to UTF8String | Assigned to UnicodeString | Assigned to CP1252String | Assigned to RawByteString | Assigned to ShortString | Assigned to PChar |
189-|----|----|----|----|----|----|----|----|----|----|
190-| const | const s = 'äöü'; | UTF-16 encoded | working | working | working | working | working | working | working |
191-| String | const s: String = 'äöü'; | working | working | working | working | working | working | working | working |
192-| ShortString | const s: String\[15\] = 'äöü'; | wrong | wrong | wrong | wrong | wrong | wrong | wrong | not available |
193-| UTF8String | const s: UTF8String = 'äöü'; | working | working | working | working | working | working | working | working |
194-| UnicodeString | const s: UnicodeString = 'äöü'; | working | working | working | working | working | working | working | wrong |
195-| String with declared code page | type CP1252String = type AnsiString(1252); | working | working | working | working | working | working | wrong | wrong |
196-| RawbyteString | const s: RawbyteString = 'äöü'; | working | working | working | working | to codepage 0 changed | working | working | working |
197-| PChar | const c: PChar = 'äöü'; | wrong | wrong | wrong | wrong | wrong | wrong | wrong | wrong |
168+String Type, UTF-8 Source | Example | Const (in Source) | Assigned to String | Assigned to UTF8String | Assigned to UnicodeString | Assigned to CP1252String | Assigned to RawByteString | Assigned to ShortString | Assigned to PChar
169+---|---|---|---|---|---|---|---|---|---
170+const | const s = 'äöü'; | working | working | wrong | wrong | wrong | working | working | working
171+String | const s: String = 'äöü'; | working | working | working | working | working | working | working | working
172+ShortString | const s: String[15] = 'äöü'; | working | working | working | working | wrong encoded | working | working | not available
173+UTF8String | const s: UTF8String = 'äöü'; | wrong | wrong | wrong | wrong | wrong | wrong | wrong | wrong
174+UnicodeString | const s: UnicodeString = 'äöü'; | wrong | wrong | wrong | wrong | wrong | wrong | wrong | wrong
175+String with declared code page | type CP1252String = type AnsiString(1252); | wrong | wrong | wrong | wrong | wrong | wrong | wrong | wrong
176+RawbyteString | const s: RawbyteString = 'äöü'; | working | working | working | working | to codepage 0 changed | working | working | working
177+PChar | const c: PChar = 'äöü'; | working | working | working | working | wrong | working | working | working
178+
179+#### With {$codepage utf8} or compilerswitch -FcUTF8
198180
199-Remember, assignment between variables of different string types always works thanks to their dynamic encoding in FPC 3+. The data is converted automatically when needed.\
200-Only string literals are a challenge.
181+String Type, UTF-8 Source | Example | Const (in Source) | Assigned to String | Assigned to UTF8String | Assigned to UnicodeString | Assigned to CP1252String | Assigned to RawByteString | Assigned to ShortString | Assigned to PChar
182+---|---|---|---|---|---|---|---|---|---
183+const | const s = 'äöü'; | UTF-16 encoded | working | working | working | working | working | working | working
184+String | const s: String = 'äöü'; | working | working | working | working | working | working | working | working
185+ShortString | const s: String[15] = 'äöü'; | wrong | wrong | wrong | wrong | wrong | wrong | wrong | not available
186+UTF8String | const s: UTF8String = 'äöü'; | working | working | working | working | working | working | working | working
187+UnicodeString | const s: UnicodeString = 'äöü'; | working | working | working | working | working | working | working | wrong
188+String with declared code page | type CP1252String = type AnsiString(1252); | working | working | working | working | working | working | wrong | wrong
189+RawbyteString | const s: RawbyteString = 'äöü'; | working | working | working | working | to codepage 0 changed | working | working | working
190+PChar | const c: PChar = 'äöü'; | wrong | wrong | wrong | wrong | wrong | wrong | wrong | wrong
191+
192+Remember, assignment between variables of different string types always works thanks to their dynamic encoding in FPC 3+. The data is converted automatically when needed.
193+Only string literals are a challenge.
201194
202195## Coming from older Lazarus + LCL versions
203196
204-Earlier (before 1.6.0) LCL supported Unicode with dedicated UTF8 functions. The code was not at all compatible with Delphi.
197+Earlier (before 1.6.0) LCL supported Unicode with dedicated UTF8 functions. The code was not at all compatible with Delphi.
205198
206-Now many old LCL applications continue to work without changes. However it makes sense to clean the code to make it simpler and more Delphi compatible. Code that reads/writes data with Windows system codepage encoding breaks and must be changed. (See #Reading / writing text file with Windows codepage).
199+Now many old LCL applications continue to work without changes. However it makes sense to clean the code to make it simpler and more Delphi compatible. Code that reads/writes data with Windows system codepage encoding breaks and must be changed. (See #Reading / writing text file with Windows codepage).
207200
208-Explicit conversion functions are only needed for I/O with Windows codepage data or when calling Windows Ansi functions. Otherwise FPC takes care of converting encodings automatically. Empty conversion functions are provided to make your old code compile.
201+Explicit conversion functions are only needed for I/O with Windows codepage data or when calling Windows Ansi functions. Otherwise FPC takes care of converting encodings automatically. Empty conversion functions are provided to make your old code compile.
209202
210-- UTF8Decode, UTF8Encode - Almost all can be removed.
211-- UTF8ToAnsi, AnsiToUTF8 - Almost all can be removed.
212-- UTF8ToSys, SysToUTF8 - All can be removed. They are now dummy no-ops and only return their parameter.
203+ * UTF8Decode, UTF8Encode - Almost all can be removed.
204+ * UTF8ToAnsi, AnsiToUTF8 - Almost all can be removed.
205+ * UTF8ToSys, SysToUTF8 - All can be removed. They are now dummy no-ops and only return their parameter.
213206
214-File functions in RTL now take care of file name encoding. All (?) file name related ...UTF8() functions can be replaced with the Delphi compatible function without UTF8 suffix. For example FileExistsUTF8 can be replaced with FileExists.
207+File functions in RTL now take care of file name encoding. All (?) file name related ...UTF8() functions can be replaced with the Delphi compatible function without UTF8 suffix. For example FileExistsUTF8 can be replaced with FileExists.
215208
216-Most UTF8...() string functions can be replaced with the Delphi compatible Ansi...() functions. For example UTF8UpperCase() -\> AnsiUpperCase().
209+Most UTF8...() string functions can be replaced with the Delphi compatible Ansi...() functions. For example UTF8UpperCase() -> AnsiUpperCase().
217210
218-Now Unicode works in non-GUI programs, too. It only requires a dependency for LazUtils and placing LazUTF8 unit into the uses section of main program file.
211+Now Unicode works in non-GUI programs, too. It only requires a dependency for LazUtils and placing LazUTF8 unit into the uses section of main program file.
219212
220213For historical reference, this was the old Unicode support in LCL: [[Old LCL Unicode Support]]
221214
222215## Technical implementation
223216
224-What actually happens in the Unicode system? These 2 FPC functions are called in an early initialization section, setting the default String encoding in FPC to UTF-8 :
225-
226-```text
227- SetMultiByteConversionCodePage(CP_UTF8);
228- SetMultiByteRTLFileSystemCodePage(CP_UTF8);
217+What actually happens in the Unicode system? These 2 FPC functions are called in an early initialization section, setting the default String encoding in FPC to UTF-8 :
229218```
219+SetMultiByteConversionCodePage(CP_UTF8);
220+ SetMultiByteRTLFileSystemCodePage(CP_UTF8);
221+```
230222
231-Under Windows the UTF8...() functions in LazUTF8 (LazUtils) are set as backends for RTL's Ansi...() string functions. Thus those functions work in a Delphi compatible way.
223+Under Windows the UTF8...() functions in LazUTF8 (LazUtils) are set as backends for RTL's Ansi...() string functions. Thus those functions work in a Delphi compatible way.
232224
233225## Open issues
234226
235-- **TFormatSettings** char (bug [27086](http://bugs.freepascal.org/view.php?id=27086)): for example: **ThousandSeparator**, **DecimalSeparator**, **DateSeparator**, **TimeSeparator**, **ListSeparator**. These should be replaced with string to support UTF-8. For example under Linux with LC_NUMERIC=ru_RU.utf8 the thousand separator is the two byte nbsp/160.
236- - **Workaround:** use single space characters instead as done in patch here: [27099](http://bugs.freepascal.org/view.php?id=27099)
227+ * **TFormatSettings** char (bug [27086](<http://bugs.freepascal.org/view.php?id=27086>)): for example: **ThousandSeparator** , **DecimalSeparator** , **DateSeparator** , **TimeSeparator** , **ListSeparator**. These should be replaced with string to support UTF-8. For example under Linux with LC_NUMERIC=ru_RU.utf8 the thousand separator is the two byte nbsp/160.
228+ * **Workaround:** use single space characters instead as done in patch here: [27099](<http://bugs.freepascal.org/view.php?id=27099>)
237229
238230#### WinAPI function calls in FPC libs
239231
240-- Unit registry, TRegistry - this unit uses Windows Ansi functions and therefore you need to use UTF8ToWinCP, WinCPToUTF8. Formerly it needed UTF8ToSys.
241- - Note: fpc's TRegistry now defaults to using UnicodeString parameters, so for LCL applications (where default codepage = CP_UTF8) conversion should not be needed anymore.
242-- All Windows API Ansi function calls in FPC's libraries must be replaced with the "W" function version. This must be done in any case for the future UTF-16 support, thus there is no conflict of interest here.
243-- **TProcess** - under Windows TProcess FPC 3.0 only supports system codepage. Use either TProcessUTF8 of unit **utf8process** or use FPC trunk where it is fixed, see issue [29136](http://bugs.freepascal.org/view.php?id=29136)
232+ * Unit registry, TRegistry - this unit uses Windows Ansi functions and therefore you need to use UTF8ToWinCP, WinCPToUTF8. Formerly it needed UTF8ToSys.
233+ * Note: fpc's TRegistry now defaults to using UnicodeString parameters, so for LCL applications (where default codepage = CP_UTF8) conversion should not be needed anymore.
234+ * All Windows API Ansi function calls in FPC's libraries must be replaced with the "W" function version. This must be done in any case for the future UTF-16 support, thus there is no conflict of interest here.
235+ * **TProcess** \- under Windows TProcess FPC 3.0 only supports system codepage. Use either TProcessUTF8 of unit **utf8process** or use FPC trunk where it is fixed, see issue [29136](<http://bugs.freepascal.org/view.php?id=29136>)
244236
245-ToDo: List all related FPC bug tracker issues and patches that should be applied.
237+ToDo: List all related FPC bug tracker issues and patches that should be applied.
246238
247239## Future
248240
249-The goal of FPC project is to create a Delphi compatible UnicodeString (UTF-16) based solution, but it is not ready yet. It may take some time to be ready.
241+The goal of FPC project is to create a Delphi compatible UnicodeString (UTF-16) based solution, but it is not ready yet. It may take some time to be ready.
250242
251-This UTF-8 solution of LCL in its current form can be considered temporary. In the future, when FPC supports UnicodeString fully in RTL and FCL, Lazarus project will provide a solution for LCL that uses it. At the same time the goal is to preserve UTF-8 support although it may require changes to string types or something. Nobody know the details yet. We will tell when we know...
243+This UTF-8 solution of LCL in its current form can be considered temporary. In the future, when FPC supports UnicodeString fully in RTL and FCL, Lazarus project will provide a solution for LCL that uses it. At the same time the goal is to preserve UTF-8 support although it may require changes to string types or something. Nobody know the details yet. We will tell when we know...
252244
253-In essence LCL will probably have 2 versions, one for UTF-8 and one for UTF-16.
245+In essence LCL will probably have 2 versions, one for UTF-8 and one for UTF-16.
254246
255247## FAQ
256248
257249### What about Mode DelphiUnicode?
258250
259-The **{\$mode delphiunicode}** was added in FPC 2.7.1 and is like *{\$Mode Delphi}* with *{\$ModeSwitch UnicodeStrings}*. See the next question about *ModeSwitch UnicodeStrings*.
251+The **{$mode delphiunicode}** was added in FPC 2.7.1 and is like _{$Mode Delphi}_ with _{$ModeSwitch UnicodeStrings}_. See the next question about _ModeSwitch UnicodeStrings_.
260252
261253### What about ModeSwitch UnicodeStrings?
262254
263-The **{\$ModeSwitch UnicodeStrings}** was added in FPC 2.7.1 and defines "String" as "UnicodeString" (UTF-16), "Char" as "WideChar", "PChar" as "PWideChar" and so forth. This affects only the current unit. Other units including those used by this unit have their own "String" definition. Many RTL strings and types (e.g. TStringList) uses 8-bit strings, which require conversions from/to UnicodeString, which are added automatically by the compiler. The LCL uses UTF-8 strings. It is recommended to use UTF-8 sources with or without "-FcUTF8".
255+The **{$ModeSwitch UnicodeStrings}** was added in FPC 2.7.1 and defines "String" as "UnicodeString" (UTF-16), "Char" as "WideChar", "PChar" as "PWideChar" and so forth. This affects only the current unit. Other units including those used by this unit have their own "String" definition. Many RTL strings and types (e.g. TStringList) uses 8-bit strings, which require conversions from/to UnicodeString, which are added automatically by the compiler. The LCL uses UTF-8 strings. It is recommended to use UTF-8 sources with or without "-FcUTF8".
264256
265257### Why not use UTF8String in Lazarus?
266258
267-Short answer: Because the FCL does not use it.
268-
269-Long answer:
259+Short answer: Because the FCL does not use it.
270260
271-**UTF8String** is defined in the **system** unit as
261+Long answer:
272262
273-``` pascal
263+**UTF8String** is defined in the **system** unit as
264+```
274265UTF8String = type AnsiString(CP_UTF8);
275266```
276267
277-The compiler always assumes it has UTF-8 encoding (CP_UTF8), which is a multi byte encoding (i.e. 1-4 bytes per codepoint). Note that the \[\] operator accesses bytes, not characters nor codepoints. Same for UnicodeString, but words instead of bytes. On the other hand a **String** is assumed at compile time to have *DefaultSystemCodePage* (CP_ACP). *DefaultSystemCodePage* is defined at run time, so the compiler conservatively assumes that *String* and *UTF8String* have different encodings. When you assign or combine *String* and *UTF8String* the compiler inserts conversion code. Same for *ShortString* and *UTF8String*.
268+The compiler always assumes it has UTF-8 encoding (CP_UTF8), which is a multi byte encoding (i.e. 1-4 bytes per codepoint). Note that the [] operator accesses bytes, not characters nor codepoints. Same for UnicodeString, but words instead of bytes. On the other hand a **String** is assumed at compile time to have _DefaultSystemCodePage_ (CP_ACP). _DefaultSystemCodePage_ is defined at run time, so the compiler conservatively assumes that _String_ and _UTF8String_ have different encodings. When you assign or combine _String_ and _UTF8String_ the compiler inserts conversion code. Same for _ShortString_ and _UTF8String_.
278269
279-Lazarus uses the FCL, which uses *String*, so using *UTF8String* would add conversions. If *DefaultSystemCodePage* is not UTF-8 you lose characters. If it is UTF-8 then there is no point to use *UTF8String*.
270+Lazarus uses the FCL, which uses _String_ , so using _UTF8String_ would add conversions. If _DefaultSystemCodePage_ is not UTF-8 you lose characters. If it is UTF-8 then there is no point to use _UTF8String_.
280271
281-*UTF8String* becomes useful when eventually there is an UTF-16 FCL.
272+_UTF8String_ becomes useful when eventually there is an UTF-16 FCL.
282273
283274### Why does UTF8String show strange characters and String works
284275
285-For example:
286-
287-``` pascal
276+For example:
277+```
288278var s: UTF8String = 'ä'; // with default flags (i.e. no -Fc) this creates garbage
289- // even on a UTF-8 Linux system
279+ // even on a UTF-8 Linux system
290280```
291281
292-Question: Is it a bug? Answer: No, because it works as documented.
282+Question: Is it a bug? Answer: No, because it works as documented.
293283
294-FPC ignores the LANG variable to create on every system the same result. For historical reasons/Delphi compatibility it uses ISO-8859-1 as default. Lazarus prefers UTF-8.
284+FPC ignores the LANG variable to create on every system the same result. For historical reasons/Delphi compatibility it uses ISO-8859-1 as default. Lazarus prefers UTF-8.
295285
296-- UTF-8 sources work with *String*, because
297- 1. FPC does not add conversion code for normal String literals by default.
298- 2. The *source codepage* is equal to the *runtime codepage*. On Windows LazUTF8 sets it to CP_UTF8.
299-- UTF8String requires UTF-8 sources. Since FPC 3.0 for UTF8String you must tell the compiler, that the source is UTF8 (*-FcUTF8*, *{\$codepage UTF8}*, or save file as UTF-8 with BOM).
300- - Note: If you tell the compiler the source encoding UTF-8 it changes all non ASCII string literals of this unit to UTF-16, increasing the size of the binary, adding some overhead and PChar on literals require an explicit conversion. That's why Lazarus does not add it by default.
286+ * UTF-8 sources work with _String_ , because
287+ 1. FPC does not add conversion code for normal String literals by default.
288+ 2. The _source codepage_ is equal to the _runtime codepage_. On Windows LazUTF8 sets it to CP_UTF8.
289+ * UTF8String requires UTF-8 sources. Since FPC 3.0 for UTF8String you must tell the compiler, that the source is UTF8 (_-FcUTF8_ , _{$codepage UTF8}_ , or save file as UTF-8 with BOM).
290+ * Note: If you tell the compiler the source encoding UTF-8 it changes all non ASCII string literals of this unit to UTF-16, increasing the size of the binary, adding some overhead and PChar on literals require an explicit conversion. That's why Lazarus does not add it by default.
301291
302-### What happens when I use \$codepage utf8?
292+### What happens when I use $codepage utf8?
303293
304-FPC has very limited UTF-8 support. In fact, FPC only supports storing literals as either "default" encoded 8-bit strings or widestrings. So any non default codepage is converted to widestring, even if it is the system codepage. For example most Linux/Mac/BSD use UTF-8 as system codepage. Passing -Fcutf8 to the compiler will store the string literal as widestring.
294+FPC has very limited UTF-8 support. In fact, FPC only supports storing literals as either "default" encoded 8-bit strings or widestrings. So any non default codepage is converted to widestring, even if it is the system codepage. For example most Linux/Mac/BSD use UTF-8 as system codepage. Passing -Fcutf8 to the compiler will store the string literal as widestring.
305295
306-At run time the widestring literal is converted. When you assign the literal to an AnsiString the widestring literal is converted using the widestringmanager to the system encoding. The default widestringmanager under Unix simply converts the widechars to chars, destroying any non ASCII character. You must use a widestringmanager like the unit cwstring to get correct conversion. Unit LazUTF8 does that.
296+At run time the widestring literal is converted. When you assign the literal to an AnsiString the widestring literal is converted using the widestringmanager to the system encoding. The default widestringmanager under Unix simply converts the widechars to chars, destroying any non ASCII character. You must use a widestringmanager like the unit cwstring to get correct conversion. Unit LazUTF8 does that.
307297
308298## See also
309299
310-- Information about Unicode, codepages, string types and RTL in FPC. Note, that information is not entirely useful with Lazarus UTF-8 system, because from FPC's point of view it is a hack and changes the default codepage.
311-- [[FPC Unicode support]]
300+ * Information about Unicode, codepages, string types and RTL in FPC. Note, that information is not entirely useful with Lazarus UTF-8 system, because from FPC's point of view it is a hack and changes the default codepage.
301+ * [[FPC Unicode support]]
312302
313303---
314-*From the Free Pascal wiki, page "Unicode Support in Lazarus", as it stood on 2023-10-20 after a change by Bart, out of the wiki's dump of 2024-05-26. Copyright stays with the author who wrote it, according to the old wiki's [[Licensing]] page. If you have concerns about this material, [message the admin](/wiki-contact).*
304+*From the Free Pascal wiki, page "Unicode Support in Lazarus", as the Internet Archive kept it on 2025-06-24, last changed on 2024-07-26. Copyright stays with the author who wrote it, according to the old wiki's [[Licensing]] page. If you have concerns about this material, [message the admin](/wiki-contact).*

Back to history