Sign in Join
Wiki copytemporary

Unicode Support in Lazarus/ja: what changed

Revision 2 to 3. Removed lines have a minus sign; added lines have a plus sign.

OldNewLine
1+**日本語版メニュー**
2+---
3+[[Main Page/ja|メインページ]] \- [[Lazarus Documentation/ja|Lazarus Documentation日本語版]] \- [[Japanese Localization Notes|翻訳ノート]] \- [[Bug Tracking/ja|日本語障害情報]]
4+
5+
16# 導入
27
3-このページのカバーする範囲は、 FPC 3.0+ 以降の機能を用いた、 Lazarus での **プログラム** (コンソールやサーバなどの GUI 無し) および **アプリケーション** (LCL による GUI を用いた) におけるユニコードへの対応です。
8+このページのカバーする範囲は、 FPC 3.0+ 以降の機能を用いた、 Lazarus での **プログラム** (コンソールやサーバなどの GUI 無し) および **アプリケーション** (LCL による GUI を用いた) におけるユニコードへの対応です。
49
5-ソリューションは、クロス・プラットフォームであり、 UTF-8 エンコーディングを用います。これは、 Delphi の UTF-16 とは異なっていますが、いくつかのルールを守ることで、ソースコードレベルでの Delphi とのコンパチビリティを確保できます。
10+ソリューションは、クロス・プラットフォームであり、 UTF-8 エンコーディングを用います。これは、 Delphi の UTF-16 とは異なっていますが、いくつかのルールを守ることで、ソースコードレベルでの Delphi とのコンパチビリティを確保できます。
611
7-ユニコードへのサポートは、 FPC 3.0+ でコンパイルされた Lazarus 1.6.0 以降より自動的に可能となっています。
12+ユニコードへのサポートは、 FPC 3.0+ でコンパイルされた Lazarus 1.6.0 以降より自動的に可能となっています。
813
14+
15+
916## RTL と デフォルトのコードページ UTF-8
1017
11-RTL のデフォルトでは、システムのコードページを AnsiStrings (例えば FileExists や TStringList.LoadFromFile) で扱います。 Windows では、これはユニコードエンコーディングではないので、あなたは自身の言語グループから文字を用いることしかできません (ほとんどは 256 文字です)。 Linux と Mac OS X では、 UTF-8 は一般的なシステムのコードページですので、 RTL はデフォルトで CP_UTF8 を使用します。
18+RTL のデフォルトでは、システムのコードページを AnsiStrings (例えば FileExists や TStringList.LoadFromFile) で扱います。 Windows では、これはユニコードエンコーディングではないので、あなたは自身の言語グループから文字を用いることしかできません (ほとんどは 256 文字です)。 Linux と Mac OS X では、 UTF-8 は一般的なシステムのコードページですので、 RTL はデフォルトで CP_UTF8 を使用します。
1219
13-バージョン 3.0 以降の FPC は、 RTL のデフォルトシステムコードページを別のものに変える API を提供します。 Lazarus (実際にはその LazUtils パッケージ) はその API を利用しそれを UTF-8 (CP_UTF8)に変えます。 それは今では Windows ユーザーも UTF-8 文字列を RTL の中で使える事を意味します。
20+バージョン 3.0 以降の FPC は、 RTL のデフォルトシステムコードページを別のものに変える API を提供します。 Lazarus (実際にはその LazUtils パッケージ) はその API を利用しそれを UTF-8 (CP_UTF8)に変えます。 それは今では Windows ユーザーも UTF-8 文字列を RTL の中で使える事を意味します。
1421
22+
23+
1524# 使い方
1625
17-以下の簡単なルールがあります。:
26+以下の簡単なルールがあります。:
1827
19-- 一般的に、"String" 種別を UTF8String や UnicodeString の代わりに使う。
20-- Assign a constant always to a type String variable.
21-- Use type UnicodeString explicitly for API calls that need it.
28+ * 一般的に、"String" 種別を UTF8String や UnicodeString の代わりに使う。
29+ * Assign a constant always to a type String variable.
30+ * Use type UnicodeString explicitly for API calls that need it.
2231
23-These rules make most code already compatible with Delphi when using default project settings.
32+These rules make most code already compatible with Delphi when using default project settings.
2433
2534## UTF-8 を非 LCL プログラムで使うには
2635
27-非 LCL プロジェクトの依存物に **LazUtils** パッケージを追加してください。そして **LazUTF8** ユニットをメインプログラムファイルの uses セクションに追加してください。 それは出来るだけコードの最初に近い必要があります。 それはちょうど重要なメモリマネージャとスレッディング要素(例えば、 cmem や heaptrc, cthreads)の後です。
36+非 LCL プロジェクトの依存物に **LazUtils** パッケージを追加してください。そして **LazUTF8** ユニットをメインプログラムファイルの uses セクションに追加してください。 それは出来るだけコードの最初に近い必要があります。 それはちょうど重要なメモリマネージャとスレッディング要素(例えば、 cmem や heaptrc, cthreads)の後です。**CRT** ユニットを使用する場合は、**LazUTF8** の前に指定しなければならない。そうしないと、コンソール出力が文字化けしてしまう。
2837
2938## Windows APIを呼ぶ
3039
31-"W" バージョンのWindows　API関数が呼ばれるべきである。 Delphiは、API文字列へ/から、UnicodeString変数またはUnicodeString()の型キャストへのAPI呼び出しを代入しなければならないことを除外するようである。 これはDelphiまたは以前のLazarusコードと互換性がない。実際にはシステムのコードページを扱いコードをカプセル化し、可能な限り迅速にデータをUTF-8へ変換しなければならない。
40+"W" バージョンのWindows API関数が呼ばれるべきである。 Delphiは、API文字列へ/から、UnicodeString変数またはUnicodeString()の型キャストへのAPI呼び出しを代入しなければならないことを除外するようである。 これはDelphiまたは以前のLazarusコードと互換性がない。実際にはシステムのコードページを扱いコードをカプセル化し、可能な限り迅速にデータをUTF-8へ変換しなければならない。
3241
3342**RawByteStringと明示的な変換の双方を用いること**
34-
35-``` pascal
36- uses ... , LConvEncoding;
37- ...
38- var
39- StrIn: RawByteString;
40- StrOut: String;
41- ...
42- StrOut := CP1252ToUTF8(StrIn,true); // 直されたコードページ
43- // または
44- StrOut := WinCPToUTF8(StrIn,true); // この特定のコンピューターでシステムコードページを用いる
45-
46-''' ... または今存在している正しいコードページをセットする ...'''
47- var
48- StrIn, StrOut: String;
49- ...
50- SetCodePage(RawByteString(StrIn), 1252, false); // 1252で修正された (または、 Windows.GetACP())
5143```
44+uses ... , LConvEncoding;
45+ ...
46+ var
47+ StrIn: RawByteString;
48+ StrOut: String;
49+ ...
50+ StrOut := CP1252ToUTF8(StrIn,true); // 直されたコードページ
51+ // または
52+ StrOut := WinCPToUTF8(StrIn,true); // この特定のコンピューターでシステムコードページを用いる
53+
54+ ''' ... または今存在している正しいコードページをセットする ...'''
55+ var
56+ StrIn, StrOut: String;
57+ ...
58+ SetCodePage(RawByteString(StrIn), 1252, false); // 1252で修正された (または、 Windows.GetACP())
59+```
5260
53-注意: 文字列変数でテキストが存在しなければならない。空の文字列は実際にnilポインタであり、そのコードページを設定することはできない。
61+注意: 文字列変数でテキストが存在しなければならない。空の文字列は実際にnilポインタであり、そのコードページを設定することはできない。
5462
55-Windows.GetACP() Windows systemのコードページを返す.
63+Windows.GetACP() Windows systemのコードページを返す.
5664
5765# Code that depends very much on Windows codepage
5866
59-There is a "plan B" for code that depends very much on Windows system codepage, or must write to Windows console beyond the console codepage.\
60-See: [[Lazarus with FPC3.0 without UTF-8 mode]].\
61-Fortunately it is not needed often. In most cases it is easier to convert data to UTF-8.
67+There is a "plan B" for code that depends very much on Windows system codepage, or must write to Windows console beyond the console codepage.
68+See: [[Lazarus with FPC3.0 without UTF-8 mode]].
69+Fortunately it is not needed often. In most cases it is easier to convert data to UTF-8.
6270
6371## Writing to console
6472
65-Windows console output works **if** your characters belong to the console codepage. For example a box-drawing characters '╩' in codepage CP437 is one byte **\#202** and is often used like this:
66-
67-``` pascal
73+Windows console output works **if** your characters belong to the console codepage. For example a box-drawing characters '╩' in codepage CP437 is one byte **#202** and is often used like this:
74+```
6875write('╩');
6976```
7077
71-When you convert the code to UTF-8, for example by using Lazarus' source editor popup menu item **File Settings / Encoding / UTF-8** and clicking on the dialog button "Change file on disk", the *╩* becomes 3 bytes (#226#149#169), so the literal becomes a *string*. The procedures *write* and *writeln* convert the UTF-8 string to the current console codepage. So your console program now outputs the '╩' on Windows with any codepage (i.e. not only CP437) and it even works on Linux and Mac OS X. You can also use the '╩' in LCL strings, for instance Memo1.Lines.Add('╩');
78+When you convert the code to UTF-8, for example by using Lazarus' source editor popup menu item **File Settings / Encoding / UTF-8** and clicking on the dialog button "Change file on disk", the _╩_ becomes 3 bytes (#226#149#169), so the literal becomes a _string_. The procedures _write_ and _writeln_ convert the UTF-8 string to the current console codepage. So your console program now outputs the '╩' on Windows with any codepage (i.e. not only CP437) and it even works on Linux and Mac OS X. You can also use the '╩' in LCL strings, for instance Memo1.Lines.Add('╩');
7279
73-If your characters do **not** belong to the console codepage, things get trickier. Then you cannot use the new Unicode system at all.\
80+If your characters do **not** belong to the console codepage, things get trickier. Then you cannot use the new Unicode system at all.
7481See: [[Lazarus with FPC3.0 without UTF-8 mode|Lazarus with FPC3.0 without UTF-8 mode#Problem System encoding and Console encoding .28Windows.29]]
7582
7683# Unicode characters and codepoints in code
7784
78-See details for UTF-8 in [[UTF8 strings and characters]].
85+See details for UTF-8 in [[UTF8 strings and characters]].
7986
8087## CodePoint functions for encoding agnostic code
8188
82-LazUtils package has **unit LazUnicode** with special functions for dealing with codepoints, regardless of encoding. They use the UTF8...() functions from LazUTF8 when used in the UTF-8 mode, and UTF16...() functions from LazUTF16 when used in FPC's {\$ModeSwitch UnicodeStrings} or in Delphi (yes, Delphi is supported!).
83-
84-Currently the {\$ModeSwitch UnicodeStrings} can be tested by defining "UseUTF16". There is also a test program LazUnicodeTest in components/lazutils/test directory. It has 2 build modes, UTF8 and UTF16, for easy testing. The test program also supports Delphi, then the UTF-16 mode is used obviously.
89+LazUtils package has **unit LazUnicode** with special functions for dealing with codepoints, regardless of encoding. They use the UTF8...() functions from LazUTF8 when used in the UTF-8 mode, and UTF16...() functions from LazUTF16 when used in FPC's {$ModeSwitch UnicodeStrings} or in Delphi (yes, Delphi is supported!).
8590
86-LazUnicode allows one source code to work between :
91+Currently the {$ModeSwitch UnicodeStrings} can be tested by defining "UseUTF16". There is also a test program LazUnicodeTest in components/lazutils/test directory. It has 2 build modes, UTF8 and UTF16, for easy testing. The test program also supports Delphi, then the UTF-16 mode is used obviously.
8792
88-- Lazarus with its UTF-8 solution.
89-- Future FPC and Lazarus with Delphi compatible UTF-16 solution.
90-- Delphi, where String = UnicodeString.
93+LazUnicode allows one source code to work between :
9194
92-It provides these encoding agnostic functions:
95+ * Lazarus with its UTF-8 solution.
96+ * Future FPC and Lazarus with Delphi compatible UTF-16 solution.
97+ * Delphi, where String = UnicodeString.
9398
94-- CodePointCopy() - Like UTF8Copy()
95-- CodePointLength() - Like UTF8Length()
96-- CodePointPos() - Like UTF8Pos()
97-- CodePointSize() - Like UTF8CharacterLength()
98-- UnicodeToWinCP() - Like UTF8ToWinCP()
99-- WinCPToUnicode() - Like WinCPToUTF8()
99+It provides these encoding agnostic functions:
100100
101-It also provides an enumerator for CodePoints which the compiler uses for its for-in loop. As a result, regardless of encoding, this code works:
101+ * CodePointCopy() - Like UTF8Copy()
102+ * CodePointLength() - Like UTF8Length()
103+ * CodePointPos() - Like UTF8Pos()
104+ * CodePointSize() - Like UTF8CharacterLength()
105+ * UnicodeToWinCP() - Like UTF8ToWinCP()
106+ * WinCPToUnicode() - Like WinCPToUTF8()
102107
103-``` pascal
104- var s, ch: String;
105- ...
106- for ch in s do
107- writeln('ch=',ch);
108+It also provides an enumerator for CodePoints which the compiler uses for its for-in loop. As a result, regardless of encoding, this code works:
108109```
110+var s, ch: String;
111+ ...
112+ for ch in s do
113+ writeln('ch=',ch);
114+```
109115
110-Delphi does not provide similar functions for CodePoints for its UTF-16 solution. Practically most Delphi code treats UTF-16 as a fixed-width encoding which has lead to lots of broken UTF-16 code out there. It means using LazUnicode also for Delphi will improve code quality!
116+Delphi does not provide similar functions for CodePoints for its UTF-16 solution. Practically most Delphi code treats UTF-16 as a fixed-width encoding which has lead to lots of broken UTF-16 code out there. It means using LazUnicode also for Delphi will improve code quality!
111117
112-Both units LazUnicode and LazUTF16 are needed for Delphi usage.
118+Both units LazUnicode and LazUTF16 are needed for Delphi usage.
113119
120+
121+
114122# 文字列リテラル
115123
116-Sources should be saved in UTF-8 encoding. Lazarus creates such files by default. You can change the encoding of imported files via right click in source editor / File Settings / Encoding.
124+Sources should be saved in UTF-8 encoding. Lazarus creates such files by default. You can change the encoding of imported files via right click in source editor / File Settings / Encoding.
117125
118-Usually {\$codepage utf8} / -FcUTF8 is not needed. This is rather counter-intuitive because the meaning of that flag is to treat string literals as UTF-8. However the new UTF-8 mode switches the encoding at run-time, yet constants are evaluated at compile-time.
126+Usually {$codepage utf8} / -FcUTF8 is not needed. This is rather counter-intuitive because the meaning of that flag is to treat string literals as UTF-8. However the new UTF-8 mode switches the encoding at run-time, yet constants are evaluated at compile-time.
119127
120-So, without -FcUTF8 the compiler (wrongly) thinks the constant string is encoded with system code page. Then it sees a String variable with default encoding (which will be changed to UTF-8 at run-time but the compiler does not know it). Thus, same default encodings, no conversion needed, the compiler happily copies the characters and everything goes right, while actually it was fooled twice during the process.
128+So, without -FcUTF8 the compiler (wrongly) thinks the constant string is encoded with system code page. Then it sees a String variable with default encoding (which will be changed to UTF-8 at run-time but the compiler does not know it). Thus, same default encodings, no conversion needed, the compiler happily copies the characters and everything goes right, while actually it was fooled twice during the process.
121129
122130**Example:**
123131
124-As a rule of thumb, use "String" type and assigning literals works. Note: UTF-8 string can be composed from numbers, as done for s2.
125-
126-``` pascal
132+As a rule of thumb, use "String" type and assigning literals works. Note: UTF-8 string can be composed from numbers, as done for s2.
133+```
127134const s1: string = 'äй';
128-const s2: string = #$C3#$A4; // ä
135+ const s2: string = #$C3#$A4; // ä
129136```
130137
131-Assigning a string literal to other string types than plain "String" is more tricky. See the tables for what works and what doesn't.
138+Assigning a string literal to other string types than plain "String" is more tricky. See the tables for what works and what doesn't.
132139
133140## Assign string literals to different string types
134141
135-Here *working* means correct codepage and correct codepoints. Codepage 0 or codepage 65001 are both correct, they mean UTF-8.\
136-
137-### Without {\$codepage utf8} or compilerswitch -FcUTF8
142+Here _working_ means correct codepage and correct codepoints. Codepage 0 or codepage 65001 are both correct, they mean UTF-8.
138143
139-| String Type, UTF-8 Source | Example | Const (in Source) | Assigned to String | Assigned to UTF8String | Assigned to UnicodeString | Assigned to CP1252String | Assigned to RawByteString | Assigned to ShortString | Assigned to PChar |
140-|----|----|----|----|----|----|----|----|----|----|
141-| const | const s = 'äöü'; | working | working | wrong | wrong | wrong | working | working | working |
142-| String | const s: String = 'äöü'; | working | working | working | working | working | working | working | working |
143-| ShortString | const s: String\[15\] = 'äöü'; | working | working | working | working | wrong encoded | working | working | not available |
144-| UTF8String | const s: UTF8String = 'äöü'; | wrong | wrong | wrong | wrong | wrong | wrong | wrong | wrong |
145-| UnicodeString | const s: UnicodeString = 'äöü'; | wrong | wrong | wrong | wrong | wrong | wrong | wrong | wrong |
146-| String with declared code page | type CP1252String = type AnsiString(1252); | wrong | wrong | wrong | wrong | wrong | wrong | wrong | wrong |
147-| RawbyteString | const s: RawbyteString = 'äöü'; | working | working | working | working | to codepage 0 changed | working | working | working |
148-| PChar | const c: PChar = 'äöü'; | working | working | working | working | wrong | working | working | working |
144+### Without {$codepage utf8} or compilerswitch -FcUTF8
149145
150-### With {\$codepage utf8} or compilerswitch -FcUTF8
146+String Type, UTF-8 Source | Example | Const (in Source) | Assigned to String | Assigned to UTF8String | Assigned to UnicodeString | Assigned to CP1252String | Assigned to RawByteString | Assigned to ShortString | Assigned to PChar
147+---|---|---|---|---|---|---|---|---|---
148+const | const s = 'äöü'; | working | working | wrong | wrong | wrong | working | working | working
149+String | const s: String = 'äöü'; | working | working | working | working | working | working | working | working
150+ShortString | const s: String[15] = 'äöü'; | working | working | working | working | wrong encoded | working | working | not available
151+UTF8String | const s: UTF8String = 'äöü'; | wrong | wrong | wrong | wrong | wrong | wrong | wrong | wrong
152+UnicodeString | const s: UnicodeString = 'äöü'; | wrong | wrong | wrong | wrong | wrong | wrong | wrong | wrong
153+String with declared code page | type CP1252String = type AnsiString(1252); | wrong | wrong | wrong | wrong | wrong | wrong | wrong | wrong
154+RawbyteString | const s: RawbyteString = 'äöü'; | working | working | working | working | to codepage 0 changed | working | working | working
155+PChar | const c: PChar = 'äöü'; | working | working | working | working | wrong | working | working | working
156+
157+### With {$codepage utf8} or compilerswitch -FcUTF8
151158
152-| String Type, UTF-8 Source | Example | Const (in Source) | Assigned to String | Assigned to UTF8String | Assigned to UnicodeString | Assigned to CP1252String | Assigned to RawByteString | Assigned to ShortString | Assigned to PChar |
153-|----|----|----|----|----|----|----|----|----|----|
154-| const | const s = 'äöü'; | UTF-16 encoded | working | working | working | working | working | working | working |
155-| String | const s: String = 'äöü'; | working | working | working | working | working | working | working | working |
156-| ShortString | const s: String\[15\] = 'äöü'; | wrong | wrong | wrong | wrong | wrong | wrong | wrong | not available |
157-| UTF8String | const s: UTF8String = 'äöü'; | working | working | working | working | working | working | working | working |
158-| UnicodeString | const s: UnicodeString = 'äöü'; | working | working | working | working | working | working | working | wrong |
159-| String with declared code page | type CP1252String = type AnsiString(1252); | working | working | working | working | working | working | wrong | wrong |
160-| RawbyteString | const s: RawbyteString = 'äöü'; | working | working | working | working | to codepage 0 changed | working | working | working |
161-| PChar | const c: PChar = 'äöü'; | wrong | wrong | wrong | wrong | wrong | wrong | wrong | wrong |
159+String Type, UTF-8 Source | Example | Const (in Source) | Assigned to String | Assigned to UTF8String | Assigned to UnicodeString | Assigned to CP1252String | Assigned to RawByteString | Assigned to ShortString | Assigned to PChar
160+---|---|---|---|---|---|---|---|---|---
161+const | const s = 'äöü'; | UTF-16 encoded | working | working | working | working | working | working | working
162+String | const s: String = 'äöü'; | working | working | working | working | working | working | working | working
163+ShortString | const s: String[15] = 'äöü'; | wrong | wrong | wrong | wrong | wrong | wrong | wrong | not available
164+UTF8String | const s: UTF8String = 'äöü'; | working | working | working | working | working | working | working | working
165+UnicodeString | const s: UnicodeString = 'äöü'; | working | working | working | working | working | working | working | wrong
166+String with declared code page | type CP1252String = type AnsiString(1252); | working | working | working | working | working | working | wrong | wrong
167+RawbyteString | const s: RawbyteString = 'äöü'; | working | working | working | working | to codepage 0 changed | working | working | working
168+PChar | const c: PChar = 'äöü'; | wrong | wrong | wrong | wrong | wrong | wrong | wrong | wrong
169+
170+Remember, assignment between variables of different string types always works thanks to their dynamic encoding in FPC 3+. The data is converted automatically when needed.
171+Only string literals are a challenge.
162172
163-Remember, assignment between variables of different string types always works thanks to their dynamic encoding in FPC 3+. The data is converted automatically when needed.\
164-Only string literals are a challenge.
173+
165174
166175# Coming from older Lazarus + LCL versions
167176
168-Earlier (before 1.6.0) LCL supported Unicode with dedicated UTF8 functions. The code was not at all compatible with Delphi.
177+Earlier (before 1.6.0) LCL supported Unicode with dedicated UTF8 functions. The code was not at all compatible with Delphi.
169178
170-Now many old LCL applications continue to work without changes. However it makes sense to clean the code to make it simpler and more Delphi compatible. Code that reads/writes data with Windows system codepage encoding breaks and must be changed. (See #Reading / writing text file with Windows codepage).
179+Now many old LCL applications continue to work without changes. However it makes sense to clean the code to make it simpler and more Delphi compatible. Code that reads/writes data with Windows system codepage encoding breaks and must be changed. (See #Reading / writing text file with Windows codepage).
171180
172-Explicit conversion functions are only needed when calling Windows Ansi functions. Otherwise FPC takes care of converting encodings automatically. Empty conversion functions are provided to make your old code compile.
181+Explicit conversion functions are only needed when calling Windows Ansi functions. Otherwise FPC takes care of converting encodings automatically. Empty conversion functions are provided to make your old code compile.
173182
174-- UTF8Decode, UTF8Encode - Almost all can be removed.
175-- UTF8ToAnsi, AnsiToUTF8 - Almost all can be removed.
176-- UTF8ToSys, SysToUTF8 - All can be removed. They are now dummy no-ops and only return their parameter.
183+ * UTF8Decode, UTF8Encode - Almost all can be removed.
184+ * UTF8ToAnsi, AnsiToUTF8 - Almost all can be removed.
185+ * UTF8ToSys, SysToUTF8 - All can be removed. They are now dummy no-ops and only return their parameter.
177186
178-File functions in RTL now take care of file name encoding. All (?) file name related ...UTF8() functions can be replaced with the Delphi compatible function without UTF8 suffix. For example FileExistsUTF8 can be replaced with FileExists.
187+File functions in RTL now take care of file name encoding. All (?) file name related ...UTF8() functions can be replaced with the Delphi compatible function without UTF8 suffix. For example FileExistsUTF8 can be replaced with FileExists.
179188
180-Most UTF8...() string functions can be replaced with the Delphi compatible Ansi...() functions. For example UTF8UpperCase() -\> AnsiUpperCase().
189+Most UTF8...() string functions can be replaced with the Delphi compatible Ansi...() functions. For example UTF8UpperCase() -> AnsiUpperCase().
181190
182-Now Unicode works in non-GUI programs, too. It only requires a dependency for LazUtils and placing LazUTF8 unit into the uses section of main program file.
191+Now Unicode works in non-GUI programs, too. It only requires a dependency for LazUtils and placing LazUTF8 unit into the uses section of main program file.
183192
184193For historical reference, this was the old Unicode support in LCL: [[Old LCL Unicode Support]]
185194
186195# Technical implementation
187196
188-What actually happens in the Unicode system? These 2 FPC functions are called in an early initialization section, setting the default String encoding in FPC to UTF-8 :
189-
190-```text
191- SetMultiByteConversionCodePage(CP_UTF8);
192- SetMultiByteRTLFileSystemCodePage(CP_UTF8);
197+What actually happens in the Unicode system? These 2 FPC functions are called in an early initialization section, setting the default String encoding in FPC to UTF-8 :
193198```
199+SetMultiByteConversionCodePage(CP_UTF8);
200+ SetMultiByteRTLFileSystemCodePage(CP_UTF8);
201+```
194202
195-Under Windows the UTF8...() functions in LazUTF8 (LazUtils) are set as backends for RTL's Ansi...() string functions. Thus those functions work in a Delphi compatible way.
203+Under Windows the UTF8...() functions in LazUTF8 (LazUtils) are set as backends for RTL's Ansi...() string functions. Thus those functions work in a Delphi compatible way.
196204
205+
206+
197207# Open issues
198208
199-- **TFormatSettings** char (bug [27086](http://bugs.freepascal.org/view.php?id=27086)): for example: **ThousandSeparator**, **DecimalSeparator**, **DateSeparator**, **TimeSeparator**, **ListSeparator**. These should be replaced with string to support UTF-8. For example under Linux with LC_NUMERIC=ru_RU.utf8 the thousand separator is the two byte nbsp/160.
200- - **Workaround:** use single space characters instead as done in patch here: [27099](http://bugs.freepascal.org/view.php?id=27099)
209+ * **TFormatSettings** char (bug [27086](<http://bugs.freepascal.org/view.php?id=27086>)): for example: **ThousandSeparator** , **DecimalSeparator** , **DateSeparator** , **TimeSeparator** , **ListSeparator**. These should be replaced with string to support UTF-8. For example under Linux with LC_NUMERIC=ru_RU.utf8 the thousand separator is the two byte nbsp/160.
210+ * **Workaround:** use single space characters instead as done in patch here: [27099](<http://bugs.freepascal.org/view.php?id=27099>)
201211
202212### WinAPI function calls in FPC libs
203213
204-- Unit registry, TRegistry - this unit uses Windows Ansi functions and therefore you need to use UTF8ToWinCP, WinCPToUTF8. Formerly it needed UTF8ToSys.
205-- All Windows API Ansi function calls in FPC's libraries must be replaced with the "W" function version. This must be done in any case for the future UTF-16 support, thus there is no conflict of interest here.
206-- **TProcess** - under Windows TProcess FPC 3.0 only supports system codepage. Use either TProcessUTF8 of unit **utf8process** or use FPC trunk where it is fixed, see issue [29136](http://bugs.freepascal.org/view.php?id=29136)
214+ * Unit registry, TRegistry - this unit uses Windows Ansi functions and therefore you need to use UTF8ToWinCP, WinCPToUTF8. Formerly it needed UTF8ToSys.
215+ * All Windows API Ansi function calls in FPC's libraries must be replaced with the "W" function version. This must be done in any case for the future UTF-16 support, thus there is no conflict of interest here.
216+ * **TProcess** \- under Windows TProcess FPC 3.0 only supports system codepage. Use either TProcessUTF8 of unit **utf8process** or use FPC trunk where it is fixed, see issue [29136](<http://bugs.freepascal.org/view.php?id=29136>)
207217
208-ToDo: List all related FPC bug tracker issues and patches that should be applied.
218+ToDo: List all related FPC bug tracker issues and patches that should be applied.
209219
220+
221+
210222# Future
211223
212-The goal of FPC project is to create a Delphi compatible UnicodeString (UTF-16) based solution, but it is not ready yet. It may take some time to be ready.
224+The goal of FPC project is to create a Delphi compatible UnicodeString (UTF-16) based solution, but it is not ready yet. It may take some time to be ready.
213225
214-This UTF-8 solution of LCL in its current form can be considered temporary. In the future, when FPC supports UnicodeString fully in RTL and FCL, Lazarus project will provide a solution for LCL that uses it. At the same time the goal is to preserve UTF-8 support although it may require changes to string types or something. Nobody know the details yet. We will tell when we know...
226+This UTF-8 solution of LCL in its current form can be considered temporary. In the future, when FPC supports UnicodeString fully in RTL and FCL, Lazarus project will provide a solution for LCL that uses it. At the same time the goal is to preserve UTF-8 support although it may require changes to string types or something. Nobody know the details yet. We will tell when we know...
215227
216-In essence LCL will probably have 2 versions, one for UTF-8 and one for UTF-16.
228+In essence LCL will probably have 2 versions, one for UTF-8 and one for UTF-16.
217229
230+
231+
218232# FAQ
219233
220234## What about Mode DelphiUnicode?
221235
222-The **{\$mode delphiunicode}** was added in FPC 2.7.1 and is like *{\$Mode Delphi}* with *{\$ModeSwitch UnicodeStrings}*. See the next question about *ModeSwitch UnicodeStrings*.
236+The **{$mode delphiunicode}** was added in FPC 2.7.1 and is like _{$Mode Delphi}_ with _{$ModeSwitch UnicodeStrings}_. See the next question about _ModeSwitch UnicodeStrings_.
223237
224238## What about ModeSwitch UnicodeStrings?
225239
226-The **{\$ModeSwitch UnicodeStrings}** was added in FPC 2.7.1 and defines "String" as "UnicodeString" (UTF-16), "Char" as "WideChar", "PChar" as "PWideChar" and so forth. This affects only the current unit. Other units including those used by this unit have their own "String" definition. Many RTL strings and types (e.g. TStringList) uses 8-bit strings, which require conversions from/to UnicodeString, which are added automatically by the compiler. The LCL uses UTF-8 strings. It is recommended to use UTF-8 sources with or without "-FcUTF8".
240+The **{$ModeSwitch UnicodeStrings}** was added in FPC 2.7.1 and defines "String" as "UnicodeString" (UTF-16), "Char" as "WideChar", "PChar" as "PWideChar" and so forth. This affects only the current unit. Other units including those used by this unit have their own "String" definition. Many RTL strings and types (e.g. TStringList) uses 8-bit strings, which require conversions from/to UnicodeString, which are added automatically by the compiler. The LCL uses UTF-8 strings. It is recommended to use UTF-8 sources with or without "-FcUTF8".
227241
228242## Why not use UTF8String in Lazarus?
229243
230-Short answer: Because the FCL does not use it.
231-
232-Long answer: **UTF8String** is defined in the **system** unit as
244+Short answer: Because the FCL does not use it.
233245
234-``` pascal
246+Long answer: **UTF8String** is defined in the **system** unit as
247+```
235248UTF8String = type AnsiString(CP_UTF8);
236249```
237250
238-The compiler always assumes it has UTF-8 encoding (CP_UTF8), which is a multi byte encoding (i.e. 1-4 bytes per codepoint). Note that the \[\] operator accesses bytes, not characters nor codepoints. Same for UnicodeString, but words instead of bytes. On the other hand a **String** is assumed at compile time to have *DefaultSystemCodePage* (CP_ACP). *DefaultSystemCodePage* is defined at run time, so the compiler conservatively assumes that *String* and *UTF8String* have different encodings. When you assign or combine *String* and *UTF8String* the compiler inserts conversion code. Same for *ShortString* and *UTF8String*.
251+The compiler always assumes it has UTF-8 encoding (CP_UTF8), which is a multi byte encoding (i.e. 1-4 bytes per codepoint). Note that the [] operator accesses bytes, not characters nor codepoints. Same for UnicodeString, but words instead of bytes. On the other hand a **String** is assumed at compile time to have _DefaultSystemCodePage_ (CP_ACP). _DefaultSystemCodePage_ is defined at run time, so the compiler conservatively assumes that _String_ and _UTF8String_ have different encodings. When you assign or combine _String_ and _UTF8String_ the compiler inserts conversion code. Same for _ShortString_ and _UTF8String_.
239252
240-Lazarus uses the FCL, which uses *String*, so using *UTF8String* would add conversions. If *DefaultSystemCodePage* is not UTF-8 you lose characters. If it is UTF-8 then there is no point to use *UTF8String*.
253+Lazarus uses the FCL, which uses _String_ , so using _UTF8String_ would add conversions. If _DefaultSystemCodePage_ is not UTF-8 you lose characters. If it is UTF-8 then there is no point to use _UTF8String_.
241254
242-*UTF8String* becomes useful when eventually there is an UTF-16 FCL.
255+_UTF8String_ becomes useful when eventually there is an UTF-16 FCL.
243256
244257## Why does UTF8String show strange characters and String works
245258
246-For example:
247-
248-``` pascal
259+For example:
260+```
249261var s: UTF8String = 'ä'; // with default flags (i.e. no -Fc) this creates garbage
250- // even on a UTF-8 Linux system
262+ // even on a UTF-8 Linux system
251263```
252264
253-Question: Is it a bug? Answer: No, because it works as documented.
265+Question: Is it a bug? Answer: No, because it works as documented.
254266
255-FPC ignores the LANG variable to create on every system the same result. For historical reasons/Delphi compatibility it uses ISO-8859-1 as default. Lazarus prefers UTF-8.
267+FPC ignores the LANG variable to create on every system the same result. For historical reasons/Delphi compatibility it uses ISO-8859-1 as default. Lazarus prefers UTF-8.
256268
257-- UTF-8 sources work with *String*, because
258- 1. FPC does not add conversion code for normal String literals by default.
259- 2. The *source codepage* is equal to the *runtime codepage*. On Windows LazUTF8 sets it to CP_UTF8.
260-- UTF8String requires UTF-8 sources. Since FPC 3.0 for UTF8String you must tell the compiler, that the source is UTF8 (*-FcUTF8*, *{\$codepage UTF8}*, or save file as UTF-8 with BOM).
261- - Note: If you tell the compiler the source encoding UTF-8 it changes all non ASCII string literals of this unit to UTF-16, increasing the size of the binary, adding some overhead and PChar on literals require an explicit conversion. That's why Lazarus does not add it by default.
269+ * UTF-8 sources work with _String_ , because
270+ 1. FPC does not add conversion code for normal String literals by default.
271+ 2. The _source codepage_ is equal to the _runtime codepage_. On Windows LazUTF8 sets it to CP_UTF8.
272+ * UTF8String requires UTF-8 sources. Since FPC 3.0 for UTF8String you must tell the compiler, that the source is UTF8 (_-FcUTF8_ , _{$codepage UTF8}_ , or save file as UTF-8 with BOM).
273+ * Note: If you tell the compiler the source encoding UTF-8 it changes all non ASCII string literals of this unit to UTF-16, increasing the size of the binary, adding some overhead and PChar on literals require an explicit conversion. That's why Lazarus does not add it by default.
262274
263-## What happens when I use \$codepage utf8?
275+## What happens when I use $codepage utf8?
264276
265-FPC has very limited UTF-8 support. In fact, FPC only supports storing literals as either "default" encoded 8-bit strings or widestrings. So any non default codepage is converted to widestring, even if it is the system codepage. For example most Linux/Mac/BSD use UTF-8 as system codepage. Passing -Fcutf8 to the compiler will store the string literal as widestring.
277+FPC has very limited UTF-8 support. In fact, FPC only supports storing literals as either "default" encoded 8-bit strings or widestrings. So any non default codepage is converted to widestring, even if it is the system codepage. For example most Linux/Mac/BSD use UTF-8 as system codepage. Passing -Fcutf8 to the compiler will store the string literal as widestring.
266278
267-At run time the widestring literal is converted. When you assign the literal to an AnsiString the widestring literal is converted using the widestringmanager to the system encoding. The default widestringmanager under Unix simply converts the widechars to chars, destroying any non ASCII character. You must use a widestringmanager like the unit cwstring to get correct conversion. Unit LazUTF8 does that.
279+At run time the widestring literal is converted. When you assign the literal to an AnsiString the widestring literal is converted using the widestringmanager to the system encoding. The default widestringmanager under Unix simply converts the widechars to chars, destroying any non ASCII character. You must use a widestringmanager like the unit cwstring to get correct conversion. Unit LazUTF8 does that.
268280
281+
282+
269283# 関連情報
270284
271-- Unicode や コードページ、文字列の種別や FPC での RTL についての情報は以下にあります。注） この情報はすべてが Lazarus の UTF-8 システムで有効ではありません。なぜなら、 FPC の観点からは、これは ハック であり、デフォルトのコードページを変更することになるからです。\
272-- [[FPC Unicode support/ja]]
285+ * Unicode や コードページ、文字列の種別や FPC での RTL についての情報は以下にあります。注） この情報はすべてが Lazarus の UTF-8 システムで有効ではありません。なぜなら、 FPC の観点からは、これは ハック であり、デフォルトのコードページを変更することになるからです。
273286
287+ * [[FPC Unicode support/ja]]
288+
274289---
275-*From the Free Pascal wiki, page "Unicode Support in Lazarus/ja", as it stood on 2024-03-10 after a change by Ariben, out of the wiki's dump of 2024-05-26. Copyright stays with the author who wrote it, according to the old wiki's [[Licensing]] page. If you have concerns about this material, [message the admin](/wiki-contact).*
290+*From the Free Pascal wiki, page "Unicode Support in Lazarus/ja", as the Internet Archive kept it on 2025-03-25, last changed on 2024-07-26. Copyright stays with the author who wrote it, according to the old wiki's [[Licensing]] page. If you have concerns about this material, [message the admin](/wiki-contact).*

Back to history