Unicode Support in Lazarus/ja: what changed

Revision 1 to 2. Removed lines have a minus sign; added lines have a plus sign.

Old	New		Line
1	1		# 導入
2	2
3	3		このページのカバーする範囲は、 FPC 3.0+ 以降の機能を用いた、 Lazarus でのプログラム (コンソールやサーバなどの GUI 無し) およびアプリケーション (LCL による GUI を用いた) におけるユニコードへの対応です。
4	4
5	5		ソリューションは、クロス・プラットフォームであり、 UTF-8 エンコーディングを用います。これは、 Delphi の UTF-16 とは異なっていますが、いくつかのルールを守ることで、ソースコードレベルでの Delphi とのコンパチビリティを確保できます。
6	6
7	7		ユニコードへのサポートは、 FPC 3.0+ でコンパイルされた Lazarus 1.6.0 以降より自動的に可能となっています。
8	8
9	9		## RTL とデフォルトのコードページ UTF-8
10	10
11	11		RTL のデフォルトでは、システムのコードページを AnsiStrings (例えば FileExists や TStringList.LoadFromFile) で扱います。 Windows では、これはユニコードエンコーディングではないので、あなたは自身の言語グループから文字を用いることしかできません (ほとんどは 256 文字です)。 Linux と Mac OS X では、 UTF-8 は一般的なシステムのコードページですので、 RTL はデフォルトで CP_UTF8 を使用します。
12	12
13	13		バージョン 3.0 以降の FPC は、 RTL のデフォルトシステムコードページを別のものに変える API を提供します。 Lazarus (実際にはその LazUtils パッケージ) はその API を利用しそれを UTF-8 (CP_UTF8)に変えます。それは今では Windows ユーザーも UTF-8 文字列を RTL の中で使える事を意味します。
14	14
15	15		# 使い方
16	16
17	17		以下の簡単なルールがあります。:
18	18
19	19		- 一般的に、"String" 種別を UTF8String や UnicodeString の代わりに使う。
20	20		- Assign a constant always to a type String variable.
21	21		- Use type UnicodeString explicitly for API calls that need it.
22	22
23	23		These rules make most code already compatible with Delphi when using default project settings.
24	24
25	25		## UTF-8 を非 LCL プログラムで使うには
26	26
27	27		非 LCL プロジェクトの依存物に LazUtils パッケージを追加してください。そして LazUTF8 ユニットをメインプログラムファイルの uses セクションに追加してください。それは出来るだけコードの最初に近い必要があります。それはちょうど重要なメモリマネージャとスレッディング要素(例えば、 cmem や heaptrc, cthreads)の後です。
28	28
29	29		## Windows APIを呼ぶ
30	30
31	31		"W" バージョンのWindows　API関数が呼ばれるべきである。 Delphiは、API文字列へ/から、UnicodeString変数またはUnicodeString()の型キャストへのAPI呼び出しを代入しなければならないことを除外するようである。これはDelphiまたは以前のLazarusコードと互換性がない。実際にはシステムのコードページを扱いコードをカプセル化し、可能な限り迅速にデータをUTF-8へ変換しなければならない。
32	32
33	33		RawByteStringと明示的な変換の双方を用いること
34	34
35	35		``` pascal
36	36		uses ... , LConvEncoding;
37	37		...
38	38		var
39	39		StrIn: RawByteString;
40	40		StrOut: String;
41	41		...
42	42		StrOut := CP1252ToUTF8(StrIn,true); // 直されたコードページ
43	43		// または
44	44		StrOut := WinCPToUTF8(StrIn,true); // この特定のコンピューターでシステムコードページを用いる
45	45
46	46		''' ... または今存在している正しいコードページをセットする ...'''
47	47		var
48	48		StrIn, StrOut: String;
49	49		...
50	50		SetCodePage(RawByteString(StrIn), 1252, false); // 1252で修正された (または、 Windows.GetACP())
51	51		```
52	52
53	53		注意: 文字列変数でテキストが存在しなければならない。空の文字列は実際にnilポインタであり、そのコードページを設定することはできない。
54	54
55	55		Windows.GetACP() Windows systemのコードページを返す.
56	56
57	57		# Code that depends very much on Windows codepage
58	58
59	59		There is a "plan B" for code that depends very much on Windows system codepage, or must write to Windows console beyond the console codepage.\
60	60		See: [[Lazarus with FPC3.0 without UTF-8 mode]].\
61	61		Fortunately it is not needed often. In most cases it is easier to convert data to UTF-8.
62	62
63	63		## Writing to console
64	64
65	65		Windows console output works if your characters belong to the console codepage. For example a box-drawing characters '╩' in codepage CP437 is one byte \#202 and is often used like this:
66	66
67	67		``` pascal
68	68		write('╩');
69	69		```
70	70
71	71		When you convert the code to UTF-8, for example by using Lazarus' source editor popup menu item File Settings / Encoding / UTF-8 and clicking on the dialog button "Change file on disk", the ╩ becomes 3 bytes (#226#149#169), so the literal becomes a string. The procedures write and writeln convert the UTF-8 string to the current console codepage. So your console program now outputs the '╩' on Windows with any codepage (i.e. not only CP437) and it even works on Linux and Mac OS X. You can also use the '╩' in LCL strings, for instance Memo1.Lines.Add('╩');
72	72
73	73		If your characters do not belong to the console codepage, things get trickier. Then you cannot use the new Unicode system at all.\
74	74		See: [[Lazarus with FPC3.0 without UTF-8 mode\|Lazarus with FPC3.0 without UTF-8 mode#Problem System encoding and Console encoding .28Windows.29]]
75	75
76	76		# Unicode characters and codepoints in code
77	77
78	78		See details for UTF-8 in [[UTF8 strings and characters]].
79	79
80	80		## CodePoint functions for encoding agnostic code
81	81
82	82		LazUtils package has unit LazUnicode with special functions for dealing with codepoints, regardless of encoding. They use the UTF8...() functions from LazUTF8 when used in the UTF-8 mode, and UTF16...() functions from LazUTF16 when used in FPC's {\$ModeSwitch UnicodeStrings} or in Delphi (yes, Delphi is supported!).
83	83
84	84		Currently the {\$ModeSwitch UnicodeStrings} can be tested by defining "UseUTF16". There is also a test program LazUnicodeTest in components/lazutils/test directory. It has 2 build modes, UTF8 and UTF16, for easy testing. The test program also supports Delphi, then the UTF-16 mode is used obviously.
85	85
86	86		LazUnicode allows one source code to work between :
87	87
88	88		- Lazarus with its UTF-8 solution.
89	89		- Future FPC and Lazarus with Delphi compatible UTF-16 solution.
90	90		- Delphi, where String = UnicodeString.
91	91
92	92		It provides these encoding agnostic functions:
93	93
94	94		- CodePointCopy() - Like UTF8Copy()
95	95		- CodePointLength() - Like UTF8Length()
96	96		- CodePointPos() - Like UTF8Pos()
97	97		- CodePointSize() - Like UTF8CharacterLength()
98	98		- UnicodeToWinCP() - Like UTF8ToWinCP()
99	99		- WinCPToUnicode() - Like WinCPToUTF8()
100	100
101	101		It also provides an enumerator for CodePoints which the compiler uses for its for-in loop. As a result, regardless of encoding, this code works:
102	102
103	103		``` pascal
104	104		var s, ch: String;
105	105		...
106	106		for ch in s do
107	107		writeln('ch=',ch);
108	108		```
109	109
110	110		Delphi does not provide similar functions for CodePoints for its UTF-16 solution. Practically most Delphi code treats UTF-16 as a fixed-width encoding which has lead to lots of broken UTF-16 code out there. It means using LazUnicode also for Delphi will improve code quality!
111	111
112	112		Both units LazUnicode and LazUTF16 are needed for Delphi usage.
113	113
114	114		# 文字列リテラル
115	115
116	116		Sources should be saved in UTF-8 encoding. Lazarus creates such files by default. You can change the encoding of imported files via right click in source editor / File Settings / Encoding.
117	117
118	118		Usually {\$codepage utf8} / -FcUTF8 is not needed. This is rather counter-intuitive because the meaning of that flag is to treat string literals as UTF-8. However the new UTF-8 mode switches the encoding at run-time, yet constants are evaluated at compile-time.
119	119
120	120		So, without -FcUTF8 the compiler (wrongly) thinks the constant string is encoded with system code page. Then it sees a String variable with default encoding (which will be changed to UTF-8 at run-time but the compiler does not know it). Thus, same default encodings, no conversion needed, the compiler happily copies the characters and everything goes right, while actually it was fooled twice during the process.
121	121
122	122		Example:
123	123
124	124		As a rule of thumb, use "String" type and assigning literals works. Note: UTF-8 string can be composed from numbers, as done for s2.
125	125
126	126		``` pascal
127	127		const s1: string = 'äй';
128	128		const s2: string = #$C3#$A4; // ä
129	129		```
130	130
131	131		Assigning a string literal to other string types than plain "String" is more tricky. See the tables for what works and what doesn't.
132	132
133	133		## Assign string literals to different string types
134	134
135	135		Here working means correct codepage and correct codepoints. Codepage 0 or codepage 65001 are both correct, they mean UTF-8.\
136	136
137	137		### Without {\$codepage utf8} or compilerswitch -FcUTF8
138	138
139	139		\| String Type, UTF-8 Source \| Example \| Const (in Source) \| Assigned to String \| Assigned to UTF8String \| Assigned to UnicodeString \| Assigned to CP1252String \| Assigned to RawByteString \| Assigned to ShortString \| Assigned to PChar \|
140	140		\|----\|----\|----\|----\|----\|----\|----\|----\|----\|----\|
141	141		\| const \| const s = 'äöü'; \| working \| working \| wrong \| wrong \| wrong \| working \| working \| working \|
142	142		\| String \| const s: String = 'äöü'; \| working \| working \| working \| working \| working \| working \| working \| working \|
143	143		\| ShortString \| const s: String\[15\] = 'äöü'; \| working \| working \| working \| working \| wrong encoded \| working \| working \| not available \|
144	144		\| UTF8String \| const s: UTF8String = 'äöü'; \| wrong \| wrong \| wrong \| wrong \| wrong \| wrong \| wrong \| wrong \|
145	145		\| UnicodeString \| const s: UnicodeString = 'äöü'; \| wrong \| wrong \| wrong \| wrong \| wrong \| wrong \| wrong \| wrong \|
146	146		\| String with declared code page \| type CP1252String = type AnsiString(1252); \| wrong \| wrong \| wrong \| wrong \| wrong \| wrong \| wrong \| wrong \|
147	147		\| RawbyteString \| const s: RawbyteString = 'äöü'; \| working \| working \| working \| working \| to codepage 0 changed \| working \| working \| working \|
148	148		\| PChar \| const c: PChar = 'äöü'; \| working \| working \| working \| working \| wrong \| working \| working \| working \|
149	149
150	150		### With {\$codepage utf8} or compilerswitch -FcUTF8
151	151
152	152		\| String Type, UTF-8 Source \| Example \| Const (in Source) \| Assigned to String \| Assigned to UTF8String \| Assigned to UnicodeString \| Assigned to CP1252String \| Assigned to RawByteString \| Assigned to ShortString \| Assigned to PChar \|
153	153		\|----\|----\|----\|----\|----\|----\|----\|----\|----\|----\|
154	154		\| const \| const s = 'äöü'; \| UTF-16 encoded \| working \| working \| working \| working \| working \| working \| working \|
155	155		\| String \| const s: String = 'äöü'; \| working \| working \| working \| working \| working \| working \| working \| working \|
156	156		\| ShortString \| const s: String\[15\] = 'äöü'; \| wrong \| wrong \| wrong \| wrong \| wrong \| wrong \| wrong \| not available \|
157	157		\| UTF8String \| const s: UTF8String = 'äöü'; \| working \| working \| working \| working \| working \| working \| working \| working \|
158	158		\| UnicodeString \| const s: UnicodeString = 'äöü'; \| working \| working \| working \| working \| working \| working \| working \| wrong \|
159	159		\| String with declared code page \| type CP1252String = type AnsiString(1252); \| working \| working \| working \| working \| working \| working \| wrong \| wrong \|
160	160		\| RawbyteString \| const s: RawbyteString = 'äöü'; \| working \| working \| working \| working \| to codepage 0 changed \| working \| working \| working \|
161	161		\| PChar \| const c: PChar = 'äöü'; \| wrong \| wrong \| wrong \| wrong \| wrong \| wrong \| wrong \| wrong \|
162	162
163	163		Remember, assignment between variables of different string types always works thanks to their dynamic encoding in FPC 3+. The data is converted automatically when needed.\
164	164		Only string literals are a challenge.
165	165
166	166		# Coming from older Lazarus + LCL versions
167	167
168	168		Earlier (before 1.6.0) LCL supported Unicode with dedicated UTF8 functions. The code was not at all compatible with Delphi.
169	169
170	170		Now many old LCL applications continue to work without changes. However it makes sense to clean the code to make it simpler and more Delphi compatible. Code that reads/writes data with Windows system codepage encoding breaks and must be changed. (See #Reading / writing text file with Windows codepage).
171	171
172	172		Explicit conversion functions are only needed when calling Windows Ansi functions. Otherwise FPC takes care of converting encodings automatically. Empty conversion functions are provided to make your old code compile.
173	173
174	174		- UTF8Decode, UTF8Encode - Almost all can be removed.
175	175		- UTF8ToAnsi, AnsiToUTF8 - Almost all can be removed.
176	176		- UTF8ToSys, SysToUTF8 - All can be removed. They are now dummy no-ops and only return their parameter.
177	177
178	178		File functions in RTL now take care of file name encoding. All (?) file name related ...UTF8() functions can be replaced with the Delphi compatible function without UTF8 suffix. For example FileExistsUTF8 can be replaced with FileExists.
179	179
180	180		Most UTF8...() string functions can be replaced with the Delphi compatible Ansi...() functions. For example UTF8UpperCase() -\> AnsiUpperCase().
181	181
182	182		Now Unicode works in non-GUI programs, too. It only requires a dependency for LazUtils and placing LazUTF8 unit into the uses section of main program file.
183	183
184	184		For historical reference, this was the old Unicode support in LCL: [[Old LCL Unicode Support]]
185	185
186	186		# Technical implementation
187	187
188	188		What actually happens in the Unicode system? These 2 FPC functions are called in an early initialization section, setting the default String encoding in FPC to UTF-8 :
189	189
190		-	~~` SetMultiByteConversionCodePage(CP_UTF8);`\~~
191		-	~~` SetMultiByteRTLFileSystemCodePage(CP_UTF8);`~~
	190	+	```text
	191	+	SetMultiByteConversionCodePage(CP_UTF8);
	192	+	SetMultiByteRTLFileSystemCodePage(CP_UTF8);
	193	+	```
192	194
193	195		Under Windows the UTF8...() functions in LazUTF8 (LazUtils) are set as backends for RTL's Ansi...() string functions. Thus those functions work in a Delphi compatible way.
194	196
195	197		# Open issues
196	198
197	199		- TFormatSettings char (bug [27086](http://bugs.freepascal.org/view.php?id=27086)): for example: ThousandSeparator, DecimalSeparator, DateSeparator, TimeSeparator, ListSeparator. These should be replaced with string to support UTF-8. For example under Linux with LC_NUMERIC=ru_RU.utf8 the thousand separator is the two byte nbsp/160.
198	200		- Workaround: use single space characters instead as done in patch here: [27099](http://bugs.freepascal.org/view.php?id=27099)
199	201
200	202		### WinAPI function calls in FPC libs
201	203
202	204		- Unit registry, TRegistry - this unit uses Windows Ansi functions and therefore you need to use UTF8ToWinCP, WinCPToUTF8. Formerly it needed UTF8ToSys.
203	205		- All Windows API Ansi function calls in FPC's libraries must be replaced with the "W" function version. This must be done in any case for the future UTF-16 support, thus there is no conflict of interest here.
204	206		- TProcess - under Windows TProcess FPC 3.0 only supports system codepage. Use either TProcessUTF8 of unit utf8process or use FPC trunk where it is fixed, see issue [29136](http://bugs.freepascal.org/view.php?id=29136)
205	207
206	208		ToDo: List all related FPC bug tracker issues and patches that should be applied.
207	209
208	210		# Future
209	211
210	212		The goal of FPC project is to create a Delphi compatible UnicodeString (UTF-16) based solution, but it is not ready yet. It may take some time to be ready.
211	213
212	214		This UTF-8 solution of LCL in its current form can be considered temporary. In the future, when FPC supports UnicodeString fully in RTL and FCL, Lazarus project will provide a solution for LCL that uses it. At the same time the goal is to preserve UTF-8 support although it may require changes to string types or something. Nobody know the details yet. We will tell when we know...
213	215
214	216		In essence LCL will probably have 2 versions, one for UTF-8 and one for UTF-16.
215	217
216	218		# FAQ
217	219
218	220		## What about Mode DelphiUnicode?
219	221
220	222		The {\$mode delphiunicode} was added in FPC 2.7.1 and is like {\$Mode Delphi} with {\$ModeSwitch UnicodeStrings}. See the next question about ModeSwitch UnicodeStrings.
221	223
222	224		## What about ModeSwitch UnicodeStrings?
223	225
224	226		The {\$ModeSwitch UnicodeStrings} was added in FPC 2.7.1 and defines "String" as "UnicodeString" (UTF-16), "Char" as "WideChar", "PChar" as "PWideChar" and so forth. This affects only the current unit. Other units including those used by this unit have their own "String" definition. Many RTL strings and types (e.g. TStringList) uses 8-bit strings, which require conversions from/to UnicodeString, which are added automatically by the compiler. The LCL uses UTF-8 strings. It is recommended to use UTF-8 sources with or without "-FcUTF8".
225	227
226	228		## Why not use UTF8String in Lazarus?
227	229
228	230		Short answer: Because the FCL does not use it.
229	231
230	232		Long answer: UTF8String is defined in the system unit as
231	233
232	234		``` pascal
233	235		UTF8String = type AnsiString(CP_UTF8);
234	236		```
235	237
236	238		The compiler always assumes it has UTF-8 encoding (CP_UTF8), which is a multi byte encoding (i.e. 1-4 bytes per codepoint). Note that the \[\] operator accesses bytes, not characters nor codepoints. Same for UnicodeString, but words instead of bytes. On the other hand a String is assumed at compile time to have DefaultSystemCodePage (CP_ACP). DefaultSystemCodePage is defined at run time, so the compiler conservatively assumes that String and UTF8String have different encodings. When you assign or combine String and UTF8String the compiler inserts conversion code. Same for ShortString and UTF8String.
237	239
238	240		Lazarus uses the FCL, which uses String, so using UTF8String would add conversions. If DefaultSystemCodePage is not UTF-8 you lose characters. If it is UTF-8 then there is no point to use UTF8String.
239	241
240	242		UTF8String becomes useful when eventually there is an UTF-16 FCL.
241	243
242	244		## Why does UTF8String show strange characters and String works
243	245
244	246		For example:
245	247
246	248		``` pascal
247	249		var s: UTF8String = 'ä'; // with default flags (i.e. no -Fc) this creates garbage
248	250		// even on a UTF-8 Linux system
249	251		```
250	252
251	253		Question: Is it a bug? Answer: No, because it works as documented.
252	254
253	255		FPC ignores the LANG variable to create on every system the same result. For historical reasons/Delphi compatibility it uses ISO-8859-1 as default. Lazarus prefers UTF-8.
254	256
255	257		- UTF-8 sources work with String, because
256	258		1. FPC does not add conversion code for normal String literals by default.
257	259		2. The source codepage is equal to the runtime codepage. On Windows LazUTF8 sets it to CP_UTF8.
258	260		- UTF8String requires UTF-8 sources. Since FPC 3.0 for UTF8String you must tell the compiler, that the source is UTF8 (-FcUTF8, {\$codepage UTF8}, or save file as UTF-8 with BOM).
259	261		- Note: If you tell the compiler the source encoding UTF-8 it changes all non ASCII string literals of this unit to UTF-16, increasing the size of the binary, adding some overhead and PChar on literals require an explicit conversion. That's why Lazarus does not add it by default.
260	262
261	263		## What happens when I use \$codepage utf8?
262	264
263	265		FPC has very limited UTF-8 support. In fact, FPC only supports storing literals as either "default" encoded 8-bit strings or widestrings. So any non default codepage is converted to widestring, even if it is the system codepage. For example most Linux/Mac/BSD use UTF-8 as system codepage. Passing -Fcutf8 to the compiler will store the string literal as widestring.
264	266
265	267		At run time the widestring literal is converted. When you assign the literal to an AnsiString the widestring literal is converted using the widestringmanager to the system encoding. The default widestringmanager under Unix simply converts the widechars to chars, destroying any non ASCII character. You must use a widestringmanager like the unit cwstring to get correct conversion. Unit LazUTF8 does that.
266	268
267	269		# 関連情報
268	270
269	271		- Unicode やコードページ、文字列の種別や FPC での RTL についての情報は以下にあります。注）この情報はすべてが Lazarus の UTF-8 システムで有効ではありません。なぜなら、 FPC の観点からは、これはハックであり、デフォルトのコードページを変更することになるからです。\
270	272		- [[FPC Unicode support/ja]]
271	273
272	274		---
273	275		From the Free Pascal wiki, page "Unicode Support in Lazarus/ja", as it stood on 2024-03-10 after a change by Ariben, out of the wiki's dump of 2024-05-26. Copyright stays with the author who wrote it, according to the old wiki's [[Licensing]] page. If you have concerns about this material, [message the admin](/wiki-contact).

Back to history