There has been some heated discussion on the Lazarus lists on the subject to character encodings etc. This has exposed several issues with the FPC Manual that I wanted to record.
1. The char type
The manual says: "A Char is exactly 1 byte in size, and contains one ASCII character. "
This was probably true when Pascal was first defined, but char is often now used for any on-byte character set e.g. ISO 8859-1. Replace ASCII with ANSI.
2. WideChar
The Manual says: "A WideChar is exactly 2 bytes in size, and contains one UNICODE character in UTF-16 encoding. "
This seems to be wrong as UTF-16 is not limited to code points defined using a single 16-bit code unit, but also permits code points comprising two 16-bit code units. The definition should be updated to indicate that a WideChar was really created for the obsolescent UCS-2 and is limited to a UTF-16 subset (Unicode characters that can be expressed as a single 16-bit code unit).
Proposed replacement text: "A WideChar is exactly 2 bytes in size, and contains one UNICODE character in UCS-2 encoding or UTF-16 encoding limited to the Basic Multilingual Plane. Note that Unicode Characters represented by a UTF-16 code points that require two 16-bit code units cannot be contained in a single WideChar variable."
3. UnicodeStrings
The Manual says: "For multi-byte string types, the basic character has a size of at least 2."
Proposed improvement:
"Multi-byte string types are used to represent Unicode characters encoded as code points requiring two or four bytes".
As with UTF8String, the following caveat should also be added:
"Since a unicode character may require two or four bytes to be represented in the UTF-16 encoding, there are 2 points to take care of when using UnicodeString/WideString:
1. The character index – which retrieves a WideChar at a certain position – must be used with care: the expression S[i] will not necessarily be a valid character for a string S of type UnicodeString/WideString.
2. The length of the string is not necessarily equal to the number of elements in the array. The standard function length cannot be used to get the character length of the string, it will always return the array length.
------------------------------------------------------
Wiki Page on "Character and string Type"
1. This needs to start with a Health Warning on the use of the word Unicode. Proposed Text (borrowing from Wikipedia):
"Free Pascal supports several character and string types. They range from single ANSI characters to unicode strings and also include pointer types. Differences also apply to encodings and reference counting. ANSI is typically used to refer to single byte character encodings - although FPC also uses AnsiStrings to hold Unicode UTF-8 encoded strings.
Unicode is a computing industry standard for the consistent encoding, representation, and handling of text expressed in most of the world's writing systems. Developed in conjunction with the Universal Coded Character Set (UCS) standard and published as The Unicode Standard, the latest version of Unicode contains a repertoire of 136,755 characters covering 139 modern and historic scripts, as well as multiple symbol sets.
Unicode can be implemented by different character encodings. The Unicode standard defines UTF-8, UTF-16, and UTF-32, and several other encodings are in use. The most commonly used encodings are UTF-8, UTF-16 and UCS-2, a precursor of UTF-16.
The original idea behind Unicode was to replace the typical 256-character encodings requiring 1 byte per character with an encoding using 2^16 = 65,536 values requiring 2 bytes per character.The early 2-byte encoding was usually called "Unicode", but is now called "UCS-2". UCS-2 differs from UTF-16 by being a constant length encoding and only capable of encoding characters of Basic Multilingual Plane (BMP), it is supported by many programs. However, "UCS-2 should now be considered obsolete. It no longer refers to an encoding form in either 10646 or the Unicode Standard.
Unfortunately, the term Unicode, in common usage, is still often used to refer to the UCS-2 two byte encoding and this can give rise to much confusion e.g. when Unicode is used when referring to the UTF-8 encoding."
2. The text on WideChar is too terse and needs to be expanded. Proposed text:
"A variable of type WideChar, also referred to as UnicodeChar (which derives from the archaic use of Unicode to mean UCS-2), is exactly 2 bytes in size, and usually contains either:
(a) a single UCS-2 code point, or
(b) a single UTF-16 code unit.
In case (b), this is sufficient for Unicode Characters that have a UTF-16 code point that comprises a single 16-bit code unit i.e. characters in the Basic Multilingual Plane. However, all other UTF-16 characters have a UTF-16 code point that comprises a two 16-bit code units. FPC provides no specific support for such characters which require, e.g. a WideChar pair to encoded them."
Note: that the byte order used to store a WideChar can vary between platforms.
2. PChar
This should be identified as a synonym for PAnsiChar in FPC, It can also be as a C style pointer to any AnsiString including UTF-8.
It may also be useful to add a note that in later versions of Delphi, PChar is a synonym for PWideChar.