unicodedata unit

The fpwidestring unit relies on having relevant Unicode collation data linked in the binary. The Unicode data is managed using the routines in the unicodedata unit. The FPC project distributes some Unicode collation data in .bco files which can be loaded using the LoadCollation routines. The LoadCollation is the main routine of this unit.

All collation data requires at least the Default Unicode Collation Element Table to be registered (called DUCET). The DUCET encoding is provided by the unicodeducet unit, part of the rtl-Unicode package.

There are two ways to register collations :

1. at compile time: by including the desired collation unit, for example for Russian and Japanese languages to be available you will have to include collation_ru and collation_ja from package "rtl-unicode".
2. at runtime using the above mentioned LoadCollation function.

The two ways can co-exist: some collations may be compile time included (for example for most used collations) and others can be loaded at runtime in the same application.

The binary collation files are endian sensitive:

• there are files for little endian systems named collation_lang_le.bco (such as collation_ru_le.bco and collation_ja_le.bco)
• there are files for big endian systems named collation_lang_be.bco (such as collation_ru_be.bco and collation_ja_be.bco).

Note that the compile time units collation units (collation_lang.pas) include already the unicodeducet.pas (DUCET) unit so it is not necessary to include it manually, contrary to the binary files. So an application that only uses the binary collation files should at least include the unicodeducet unit or manually load the binary collation collation_ducet_le.bco or collation_ducet_be.bco, depending on the endianness of the platform. The LoadCollation call using a directory and the language ducet automatically select the correct file.

Classes and records

TCollationTable	record
TCollationTableItem	record
TUC_Prop	record	Unicode character properties record.
TUCA_DataBook	record	Collation data book.
TUCA_PropItemContextRec	record	Internal structure for unicode collation data encoding.
TUCA_PropItemContextTreeNodeRec	record	Internal tree node structure for sorted storage of unicode collation data.
TUCA_PropItemContextTreeRec	record	Internal tree structure for sorted storage of unicode collation data.
TUCA_PropItemRec	record	Unicode Collation data item properties.
TUCA_PropWeights	record	Weights.
TUInt24Rec	record	Unicode integer 24-bit data.

Procedures and functions

AddAliasCollation	function
BytesToName	function
BytesToString	function
CanonicalOrder	procedure	Put unicode string in canonical order.
CompareSortKey	function	Compare two sort keys.
ComputeSortKey	function	Compute the sort key for a string.
FilterString	function
FindCollation	function	Find a collation by name.
FreeCollation	procedure	Free collation data.
FromUCS4	procedure	Convert UCS4 to UNicodeChar.
GetCollationCount	function	Return the number of known collations.
GetProps	function	Get unicode character data.
GetPropUCA	function	Get unicode collation algorithm properties for a unicode character.
IncrementalCompareString	function	Compare 2 strings using the specified collation.
LoadCollation	function	Load a binary collation data file from file.
NormalizeNFD	function	Perform unicode normalization D on a string.
PrepareCollation	procedure	Prepare a collation for use in the list.
RegisterCollation	function	Register a new collation.
ToUCS4	function	Encode unicode UTF16 surrogate pair to UCS4 character.
UnicodeIsHighSurrogate	function	Check if a UTF16 character is the high character in a surrogate pair.
UnicodeIsLowSurrogate	function	Check if a UTF16 character is the low character in a surrogate pair.
UnicodeIsSurrogatePair	function	Check if a pair of UTF16 encoded characters is a valid surrogate pair.
UnicodeToLower	function	Transform unicode string to lowercase.
UnicodeToUpper	function	Transform unicode string to uppercase.
UnregisterCollation	function	Remove a collation from the list of known collections.
UnregisterCollations	procedure	Unregister all collations.

Types

PCollationTableItem	type
PUC_Prop	type	Pointer to TUC_Prop record.
PUCA_DataBook	type	Pointer to TUCA_DataBook type.
PUCA_PropItemContextRec	type	Pointer to TUCA_PropItemContextRec.
PUCA_PropItemContextTreeNodeRec	type	Pointer to TUCA_PropItemContextTreeNodeRec.
PUCA_PropItemContextTreeRec	type	Pointer to TUCA_PropItemContextTreeRec.
PUCA_PropItemRec	type	Pointer to TUCA_PropItemRec.
PUCA_PropWeights	type	Pointer to TUCA_PropWeights.
PUInt24	type	Pointer to TUInt24Rec.
TCategoryMask	set
TCollationField	enumeration	Options for the collation data.
TCollationFields	set	Set of TCollationField.
TCollationName	type	Collation name string type (fixed length).
TCollationTableItemArray	type
TCollationVersion	type
TEndianKind	enumeration	Endianness enumerator.
TSetOfByte	set
TUCA_VariableKind	enumeration	Options for weighting data.
TUCASortKey	type	Array of TUCASortKeyItem.
TUCASortKeyItem	type	Alias for WORD.
TUnicodeStringArray	type
UInt24	type	Alias for TUInt24Rec.

Constants

DEFAULT_UCA_COMPARISON_STRENGTH	constant
DirectorySeparator	constant
ENDIAN_SUFFIX	constant	Suffixes for endianness.
ERROR_INVALID_CODEPOINT_SEQUENCE	constant	Error exit code for UnicodeToLower/UnicodeToUpper.
HIGH_SURROGATE_BEGIN	constant	First value for high surrogate values.
HIGH_SURROGATE_COUNT	constant	Number of high surrogate values.
HIGH_SURROGATE_END	constant	Last value for high surrogate values.
LOW_SURROGATE_BEGIN	constant	First value for low surrogate values.
LOW_SURROGATE_END	constant	Last value for low surrogate values.
MAX_LEGAL_UTF32	constant	Maximum value of a legally encoded UTF32 value (currently unused).
MAX_WORD	constant	Maximum value of a WORD typed value.
reCodesetConversion	constant
ROOT_COLLATION_NAME	constant	Name of the root collation (used if no collation name is given).
UCS4_HALF_BASE	constant	Offset for UCS4 character encoding.
UCS4_HALF_MASK	constant	Unused currently.
UGC_ClosePunctuation	constant	Unicode token category: Close punctuation.
UGC_CombiningMark	constant	Unicode token category: Uppercase letter.
UGC_ConnectPunctuation	constant	Unicode token category: Connect punctuation.
UGC_Control	constant	Unicode token category: control token.
UGC_CurrencySymbol	constant	Unicode token category: Currency symbol.
UGC_DashPunctuation	constant	Unicode token category: Dash punctuation.
UGC_DecimalNumber	constant	Unicode token category: Uppercase letter.
UGC_EnclosingMark	constant	Unicode token category: Uppercase letter.
UGC_FinalPunctuation	constant	Unicode token category: Final punctuation.
UGC_Format	constant	Unicode token category: Formatting token.
UGC_InitialPunctuation	constant	Unicode token category: Initial punctuation.
UGC_LetterNumber	constant	Unicode token category: Letter-number.
UGC_LineSeparator	constant	Unicode token category: Line separator.
UGC_LowercaseLetter	constant	Unicode general category: Lowercase letter.
UGC_MathSymbol	constant	Unicode token category: Math symbol.
UGC_ModifierLetter	constant	Unicode general category: modifier letter.
UGC_ModifierSymbol	constant	Unicode token category: modifier symbol.
UGC_NonSpacingMark	constant	Unicode general category: Non-spacing mark.
UGC_OpenPunctuation	constant	Unicode token category: Open punctuation.
UGC_OtherLetter	constant	Unicode general category: Other letter.
UGC_OtherNumber	constant	Unicode token category: Other number.
UGC_OtherPunctuation	constant	Unicode token category: Other punctuation.
UGC_OtherSymbol	constant	Unicode token category: Other symbol.
UGC_ParagraphSeparator	constant	Unicode token category: Paragraph separator.
UGC_PrivateUse	constant	Unicode token category: For private use.
UGC_SpaceSeparator	constant	Unicode token category: Space separator.
UGC_Surrogate	constant	Unicode token category: Surrogate token.
UGC_TitlecaseLetter	constant	Unicode general category: Titlecase letter.
UGC_Unassigned	constant	Unicode token category: As yet unassigned.
UGC_UppercaseLetter	constant	Unicode general category: Uppercase letter.
UnicodeCategoryNames	constant
ZERO_UINT24	constant	Empty TUInt24Rec value.

Resource strings

SCollationNotFound

resource string

Error message when a collation is not found.