Hi all

im bit lost with all this unicode support ..im tryin to make picture in my head..here is my problem

MS-SQLserver 2008 uses utf16 to represent data
lazarus uses UTF8
FPC doesnt use unicode

my first question is ..is it possible at all to present valid data in program compiled with lazarus that connects to MSSQL server?

The function UTF8Encode converts from utf-16 to utf-8, try using that.

It does? Not by default. If you define your fields as varchar(x), it can be used to use UTF-8. nvarchar is utf-16 by default, off course.

Just set your connection to UTF-8 and you're done. It doesn't matter how the data is stored. (But if you can choose, use nvarchar, so no conversion is needed)

Joost.

S, Joost van der Sluis pi?e:

I read that MS SQL server 2008 uses UCS2 which is UTF-16 ...im realy cofused by all this

i did look up my field in table and it is nvarchar and i have odbcConnection to msSQL server i added SQLquery and dataGrid but my local characters are not shown or are represented by question mark. i did set charactercode to UTF-8.

Eduard Filipas wrote

I read that MS SQL server 2008 uses UCS2 which is UTF-16 ...im realy cofused by all this

UCS2 UTF-16

UCS2 is a subset of UTF-16. It doesn't cover all code points that Unicode defines, which UTF-8, UTF-16, UTF-32 does.

Regards,
- Graeme -

Till 1996 (Unicode 2.0) you are right. After that date they are equal.

Marc

my problem is that i dont get local characters in data aware objects

im connected via odbcConnection (charSet is set to UTF-8) to ms sql server express 2008
same problems i get vith zeos components

within Database string fields are all nvarchar(x)

regards
edo

S, Marc Weustink pi?e:

Ehm, isn't it the other way round?
The Unicode 2.0 added UTF-16 as successor of UCS2. UCS2 is still fixed 2 byte. They are not the same (Graeme is right).

BTW, the lconvencoding.pas GuessEncoding seems to treat the UTF16 BOMs as "UCS2" BOMs. Practically this is correct for many existing documents, but officially it is wrong.

Mattias

Marc Weustink wrote

Till 1996 (Unicode 2.0) you are right. After that date they are equal.

Huh?

Quoted text from the Unicode wikipedia article. Note the last three lines.

"Unicode can be implemented by different character encodings. The most commonly used encodings are UTF-8 (which uses one byte for any ASCII characters, which have the same code values in both UTF-8 and ASCII encoding, and up to four bytes for other characters), the now-obsolete UCS-2 (which uses two bytes for each character but cannot encode every character in the current Unicode standard), and UTF-16 (which extends UCS-2 to handle code points beyond the scope of UCS-2)."

Regards,
- Graeme -

Hans-Peter Diettrich wrote

IMO UCS-2 primarily is an ISO/IEC Unicode version, with 16 bit code positions, as applicable to Unicode 1. Later Unicode versions require more bits, so that UCS-4 (32 bit) had been introduced.

Thus my statement still holds true. UCS2 UTF-16
:)

Regards,
- Graeme -

IMO UCS-2 primarily is an ISO/IEC Unicode version, with 16 bit code positions, as applicable to Unicode 1. Later Unicode versions require more bits, so that UCS-4 (32 bit) had been introduced.

DoDi

as much as i do enjoy your debate what UCS2 is and what is not i still dont know how to proceed in my problem where data aware controls dont recognize my local characters...

edo

S, Hans-Peter Diettrich pi?e:

According to unicode.org, when UTF-16 got introduced, the USC2 standard was extended. So yes they are the same.
In some cases when ppl refer to USC2 they mean the old unicode 1.1 standard, but that is wrong. The name USC2 is misleading and should not be used anymore.

Marc

Hello Lazarus-List,

Thursday, May 5, 2011, 12:20:10 PM, you wrote:

MW> According to unicode.org, when UTF-16 got introduced, the USC2 standard MW> was extended. So yes they are the same.
MW> In some cases when ppl refer to USC2 they mean the old unicode 1.1 MW> standard, but that is wrong. The name USC2 is misleading and should not MW> be used anymore.

Maybe, but, taken from www.unicode.org glossary of terms:

UCS-2. ISO/IEC 10646 encoding form: Universal Character Set coded in 2 octets, limited to the Basic Multilingual Plane. (See Appendix C, Relationship to ISO/IEC 10646.)

UTF-16. A multibyte encoding for text that represents each Unicode character with 2 or 4 bytes; it is not backward-compatible with ASCII. It is the internal form of Unicode in many programming languages, such as Java, C#, and JavaScript, and in many operating systems. More technically: (1) The UTF-16 encoding form. (2) The UTF-16 encoding scheme. (3) ?Transformation format for 16 planes of Group 00,? defined in Annex C of ISO/IEC 10646:2003; technically equivalent to the definitions in the Unicode Standard.

--------------------

I think that the text that says the UCS2 has been extended, does not means that UCS2 has been extended, it says that UCS2 has been extended to UTF-16, so UCS2 can not be considered Unicode anymore as noted in ISO 10646:

UCS-2. UCS-2 stands for ?Universal Character Set coded in 2 octets? and is also known as ?the two-octet BMP form.? It was documented in earlier editions of 10646 as the two-octet (16-bit) encoding consisting only of code positions for plane zero, the Basic Multilingual Plane. This documentation has been removed from ISO/IEC 10646:2011, and the term UCS-2 should now be considered obsolete. It no longer refers to an encoding form in either 10646 or the Unicode Standard.

-- Best regards,
Jos?

From the basic FAQ:
Q: What is the difference between UCS-2 and UTF-16?

A: UCS-2 is obsolete terminology which refers to a Unicode implementation up to Unicode 1.1, before surrogate code points and UTF-16 were added to Version 2.0 of the standard. This term should now be avoided.

UCS-2 does not define a distinct data format, because UTF-16 and UCS-2 are identical for purposes of data exchange. Both are 16-bit, and have exactly the same code unit representation.

Sometimes in the past an implementation has been labeled "UCS-2" to indicate that it does not support supplementary characters and doesn't interpret pairs of surrogate code points as characters. Such an implementation would not handle processing of character properties, code point boundaries, collation, etc. for supplementary characters. [AF]

from the "Both are 16-bit, and have exactly the same code unit representation" I concluded they are the same

Anyway, we're drifting away from the original question.

Marc

My final words on this...

Not just that, the Unicode standard also defines things like Normalization, Decomposition, Collation, Rendering, Bidirectional Display Order, Character Properties etc.... all of which I believe UCS-2 doesn't. All in all, Unicode is far beyond the scope of UCS-2.

Regards,
- Graeme -

I agree that UCS-2 no longer represents the current Unicode range, but it still is a true UCS-4 subset (BMP).

The UCS standards define Unicode as ranges of values, while the UTF standards define encodings.

The UTF-7/8 encodings are purely numerical compression schemes, while UTF-16 (with surrogate pairs) more reflects a tree-like structure of "planes", "groups", "blocks", "codepages" etc., favored by the Unicode Consortium. Such a view may be interesting to font writers, which can restrict an font to part of the full Unicode range, but is of little help with handling Unicode programmatically.

DoDi

Eduard Filipas ha scritto:

as much as i do enjoy your debate what UCS2 is and what is not i still dont know how to proceed in my problem where data aware controls dont recognize my local characters...

Sort of "the operation was successful, but the patient is dead" ;-)

However, forgetting the nuances, AFAIK what SQL documentation calls UCS2 should have a rough equivalent to the WideString (and WideChar) type in FPC. Therefore if you store your database strings in WideStrings, they should be treated and converted properly. Or you may take advantage of the functions UTF8Encode and UTF8Decode to convert Widestrings back and forth into UTF-8. But maybe not always 100% correct, from what I can gather. You must double check that all symbols of your locale are handled properly. The same sort of problems arises when using php.