Hi *,
In current Delphi is String synonym for base type UnicodeString UTF-16 AFAIU ATM in FPC is String synonym for AnsiString (as in previos versions of Delphi)
Are there any plans to change meaning of String type ? (like Delphi to UnicodeString , or UTF8String?)
Are there any plans to intorduce implicit conversions between AnsiStrings (ANSI code page) to UTF8Strings (UTF-8 encoded) or something like this ?
TIA
-Laco.

LacaK wrote

In current Delphi is String synonym for base type UnicodeString UTF-16

AFAIK, in current Delphi (which I don't have) a "String" is a variable that can contain dynamically coded informations (such as locally coded 8-Bit ANSI, UTF-8, UTF-16, ...) and - of course - know which code it holds.

If a string is generated by the VCL from a Window API function, the coding will be UTF-16, though, but if you create a string with some other coding it will be automatically re-coded to UTF16 before sending it into a Windows API function.

-Michael

AFAIK, in current Delphi (which I don't have) a "String" is a variable that can contain dynamically coded informations (such as locally coded 8-Bit ANSI, UTF-8, UTF-16, ...) and - of course - know which code it holds.

I understand "By default, variables declared as type String are *UnicodeString*.**", that String=UnicodeString
See: http://docwiki.embarcadero.com/VCL/en/System.UnicodeString and also
http://docwiki.embarcadero.com/RADStudio/en/String_Types#UnicodeString

Note alse, that AnsiString holds additional informations about character encoding:
"The AnsiString
<http://docwiki.embarcadero.com/VCL/en/System.AnsiString> structure contains a 32-bit length indicator, a 32-bit reference count, a 16-bit data length indicating the number of bytes per character, and a 16-bit code page."

-Laco.

If/when this is done, it will only be with a compiler switch or directive.

Are there any plans to intorduce implicit conversions between AnsiStrings (ANSI code page) to UTF8Strings (UTF-8 encoded) or something like this ?

That would be part of the general D2009 ansistring support you referred to in your other message. There is an svn branch (cpstrnew) that contains some preliminary work for this functionality, but nobody has worked on it for a long time. Developers interested in working on finishing that functionality are welcome!

Jonas

In our previous episode, Jonas Maebe said:

(
That won't be enough, since that would not change the relevant units and classes to such type. (e.g. tstringlist would remain defined ansistring)

For this to work, we probably will have to split targets into UTF16 and ansi. (and maybe multiple ansi's for some platforms) )

If it's a D2009-style ansistring, does that matter?

Jonas

In our previous episode, Jonas Maebe said:

A lot of conversion, since it will use ansistring(0) so reading/writing ansistring(cp_utf8) will force conversions. (0 means system encoding, $FFFF means never convert)

Besides that the usual three problems:

- I don't know how VAR behaves in this case. (passing a ansistring(cp_utf8) to a "var ansistring(0)" parameter),
- maybe overloading (only cornercases?) etc.
- inheritance. FPC defines base classes as ansistring(0) parameters, and Lazarus wants to inherit and override them with a different type. This will clash.

I've thought long and hard about this. Since the discussion what the dominant type should be won't stop anytime soon, and we probably will have to support both UTF8 (*nix) and UTF16 (Windows and *nix/QT) as basetypes in the long run, plus a time ANSI as legacy, the RTL has to be prepared for it anyway, we might as well allow this on all platforms from the start. (actually releasing them is a different question and depends on manpower)

That doesn't mean that a per unit switch is useless, but I think a target switch to fixate the bulk of the cases will save both us and the users a lot of grief.

AFAIR Delphi doesn't offer such a compiler option, because units with different settings do not fit together (2 VCL versions would not be sufficient).

I'm not sure about details, but the Delphi designers certainly encountered problems that definitely forbid mixing string types. One such problem may be a slowdown due to many implicit string conversions, together with compiler warnings about possible losses on the conversion back to Ansi, and real losses as I observed in VB years ago. Another one may be the maintenance of duplicate (overloaded) procedures in the standard libraries.

When FPC implements two distinct versions, adding another Unicode/Ansi level to the unit output tree, both versions can be compiled from the same source code, possibly using conditional compilation where necessary.

For my part, I'd be happy with two definitely different Ansi and UTF(8) string types, with automatic conversion. But even then it would be wise to add another string parameter type, like Delphi RawByteString, that accepts both Ansi and UTF-8 arguments without implicit conversions.

DoDi

Why should a tstringlist force ansistring(0)? Or does Delphi force it to be that way?

Conversion may indeed be required for output (input would only pass on the encoding of the input if based on ansistring($ffff)), but I think doing that only when necessary at the lowest level should be no problem. Many existing frameworks work that way.

Besides that the usual three problems: - I don't know how VAR behaves in this case. (passing a ansistring(cp_utf8) to a "var ansistring(0)" parameter),

var-parameters may indeed pose a problem in case some parameters of OS- neutral routines are required to have a particular encoding specified.

- maybe overloading (only cornercases?) etc.

Possibly, although I guess there are probably rules for that (whether they are document is another case though, probably...)

- inheritance. FPC defines base classes as ansistring(0) parameters, and Lazarus wants to inherit and override them with a different type. This will clash.

Why ansistring(0) for base classes? OS-level interfaces: yes, but why base classes?

I agree that the RTL should work regardless of the used string encoding, but I don't see why a particular encoding should be enforced throughout the entire RTL rather than just using ansistring($ffff) almost everywhere.

I also agree that we should strive to minimize the number of conversions in the RTL for some encodings (in particular indeed ansi, utf-8 and utf-16), but again this should not require a specially compiled RTL. E.g., insert(ansistring($ffff)),
delete(ansistring($ffff)), etc. can call to special-purpose versions for certain specific encodings of the input (e.g., for the three you mentioned), and only if the encoding is not directly supported or if different encodings are mixed then perform a round trip via some generic format (utf-16, utf-32, or something that depends on the platform).

This has the advantage that you always have all optimal implementations available, regardless of the platform or default string encoding. It does not require extra work because we have to write all those versions also if we want the RTL to be compilable for different default string encodings. And three checks in a case statement are not going to define the performance in a context of atomic reference counting, dynamic memory management and the occasional code page conversion (and since this may reduce the number of code page conversions when working with "non-native" strings, it can also be a performance win).

Outside the RTL, the encoding mainly matters if you perform manual low- level processing of a string (for i:=1 to length(s) do something_with(s[i])). But in that case your your code will either work with only one encoding and you have to enforce it via the parameter type anyway, or if it has to work with multiple encodings and then you can use a technique similar to what I described above for the RTL.

It's not really clear to me which problem this would solve, but I may be missing something.

Jonas

In our previous episode, Jonas Maebe said:

I mean that if you locally (for your units) set string=utf8string, TStringList still would be ansistring(0) or whatever the default becomes. (and it could be UTF16 even)
Since TStringList inherits from TStrings so would most Lazarus components.

Or does Delphi force it to be that way?

In D2009+ it is unicodestring, period. Everything is unicodestring (UTF16), ansistring (+ variants) are for legacy only, and people try to forget shortstring as quickly as possible.

Backwards compatibility to pre D2009 is essentially abandonned. I think they didn't even try for exactly the reasons I mean to address here.

I think in the planned Embarcadero cross-compile products, string will also be utf-16 on OS X and Linux. If only because it is (1) easier, and windows remains dominant by far (including UTF16 assuming codebases) (2) they plan to target QT.

Keep in mind that soon it will not be possible to upgrade from ansistring to a current version anymore (and something like D5..D7 already is no longer upgradable). Embarcadero changed the upgrade rules.

From Delphi related forums and maillist, I get the impression that most

fulltime delphi programmers migrated to unicode, and the occasional and legacy users not. The gap between these two groups is widening, but contrary to Embarcadero, we will be dealing with significant portions of both groups for a while (as new/existing users)
--------

So the question is how we are going to deal with this information, without forcing a big bang like Embarcadero did, prepare to support both (or more? see below) schemes for a while, _AND_ deal with the fact that UTF16 is mostly alien on non-Windows.

For me, having a mandatory UTF16 Unix is not an option, and a mandatory UTF8 Windows neither. (D2009+ incompatible)

Since no one choice with one default type per target (or even one to rule them all) will satisfy anybody, I was thinking about setting up multiple targets.

Of course it is uncharted territory, and while I lean towards that solution, it could be that there are hidden caveats.

Conversion may indeed be required for output (input would only pass on the encoding of the input if based on ansistring($ffff))

ansistring(0), system encoding would be more logical than $ffff. $ffff is used more internally in string conversion routines and for strings that are not strings.

But what does that mean on Windows, where the console encoding is OEMSTRING and not ansistring(0) ?

but I think doing that only when necessary at the lowest level should be no problem. Many existing frameworks work that way.

It touches all places where you touch the OS. But indeed one could try to split this by doing the classes utf8 or tunicodestring depending on OS.

And we have to deal with Windows, where the default is UTF16.

This is the core problem. What solution will do for everybody (legacy,Lazarus,Delphi/unicode?) or (ansistring(0), ansistring(cp_utf8) or TUnicodestring) ?

And what do we do if e.g. Lazarus changes opinion and goes from utf8 to utf16 on Windows? (e.g. the Delphi/unicode becomes the dominant influx).

And do we really want Lazarus' direction to fixate this for everybody?

Or what if they bring in a new Kylix principle with utf16 base type?

I'm very reluctant to make a choice here, and say "insert conversions" if something changes. I would build in some flexibility and potential differentiation from the start.

At least in principle. As said, we can see which combinations are popular for release time.

That only solves the 1-byte case. And while that solves some of the overloading problems deep in RTL and frameworks, it might not be applicable on largescale, since afaik you need to test in the routine for codepages manually ? IOW this can't be done in every routine with a string parameter in the entire classtree

Btw, while looking up rawbytestring I saw this in the Delphi help:

"Declaring variables or fields of type RawByteString should rarely, if ever, be done, because this practice can lead to undefined behavior and potential data loss."

How will you deal with e.g. Windows? Legacy string=ansistring(0), D2009 is string=utf16 TUnicodestring?

These are not the same types, and inheritance and the other problems will kill you if you attempt to combine it. We need two separate targets for Windows anyway. Maybe three (if Lazarus persists in UTF8 in windows)

The same if you have some large D2009 server app and want to migrate to FPC/Unix. While I don't like UTF16 on Unix, having it an option might make this an workable option.

This has the advantage that you always have all optimal implementations available, regardless of the platform or default string encoding.

But only for a handful of selected routines. The solution stops above the procedural level of the RTL, and there is no solution for code that assumes something else then the string type of the classes library was compiled with.

And there are three such cases

- normal FPC and Delph 2007- code : ansistring(0)
- Lazarus : ansistring=utf8
- Delphi 2009+ UTF16.

shortstring is not really a primary stringtype anymore in the library sense, so that matters less. The few units (dos, maybe fv/ide) are isolated.

Depending on lazarus choices it might become less.

Outside the RTL, the encoding mainly matters if you perform manual low- level processing of a string (for i:=1 to length(s) do something_with(s[i])).

The RTL is not the only interface with the OS. Like e.g. a widget set that may be ansi,UTF8 or UTF16.

But in that case your your code will either work with only one encoding and you have to enforce it via the parameter type anyway, or if it has to work with multiple encodings and then you can use a technique similar to what I described above for the RTL.

No. The two are not mutually exclusive. I see yours as an implementation detail how to write a relative agnostic RTL.

IOW it means that we can change the basetype easier with less conditional code.

Mainly the question what the classtree will be. The main operating type used in applications. You always need two RTLs for that, since it can be 1 or 2 byte, and even if you fixated it on one byte encodings, rawbytestring would force you to write case statements in each and every procedure.

Keep in mind that the Lazarus classtree is rooted in the RTL.

- fpGUI: ansistring = utf-8
- MSEgui: existing FPC UnicodeString = utf-16

Martin

I meant: why not use ansistring($ffff) instead? You could even add a property to tstringlist that causes it to force the encoding of added strings to a particular code page whenever a string is added.

Then a unicodestring version is certainly required, and an ansistring($ffff) version would have to be called differently.

I think in the planned Embarcadero cross-compile products, string will also be utf-16 on OS X and Linux. If only because it is (1) easier, and windows remains dominant by far (including UTF16 assuming codebases) (2) they plan to target QT.

I think it's a good decision to keep it the same everywhere, since string=unicodestring is not an opaque type in any way. As a result, choosing a different string type on other platforms would probably break lots of code again. And regardless of which toolkit you target on Mac OS X, conversions will probably happen anyway. The encoding used by Carbon and Cocoa is not specified anywhere afaik, and the CFString/NSString they are based on can use any encoding internally (I guess that's probably also UTF-16 for ease of processing).

For me, having a mandatory UTF16 Unix is not an option, and a mandatory UTF8 Windows neither. (D2009+ incompatible)

I don't think UTF-16 everywhere would be a big problem.

The fact that the formal return type is $ffff does not mean, afaik, that you also have to return something whose internal encoding is set to "$ffff". It can still be an ansistring(0), ansistring(OEMSTRING) or whatever. It simply means that the encoding won't be forced to anything in particular when you assign a value to the function result. If you then assign this function result to another variable (which may have a forced encoding), then a conversion will happen if the forced encoding is different from the actual one. If you assign it to another ansistring($ffff), no encoding change will happen in any case, and the destination string will "inherit" the source's encoding.

But what does that mean on Windows, where the console encoding is OEMSTRING and not ansistring(0) ?

As I said: ansistring($ffff).

I'm not sure why you say "indeed", because I did not propose to do that. I only proposed keeping as many RTL interfaces as possible in ansistring($ffff) to have something that's a) generic, and
b) with the least chance of resulting in encoding conversion

However...

And we have to deal with Windows, where the default is UTF16.

... since Delphi 2009 uses (unicode)string everywhere, we need at least also unicode versions.

A unicodestring version for Delphi-compatibility, and if required an ansistring($ffff) version for all other purposes (afaik that would also work with legacy ansistring=ansistring(0), although it's not yet clear to me what happens if you pass an empty ansistring(0) to a rawbytestring var-parameter -- is it still nil like with current ansitrings, or can you somehow extract its declared encoding?)

It's true that you probably need a separate overloaded version for unicodestring (just like we currently also have separate overloads for ansistring and unicodestring).

Most routines don't process strings themselves: they store them, pass them on (to routines that may process them) or return them. In those cases, you don't have to look at the encoding.

They are right if you mainly care about code maintainability. If you however insist on supporting multiple encodings efficiently and transparently, there is no other option. The danger they are talking about mainly occurs when mixing rawbytestring and string literals. And even that could actually be solved by the compiler (the compiler could insert a conversion of the string literal to the actual encoding of the rawbytestring at run time, just like we currently do for mixing widestring constants and ansistring), but CodeGear chose not to do that, presumably for efficiency reasons.

I think at most two are required for any target: unicodestring (D2009 compatibility), and if really necessary because somehow the unicodestring version causes too much overhead, an ansistring($ffff) version as well. That's only for the classes though, I think most of the base RTL can be simply ansistring($ffff).

Changing the string type in your entire application and RTL only because a widgetset uses it does not make sense to me. Generally, you want to process everything in whatever format is most convenient, and only convert it to the required type once you are actually communicate with the component.

That last part is not true. It's only required in those cases where strings are directly manipulated and where the overhead is very important. And again: if you want to support compiling the RTL for different one-byte code pages that operate directly on those strings without any conversions, you have to write all that different code anyway. It's mainly a matter of replacing ifdef's with case statements.

Jonas

Marco van de Voort wrote

I think in the planned Embarcadero cross-compile products, string will also be utf-16 on OS X and Linux.

Yak,

I had hoped that using the dynamically encoded string type nearly everywhere would allow for a great lot of not OS-specific code in the VCL (and LCL) without the need for excessive conversions maintaining the systems' coding (UTF-16 or UTF-8) in and out with GUI-centric user code.

I thought this would have been the main reason for introducing the additional complexity of the dynamically encoded string type.

-Michael

Marco van de Voort wrote

And what do we do if e.g. Lazarus changes opinion and goes from utf8 to utf16 on Windows? (e.g. the Delphi/unicode becomes the dominant influx).

The current way Lazarus works (UTF-8 in a String Type called "ANSIString", as well with Windows as with Linux without any auto-Conversion, introducing funny problems e.g. when just assigning a string constant to a Widestring) does not seem very appropriate.

I feel the logical move would be to use the dynamically encoded string type in the LCL API, but there might be some nasty hidden problems (e.g. with var parameters).

-Michael

I think at most two are required for any target: unicodestring (D2009 compatibility), and if really necessary because somehow the unicodestring version causes too much overhead, an ansistring($ffff) version as well. That's only for the classes though, I think most of the base RTL can be simply ansistring($ffff).

So if I understand correctly, then UnicodeString and also AnsiString types must "be extended" that they will hold also information about actual codepage (encoding) of string data they hold. (AFAIK ATM they hold only information about "reference count" and "size" and of course "data")

I am not expert, so I do not understand all aspect/problems which are joined with proper string handling, but some kind of implicit conversions (based on actual encoding of string data) is necessary (ANSI <-> UTF-8 <-> UTF-16 <-> ANSI ... etc.).

For example known problem with Euro currency symbol. In Windows is in CurrencyString global variable stored using ANSI codepage, but used in LCL (which expect UTF-8 encoding) without any explicit conversion, what leads to displayng "?" instead of "€" (for example in TDBEdit or TDBGrid)

Another problem when displaying character data in data-aware database controls (TDBEdit, TDBGrid). Data-aware controls (LCL) reads data from TField descendatns (FCL) using TField.Text property which returns "string" (without codepage information is not clear if it is AnsiString or UTF8String or UnicodeString). LCL expect UTF-8 strings, but it is not true in all cases (for example in case of ODBC)

-Laco.

In our previous episode, Michael Schnell said:

I had hoped that using the dynamically encoded string type nearly everywhere would allow for a great lot of not OS-specific code in the VCL (and LCL) without the need for excessive conversions maintaining the systems' coding (UTF-16 or UTF-8) in and out with GUI-centric user code.

That was our original idea. But it also required the input granularity (1,2 maybe 4) to be a variable.

I thought this would have been the main reason for introducing the additional complexity of the dynamically encoded string type.

Embacadero however decided otherwise and kept a wall between the 1 and 2 byte types. So at least 1 and 2 byte types as basetype are different targets.

I still have to study Jonas last message. It seems to indicate that I misunderstood what rawbytestring. If that is true, Jonas is right, separating the targets will result in two targets (rawbytestring and unicodestring)

Marco van de Voort wrote

I still have to study Jonas last message. It seems to indicate that I misunderstood what rawbytestring. If that is true, Jonas is right, separating the targets will result in two targets (rawbytestring and unicodestring)

Here's some nice explanation about how rawbytestring behaves in practice: http://www.micro-isv.asia/2008/08/using-rawbytestring-effectively/

And here's an answer by Barry Kelly to a post about rawbytestring explaining what the purpose of the type is (similar to what I said): http://www.codegod.de/WebAppCodeGod/Delphi-2009-RawByteString-vagaries-QID85470.aspx

He mentions using them as parameter types to reduce the number of overloads, but I'm still wondering about var-parameters in particular. I would guess that it may very well be forbidden to pass an ansistring(0) to a rawbytestring var-parameter, so it would still not solve everything in that case (and if it's not forbidden, I'm curious how you can obtain the statically defined codepage of the ansistring(0) at the callee side in case the input string was empty).

Jonas

Marco van de Voort wrote

But it also required the input granularity (1,2 maybe 4) to be a variable.

Sorry I don't understand what you mean with this.

Embacadero however decided otherwise and kept a wall between the 1 and 2 byte types. So at least 1 and 2 byte types as basetype are different targets.

Unfortunately I don't have Delphi > 2007. From what O read I understand that the dynamically code string type can hold 1, 2, and 4 byte (maybe even more) Codes for it's elements (denoted in one control-value) and each of those (theoretically) in different coding schemes (denoted in another control-value), allowing e.g. for UTF-8, UTF-16, UCS4, German ANSI, raw Byte, string....

Each assignment would auto recode the string "if necessary". I suppose that s1 := s2 would not do any recoding, but s1 := s2 + s3; would automatically synchronize the coding.

I suppose there are ways do define the coding (and force recoding), maybe similar to "setlength(s, 10)".

-Michael

Jonas Maebe wrote

in case the input string was empty).

As the coding scheme and element size are control-block-variables it seems that even an empty string should have the appropriate definitions.

-Michael

In our previous episode, Michael Schnell said:

That is wrong. Better read up on that.

Marco van de Voort wrote

That is wrong. Better read up on that.

AFAIK, this is what they announced some time ago, Seemingly it turned out to be done some other way...

Nonetheless fpc seems to intend to offer something like this (right now in an experimental branch).

-Michael

Marco van de Voort wrote

Embacadero however decided otherwise and kept a wall between the 1 and 2 byte types. So at least 1 and 2 byte types as basetype are different targets.

I'm actually not sure about that: the new ansistring type has a hidden "element size" field (in addition to the reference count, length and codepage), and from what I can see at page 10 of http://edn.embarcadero.com/article/images/38980/Delphi_and_Unicode.pdf , Delphi 2009's unicodestring is simply an ansistring(1200).

Jonas

IIRC RawByteString should be used like OpenString, as subroutine argument type only. In contrast to the name, a RawByteString has a variable encoding, i.e. implicit conversions are inserted for every use with other string types. Thus AnyByteString had been a better name for that type, IMO. Delphi does no more support (officially) non-textual data in strings, and TBytes should be used for such data.

How will you deal with e.g. Windows? Legacy string=ansistring(0), D2009 is string=utf16 TUnicodestring?

Is an Delphi UnicodeString really compatible with an WinAPI WideString/BSTR? AFAIR all BSTRs must reside in shared memory, so that copies are required for every API call.

Mainly the question what the classtree will be. The main operating type used in applications. You always need two RTLs for that, since it can be 1 or 2 byte, and even if you fixated it on one byte encodings, rawbytestring would force you to write case statements in each and every procedure.

UTF-8 combines an single (byte-based) storage type with lossless encoding of full Unicode. Ansi and UCS2 (really UTF-16) only *look* easier to handle in user code, but both will fail and require special code whenever characters outside the assumed codepage may occur.

DoDi

IMO a single encoding, i.e. UTF-8, can cover all cases. While some hard core Ansi coders may whine about such a convention, the absence of implicit string conversions (except in external library calls) will make such applications more performant than mixed-encoding versions.

The argument "my characters *always* will be inside my preferred codepage" will prove false sooner or later. While it's not up to a programming language to teach people the "better way of coding", the required efforts of the FPC/Lazarus developers IMO should have more weight. Why spend time in the design of multiple RTL/LCL versions, when a single version will be perfectly sufficient?

DoDi

> And we have to deal with Windows, where the default is UTF16. ... since Delphi 2009 uses (unicode)string everywhere, we need at least also unicode versions.

Since the generic Delphi "string" type can be any Unicode encoding now, it IMO would be legal to use UTF-8 or UTF-32 for it internally, in FPC. Some code, expecting UCS2/BMP text only, may become a bit slower due to according conversions in indexed access to chars, but no other *implicit* conversions will ever occur. Likewise the generic "char" type could become a 32 bit type, so that it can hold *every* Unicode codepoint.

For both "string" and "array of char" the "packed" keyword could be used to distinguish between different bytecount and encoding, where unpacked types contain UTF-32 chars. This would speed up user code with indexed access, in contrast to both UTF-8 and -16 encodings, and it would allow the user to optimize his code for either speed or size. Indexed access to packed types simply could be disallowed, without breaking anything since the default is "not packed".

Just some more ideas...

DoDi

Preface: I don't write international apps, and probably won't for the foreseeable future...

Isn't all of this concentration on trying to make strings have single byte characters (who cares how they are encoded), using the argument that it is somehow faster, incorrect for just about any modern processor, including embedded CPU's such as ARM? It was my understanding that 32 bit aligned access was always faster than byte aligned access on just about any CPU FPC still supports.

The argument holds just fine for memory, but I don't really get the speed argument. Maybe I'm missing something.

Jeff.

So it seems, that if we will have any "GenericString", with properties "reference count", "size", "character width", "codepage", then all other string types can be based on this string type. So other strings will be only any "shortcuts", and internaly will use same structure: AnsiString = GenericString(with actual system ANSI code page (0) ... or ... without any explicit codepage ($ffff))
UTF8String = GenericString(with UTF-8 encoding)
UnicodeString = GenericString(with UTF-16 encoding)

So it seems to me, that there is agreement on adding "character width", "codepage" to internal "string" record structure and provide conversions where needed, isn't it ? (more or less same approach like in Delphi)

Where is not agreement, it is fact what should be default string encoding (AnsiString($ffff) or UTF-8 or UTF-16 or UTF-32)

So if I revert to my original question ... is there any agreement on some points related to "future of String type" ?

P.S. I still does not understand, how can things work correctly if LCL expect that all AnsiStrings (String) are UTF8Strings, byt RTL/FCL does not strictly follow this (at least in Windows) ?

-Laco.

In our previous episode, Jeff Wormsley said:

1-byte access is always 1-byte aligned, and the memory system is still slower than these kind of issues. And you shuffle a lot of zeroes extra around.

But the trouble is also that 2-byte situation doesn't really solve anything, (you still have surrogates and it never will be as simple as it was), and a much bigger problem with legacy (how many two byte data do you get daily, and how much 1 byte?)

The argument holds just fine for memory, but I don't really get the speed argument. Maybe I'm missing something.

Shoveling twice as much memory around IS the speed argument :-)

LacaK wrote

P.S. I still does not understand, how can things work correctly if LCL expect that all AnsiStrings (String) are UTF8Strings, byt RTL/FCL does not strictly follow this (at least in Windows) ?

LCL uses SysToUTF8 and UTF8ToSys if it uses the RTL (and the FCL). This is often done with wrappers that wrap the RTL method and do the conversion (e.g. FileExistsUTF8, etc.).

Regards,
Sven

Sven Barth wrote / napísal(a):

As I wrote in any of my previous message, AFAIK this is not true in case of "fcl-db" and Lazarus data-aware components like TDBGrid, TDBEdit ... They use "TField.Text: String" property to get string conent of field and display them.
AFAIU LCL expects, that TField.Text will always return UTF-8 encoded string (because no conversion (SysToUTF8) is done in dbgrids.pas or dbedit.inc) , but this is not true always.

So where is error ?
1. Is it wrong expectation by LCL, that TField.Text is always UTF8 string -or-
2. Is it wrong in implementation of TSQLConnectors, which write data into record buffer (of TStringField) and do not convert them always into UTF-8 ?
(if data should be always in UTF-8 then it will be good redefine TField.Text property like "property Text: UTF8String" to be clear, that we always work with UTF-8 strings)
-or
3. I missed something ? ;-)

-Laco.

Hans-Peter Diettrich wrote

Since the generic Delphi "string" type can be any Unicode encoding now,

This

is what I (not owning a Delphi > 2007) thought, too, and have been bashed for.

But The document "Delphi and Unicode" by Marco Cantu ( http://edn.embarcadero.com/article/images/38980/Delphi_and_Unicode.pdf ), dated Nov, 2008, in fact states:

length, the second element is the reference count. In Delphi 2009 the representation for
reference-counted strings becomes:

-12 -10 -8 -4 String reference address
Code page Elem size Ref count length First char of string

Beside the length and reference count, the new fields represent the element size and the code
page. While the element size is used to discriminate between AnsiString and UnicodeString, the
code page makes sense in particular for the AnsiString type (as it works in Delphi 2009), as the
UnicodeString type has the fixed code page 1200.
A corresponding support data structure is declared in the implementation section of System unit as:
type

   PStrRec = ^StrRec;
     StrRec = packed record
     codePage: Word;
     elemSize: Word;
     refCnt: Longint;
     length: Longint;
   end;

But maybe the document is outdated.

-Michael

Hello FPC,

Wednesday, January 12, 2011, 9:45:47 AM, you wrote:

L> 2. Is it wrong in implementation of TSQLConnectors, which write data L> into record buffer (of TStringField) and do not convert them always into L> UTF-8 ?

Do you set the CHARSET field in your TSQLConnector to UTF-8 ? Do you define the right code page in each field of your database ?

Hans-Peter Diettrich wrote

IMO a single encoding, i.e. UTF-8, can cover all cases.

Of course you are right here, but there are some things to be considered:

In Windows (and maybe elsewhere, too) a two-Byte API (e.g. UTF-16) needs to be used, forcing lots of conversions when doing GUI applications.

_All_ beginners will use s[i] and expect to get a character without any afterthought. They will be very disappointed when not using English if they get bytes instead of characters. The count of the frustrated will be much smaller (but >Zero) when doing Widestring/Widechar and they get Words instead of Characters.

Eliminating the s[i] syntax would trash a lot of legacy code and the decent replacement (finding the correct character and moving it into a DWord in UCS4) is slow and still does not handle all the funny Unicode character-combining stuff. But the count of frustrated beginners might be further reduced.

-Michael

not all connectors supports CharSet property. When I look into sources only MySQL and IB support them (SQLite always return UTF-8 encoded ... ODBC, Postgre and Oracle ignore it)

Do you define the right code page in each field of your database ?

Yes, this is not primary question of database side, but db client library api, which is used by SQLConnector to retrieve data. For example in ODBC we use SQLGetData in LoadField method to retrieve data from odbc interface.
And for example in case of MS SQL Server character data are retrieved in current ANSI code page (in Windows of course, may be that for example in *nix data are retrieved in UTF-8 naturaly) .
(AFAIK there is no universal way how to explicitly request character encoding from ODBC interface)

So it is true, that every sql connector is mandatory write character data in UTF-8 ?
or can write in some native format (Ansi, UTF-16) ... but in this case must somewhere write additional info about actual encoding.

-Laco.

MSEgui sqldb version converts to UTF-16 from/to system encoding or utf-8 (selectable by option properties) and uses FPC 16bit UnicodeString to store string field values in the dataset, the tmsestringfield returns UnicodeString values. So one can either use utf-8 encoded databaseconnections or connections with the current system encoding.
MSEgui uses 16 bit UnicodeString everywhere, the conversion from/to system encoding is done transparently by the FPC unicode/widestring-manager if necessary.
This is a solution which works now, no additional complicated and possibly less performant codepage and encoding aware stringtype necessary...

Martin

Hello FPC,

you wrote

> L> 2. Is it wrong in implementation of TSQLConnectors, which write data > L> into record buffer (of TStringField) and do not convert them always into > L> UTF-8 ? > Do you set the CHARSET field in your TSQLConnector to UTF-8 ?

L> not all connectors supports CharSet property. When I look into sources L> only MySQL and IB support them (SQLite always return UTF-8 encoded ... L> ODBC, Postgre and Oracle ignore it)

So partially it is a lack of support in TSQLConnector. Also UTF-8 in Firebird does not work as expected due a design decision (I think).

L> Yes, this is not primary question of database side,

Oh yes it is! If you miss any of the three steps, it will fail:
1) Database field
2) SQLConnector and Client DLL/so
3) GUI

L> but db client library api, which is used by SQLConnector to L> retrieve data.

How an UTF8 SQLConnector can retrieve UTF8 data from a field defined as binary ? Client libraries have all the needed resources to handle the database, a different thing is that SQLConnector implements them and/or do it right.

L> For example in ODBC we use SQLGetData in LoadField L> method to retrieve data from odbc interface. And for example in L> case of MS SQL Server character data are retrieved in current ANSI L> code page (in Windows of course, may be that for example in *nix L> data are retrieved in UTF-8 naturaly) .

Via ODBC ?

L> (AFAIK there is no universal way how to explicitly request L> character encoding from ODBC interface)

But that's a problem of ODBC, but:

http://web.datadirect.com/resources/odbc/unicode/unix.html

L> So it is true, that every sql connector is mandatory write character L> data in UTF-8 ?

No. It is mandatory that you send/receive UTF8 to/from GUI LCL elements. In case you are using a DBF, in example which does not have encoding information, you can use the transliterate facility of dataset, but it is a bit awful.

Nice from management view, but resulting in an ugly implementation. Apart from the generic form of (internal) subroutines we still need explicit code for most variations. Also translation tables for *all* codpages must become part of every executable.

A true polymorphic string class (or equivalent) would be more performant, and would allow to add only really used codepages to the applications. Such an implementation could add another VMT pointer to the string prefix, and the UnicodeString could be implemented by a simple type cast from any (generic) string reference into a class reference.

Where is not agreement, it is fact what should be default string encoding (AnsiString($ffff) or UTF-8 or UTF-16 or UTF-32)

The default (internal) string type must be an UTF type, else losses are inevitable during (implicit) conversions. This means that SBCS AnsiString never can become the default encoding.

The default type could be made platform dependent, so that UTF-16 would be used for Windows and UTF-8 for Linux platforms. But this will cause problems with code that assumes exactly one of these encodings, and uses indexed access to characters, when such code is recompiled for a platform with a different default encoding. The introduction of another type OSString or TFileName can eliminate many implicit conversions in passing such strings to subroutines, but OTOH can cause slowdown of all other operations with that string type.

I'd ban indexed access at all, in the future, unless the default encoding is UTF-32; else the user has to accept an possible more or less significant slowdown of his code, what stands in contrast to the *intented* optimization by direct (indexed) access to the string content.

Delphi has eliminated that discussion by declaring the (default) UnicodeString fixed to UTF-16, for all targets. The only remaining question is, whether this was the best choice at all.

P.S. I still does not understand, how can things work correctly if LCL expect that all AnsiStrings (String) are UTF8Strings, byt RTL/FCL does not strictly follow this (at least in Windows) ?

Right, UTF8String should be really different from AnsiString, so that all eventually required conversions can be inserted by the compiler.

DoDi

Then you may be bound to some legacy compiler version when the stringhandling will change in some future time, as happened to Delphi users. Continued support of AnsiString type(s) is not enough, because legacy code can be broken by (eventually) required changes to "set of char", sizeof(char) and PChar, sizeof(string) as opposed to Length(string), upper/lower conversion, and many more not so obvious consequences.

See Marco's comment about data size etc.

The argument holds just fine for memory, but I don't really get the speed argument. Maybe I'm missing something.

FPC (the compiler) still uses ShortStrings wherever possible, because that was found out as the most efficient string representation. This is partially due to the ASCII encoding of source code, except for string literals. But like you, I'm not sure that this argument still holds on modern hardware.

Speed loss may occur due to:
- data shuffling in general (total byte count)
- (implied) string conversion
- indexed access to MBCS[1] strings (including UTF-8/16)

[1] All encodings of variable "character" size discourage indexed access to strings. Then "char" can have multiple meanings, as either representing the (physical) string/array *element* size, or the (logical) size of an *codepoint*. Until now most users, including you, most probably don't realize that difference between phyiscal and logical characters, and assume that sizeof(char) always is 1, and eventually that sizeof(WideChar) is 2. IMO variables of type "char" should have at least 3 (better 4) bytes in an Unicode environment, in order to maintain the correspondence between physical and logical characters. As already suggested the "packed" keyword could be applied to strings and char arrays, to definitely signal to the user that indexed access should not be used with such variables, unless a speed penalty is acceptable.

DoDi

Martin Schreiber wrote / napísal(a):

Yes, sounds logicaly to me.
Then you propose same way for TStringField ? (internaly store as UnicodeString UTF-16 and also TStringField.Text should return UnicodeString instead of String ? ... what will happens in LCL, when visual component will read UTF-16 string, will they be translated into UTF-8 automagicaly?)
-Laco.

Didn't I explain this to you and others a few times?

The database-components itself are encoding-agnostic. This means: encoding in = encoding out.

So it is up to the developer what codepage he want to use. So TField.Text can have the encoding _you_ want.

So, if you want to work with Lazarus, which uses UTF-8, you have to use UTF-8 encoded strings in your database.

If there is some strange reason why you don't want the strings in your database to be UTF-8 encoded, you have to convert the strings from the encoding your database uses to UTF-8 while reading data from the database.

Luckily, you can specify the encoding of strings you want to use for most databases. Not only the encoding in which the strings are stored, but also the encoding which has to be used when you send and retrieve data from the database. And you can set this for each connection made.

Ie: you can resolve the problem by changing the connection-string, or by adding some connection-parameter.

There's also another solution you can find on the forum and other places. You can convert the strings to UTF-8 not only when they are read from the database, but also when they are read from the internal memory. There's a hook for that.

Joost.

Almost each DB-server has a setting to specify the encoding, which has to be added to the connection-string.

So it is true, that every sql connector is mandatory write character data in UTF-8 ? or can write in some native format (Ansi, UTF-16) ... but in this case must somewhere write additional info about actual encoding.

If you add a hook that converts this data, yes. (I woudn't do that, use the database-servers functionality instead)

Joost.

It is done so in MSEgui fork of sqldb.
In case you don't know MSEide+MSEgui, it is here:
http://developer.berlios.de/projects/mseide-msegui/

... what will happens in LCL, when visual component will read UTF-16 string, will they be translated into UTF-8 automagicaly?)

It works for MSEgui where all strings are utf-16 FPC UnicodeString. It does not work for Lazarus with the utf-8 encoded ansistrings.

Martin

It cann't .
Here I am speaking about TStringField, which is IMHO designed for character data, for binary data is designed TBinaryField

Yes in UNIX world it may be so (I do not know),
but in Windows ODBC we have no such possibility AFAIK

As LCL elements are using TStringField.Text property, then this property should return UTF8String, right (not AnsiString in ANSI code page) ? If yes, then also TStringField must store internaly data in any unicode format (to not lose any characters), right ?
So it can be UTF-8, UTF-16 or UTF-32 ... in all cases we must allocate space 4*[max.number of characters in field], right ? So in what encoding are string data stored now in TStringField ?

-Laco.

The encoding you've specified. In the connection-string or some other database-server dependent setting.

Not that when you want to use UTF-16 (or 32) you have to use TWideStringFields.

Joost.

Hello FPC,

you wrote

> L> but db client library api, which is used by SQLConnector to > L> retrieve data. > How an UTF8 SQLConnector can retrieve UTF8 data from a field defined > as binary ?

L> It cann't .
L> Here I am speaking about TStringField, which is IMHO designed for L> character data, for binary data is designed TBinaryField

And a binary field is an string without encoding, collate and other text explicit attributes.

> But that's a problem of ODBC, but: > http://web.datadirect.com/resources/odbc/unicode/unix.html

L> Yes in UNIX world it may be so (I do not know), L> but in Windows ODBC we have no such possibility AFAIK

Quote from Microsoft:
"The ODBC 3.5 (or higher) Driver Manager supports both ANSI and Unicode versions of all functions that accept pointers to character strings or SQLPOINTER in their arguments. The Unicode functions are implemented as functions (with a suffix of W), not as macros. The ANSI functions (which can be called with or without a suffix of A) are identical to the current ODBC API functions."

ODBC 3.5 was launched around 2000-2001.

L> So it can be UTF-8, UTF-16 or UTF-32 ... in all cases we must allocate L> space 4*[max.number of characters in field], right ? L> So in what encoding are string data stored now in TStringField ?

In the same format the database bring them to it. Database returns a bunch of bytes and a description of that bytes.

I don't believe that PChar will be touched, because to much code that interfaces with C code depends on that. Although its declaration might not be the same then and become "PChar = PAnsiChar" instead of "PChar = ^Char" if Char is changed (currently its "PAnsiChar = PChar").

Regards,
Sven

In our previous episode, Hans-Peter Diettrich said:

Well, for starters, it doesn't cover the existing Delphi/unicode codebase.

While some hard core Ansi coders may whine about such a convention, the absence of implicit string conversions (except in external library calls) will make such applications more performant than mixed-encoding versions.

I don't see why this is the case. A current system encoding application does not do any conversion. (except for GUI output, and that can be considered negiable to the actual GUI overhead)

Why spend time in the design of multiple RTL/LCL versions, when a single version will be perfectly sufficient?

Why spent 13 years being compatible when you can throw it away in a second?

In our previous episode, Sven Barth said:

Current Delphi _does_ regard char as equivalent lowlevel type to string. So whatever you choose as string (8 or 16-bit), pchar will match it by changing to pansichar or pwidechar

Martin Schreiber kirjoitti maanantai 10 tammikuu 2011 19:22:49:

Without studying your code myself I guess you had to make many utility functions and classes yourself for UTF-16 ?
Even the normal TStringList doesn't work.

Juha

Hans-Peter Diettrich wrote

Until now most users, including you, most probably don't realize that difference between phyiscal and logical characters, and assume that sizeof(char) always is 1

Oh, I'm aware of it. But to date, I haven't had to really deal with it in Delphi or FPC. My use of strings is either ancient legacy (from TP/BP days) where I simply changed all references to "string" to "shortstring" or low level Windows API code, where I'm dealing with PChar.

I find these discussions fascinating, but as they say in the southern US, I don't have a dog in this hunt. Whatever the decision, I'll probably continue to use shortstring.

Jeff.

AFAIK Delphi did so, with PChar=PWideChar now. For external Ansi interfaces PAnsiChar has to be used.

Please note that the WinAPI headers are configurable for either Ansi or Unicode, changing TCHAR etc. as appropriate.

DoDi

Because it's bound to UTF-16? That's not a problem, because WideString will continue to exist, and according conversions are still inserted by the compiler.

When system encoding changes with the target platform, indexed access to such strings can lead to different results. Unless the compiler can read the coder's mind...

> Why spend time in the design of multiple RTL/LCL versions, when > a single version will be perfectly sufficient? Why spent 13 years being compatible when you can throw it away in a second?

It's sufficient to throw away what's no more needed :-)

DoDi

Correct. MSEgui has a complete development environment for UnicodeString with an own set of lists, streams, file and directory functions and the like.

Martin

Didn't I explain this to you and others a few times?

;-) If so, then please excuse me

So this is answer, which i have looked for:
"In Lazarus TStringField MUST hold UTF-8 encoded strings."

But I guess (I have theory), that in time, when Borland introduced TStringField, the design goal was:
TStringField was designed for SBCS (because DataSize=Size+1) string data encoded in system ANSI code page
and
TWideStringField was designed for DBCS widestring (UTF-16) character data

May be, that I was mistaken by this view.
(or may be, that there is different approach in Delphi ("no agnostic") and different in FPC ("agnostic")?)

If there is some strange reason why you don't want the strings in your database to be UTF-8 encoded,

SQL Server does not support UTF-8 (AFAIK)
SQL Server provides non-UNICODE datatypes - char, varchar, text and UNICODE (UCS-2) datatypes - nchar, nvarchar, ntext

Yes, it is true for example for MySQL or Firebird ODBC driver, but for SQL Server or PostgreSQL ODBC driver there are no such options (but PostgreSQL ODBC driver exists in ANSI and UNICODE version) SQL Server ODBC driver supports "AutoTranslate", see: http://msdn.microsoft.com/en-us/library/ms130822.aspx "SQL Server *char*, *varchar*, or *text* data sent to a client SQL_C_CHAR variable is converted from character to Unicode using the server ACP, then converted from Unicode to character using the client ACP."

Thanks for your patience
-Laco.

Joost van der Sluis wrote / nap�sal(a):

ok.
But then there is problem in buffer size allocated for TStringField (ftString), does not ?
See please at bug report: http://bugs.freepascal.org/view.php?id=17376 There is described situation with SQLite (TSQLite3Connectin) , which returns UTF-8 strings, so there is no problem in encoding, but problem is in fact, that for char(n),varchar(n) fields is created TStringField with Size=n and in record buffer is also allocated space with Size+1, where n is number of characters (not bytes). So truncation of data occurs, when writting UTF-8 encoded string into record buffer. So IMHO there must be:
1. allocated space in record buffer in size 4*TFieldDef.Size+1 (and so on) or
2. redefine meaning of Size property (as number of bytes not characters) and create fielddefs with Size*4
hm, according to
http://docwiki.embarcadero.com/VCL/XE/en/DB.TStringField.Size is Size number of characters
but according to http://docwiki.embarcadero.com/VCL/en/DB.TFieldDef.Size is Size number of bytes in underlaying database

but TField is created from TFieldDef and TField.Size=TFieldDef.Size ... so isn't it curious ?

Not that when you want to use UTF-16 (or 32) you have to use TWideStringFields.

So TWideStringField is "no-encoding-agnostic" field (is it designed to be everytime UTF-16 encoded) ?

-Laco.

But this approach will require changes in packages/odbc/src/odbcsql.inc
like, does not ?:
-    pointer(SQLGetData) :=
GetProcedureAddress(ODBCLibraryHandle,'SQLGetData');
+    pointer(SQLGetData) :=
GetProcedureAddress(ODBCLibraryHandle,'SQLGetDataW');
And I do not know how it affect compatibility for example in UNIX or if
all ODBC drivers support this functionality.

But also in this case we will get UTF-16 widestrings (in Windows) not UTF-8, does not ?

-Laco.

Hello FPC,

you wrote

> ODBC 3.5 was launched around 2000-2001.

L> But this approach will require changes in packages/odbc/src/odbcsql.inc
L> like, does not ?:
L> -    pointer(SQLGetData) :=
L> GetProcedureAddress(ODBCLibraryHandle,'SQLGetData');
L> +    pointer(SQLGetData) :=
L> GetProcedureAddress(ODBCLibraryHandle,'SQLGetDataW');
L> And I do not know how it affect compatibility for example in UNIX or if
L> all ODBC drivers support this functionality.

Most probably it needs a "flag" to indicate that the ODBC must work in Unicode, and then dynamic link to *W functions if this flag is set. I think ODBC drivers since 2002+/- should have this set of APIs, but I had never used ODBC in my life... :)

L> But also in this case we will get UTF-16 widestrings (in Windows) not L> UTF-8, does not ?

That's not important, you get unicode in the specified by the API format, then SQLConnector fills information in the expected target format (WideString, UTF8String over AnsiString, Raw bytes...).

In our previous episode, Hans-Peter Diettrich said:

That is DIY compatibility, or, in other words, no compaibility.

Widestring will also grind the application to a halt due to being COM based on Windows.

You don't have to. The Delphi model provides a stringtype for the system encoding, and then as such all strings from the system can be labeled. With other stringtypes, the necessary conversions can be edited.

Likewise, e.g. win32 console routines can be labeled with OEMString. (Since windows uses a different default encoding for the console)

The previous message from Jeff shows that even shortstring is still in major production use. Nothing is unused and can be clipped without a long winded transition, or Delphi 2009 like painful breaks.

Moreover, these discussions are useless since you know as well as I do that no one stringtype will ever satisfy everybody. So IMHO it is time to take the consequences from the 500 posts on this subject on the unicode subject on this and other FPC/Lazarus lists and start thinking in solutions to manage that, instead of reiterating the "one type to rule them all" mantra ad infinitum.

It seems, that Driver Manager automaticaly performs conversions on non-unicode driver. See
http://web.datadirect.com/resources/odbc/unicode/odbc-driver.html ( and
http://web.datadirect.com/resources/odbc/unicode/char-background.html ) "If the driver is a non-Unicode driver, it cannot understand W function calls, and the Driver Manager must convert them to ANSI calls before sending them to the driver."

Also it seems to me, that when you call ANSI version of ODBC API functions, then you receive data in ANSI encoding. If it is so, then it is always safe use ansitoutf8() (or UTF8Encode()) on receved data.

In Windows we get UTF-16, in Linux/UNIX we get UTF-8 So it is so, that in Windows is "widestring=UTF-16" and in Linux/UNIX is "widestring=UTF-8" string ?
(So there is different meaning of widestring type on different OSeses ? I am only Windows developer, so I have no understanding of others OSeses ;-))

-Laco.

Hello FPC,

Thursday, January 13, 2011, 1:01:57 PM, you wrote:

L> Also it seems to me, that when you call ANSI version of ODBC API L> functions, then you receive data in ANSI encoding. L> If it is so, then it is always safe use ansitoutf8() (or UTF8Encode()) L> on receved data.

No, because ANSI is not UTF-* so any char outside of your ANSI codepage will be discarded or generate and exception, or whatever the developer decides (driver developer).

L> In Windows we get UTF-16, in Linux/UNIX we get UTF-8

If you use ANSI calls in linux you will receive UTF8 because in most linux the ANSI page is UTF8. If you call the *W APIs you MUST receive WideString or PWideCharArray or alike, or the API will not work.

Not entirely true. You could also choose to bind the fields to some Lazarus-components manually, not using the db-components. (Tedit.Text := convertFunc(StringField.Text)) Or you can add a hook so that the .text property always does a conversion to UTF-8. First option can be used if you use a mediator or view. Second options I woudn't use.

You have to be really careful in what you type, when you are writing about encodings. The above is nonsense, because of a very tiny mistake.

If you compare DBCS widestring with UTF-16, you can also compare a stringfield with UTF-8. Exactly the same problem. (A character can be made up from more then one UTF-8 or UTF-16 codepoint)

But TStringField's datasize by default is indeed Size+1. So if you use it t store UTF-8, you have to define the size as four times the field-size given by the database. Note that this is done in some cases.

May be, that I was mistaken by this view. (or may be, that there is different approach in Delphi ("no agnostic") and different in FPC ("agnostic")?)

No, Delphi does the same. Only newer Delphi versions have a string-type which contains the used encoding (details can be found in this thread), so can do some conversions for you. But that has nothing to do with the database-code. Also, you don't need it. People all over the world have used older Delphi versions all the time... (But offcourse, it's easier now)

> If there is some strange reason why you don't want the strings in your > database to be UTF-8 encoded, SQL Server does not support UTF-8 (AFAIK)

Rofl. You mean that Microsoft SQL Server can't handle unicode completely? If they say that in an advertisement they can forget that any big commercial client will choose their product...

SQL Server provides non-UNICODE datatypes - char, varchar, text

ie: TStringField

and UNICODE (UCS-2) datatypes - nchar, nvarchar, ntext

ie: TWideStringField.

What does this have to do with your problem? Nothing. Only things what matters is what encoding is used while communicating with the client. (Which you can set)

Then that option has to be added. I think it's already possible but you simply don't know how. (Sql-Server is ODBC only, so that one is fixed. For firebird there's a 'serverencoding' parameter, or something like that. Postgres also has some setting.

(but PostgreSQL ODBC driver exists in ANSI and UNICODE version)

I saw that in an earlier message, but also this has nothing to do with your problem. You only need the different calls when you want to use UTF-8 in your fieldnames. (Or, and this one was tricky, in the connection-string. But this was more then a year ago.)

This is what you use when you set the encoding when you connect to the client. The solution to all your problems. As explained three times, in this message alone.

In fact it's simple: incoming data=outgoing data.

If you need UTF-8 encoding for the outgoing data (direct access to Lazarus controls) you have to select UTF-8 at the input. That's always more efficient than converting the data to/from any other encoding.

And, luckily, you can instruct the Database-server which encoding to use when it's communicating with the outer world. So your problem is solved.

Now, if you also choose UTF-8 as the Database-server field encoding (the encoding the data is stored in) there's no conversion that's a bonus, not a necessity.

Joost.

Yes, those are the possible solutions. Good thing about the second option, is that a user can do that on his own if he wants to use UTF-8, just create persistent fields with a field size of 4*the amount of characters. I'm not sure if we have to change this. It's a problem the programmer has to deal with, I think...

Yes, that's indeed the problem. But there's also the .DataSize property, so we could use that.

Maybe... if the pressure on the bugtracker gets too high, I'll bow and change this. (I think 25% of all existing db bugs are related to this and people who do not understand anything about encodings.)

No. It's designed to contain an array of two-bytes records. In fact you can use it to store UCS-2 data, but not UTF-16. Same story as with ansi/UTF-8.

Joost.

This has only effect on the passed parameters to the ODBC-functions. ie: the field-names, table names and such.

Joost.

Hello FPC,

Thursday, January 13, 2011, 2:58:30 PM, you wrote:

JvdS> Then that option has to be added. I think it's already possible but you JvdS> simply don't know how. (Sql-Server is ODBC only, so that one is fixed. JvdS> For firebird there's a 'serverencoding' parameter, or something like JvdS> that. Postgres also has some setting.

Are you aware about the Firebird Field(UTF8) and SQLConnection CharSet(UTF8) problem ?

Table X
-------
FieldTestName Varchar(5) UTF8

IBConnection
-------
CharSet = UTF8

//Source code file is UTF8 encoded.
DataSource.FieldByName('FieldTestName').AsString='ÑÑÑÑÑ';

This raises an exception because the string is 10 bytes and the field only allow 5 chars.

I think I had read this comment many months ago and answered as "won'n fix" for Interbase compatibility. Am I wrong ?

See my mail to Lacak about the two options to solve this.

Joost.

Hello FPC,

you wrote

> Are you aware about the Firebird Field(UTF8) and SQLConnection > CharSet(UTF8) problem ?

JvdS> See my mail to Lacak about the two options to solve this.

I think the problem is different and can be solved without any compatibility problem or at least easily detectable.

Anyway I'll write a possible patch and this way test if that's a viable solution and if it "works" send for evaluation.

Thank you.

I still don't understand the problem :-(

Widestring will also grind the application to a halt due to being COM based on Windows.

How that?

Indexed string access produces other results for Ansi and UTF-8 system encoding. Such code is not portable, and the data (ini files) are not, too. Allowing for UTF-8 as the system encoding will frustrate Windows users (dunno whether Windows allows for such a system encoding), and Linux users are frustrated when UTF-8 is disallowed.

Only solution: using OS encoding restricts the code to run on a single machine only, or on similarly configured machines.

The group of users, which accept this restriction, will be happy with a single AnsiString type and no implicit conversions. Without implicit conversions such a string type can hold UTF-8 as well.

Likewise, e.g. win32 console routines can be labeled with OEMString. (Since windows uses a different default encoding for the console)

This either implies OEM encoding as the system encoding of Win32 console applications, or the use of multiple codepages, as before. But IMO Win32 console also implements a "W" interface, so that it's up to the user to use whatever is more appropriate for his code.

The RTL has to distinguish between system-wide "filesystem" and "GUI" encoding, in file handling (CreateFile...).

It's all about the well known dilemma:
- force (possibly many) implicit conversions, or
- supply multiple RTL/LCL versions, or
- break legacy user code by moving to a different (but again unique) string type.

The discussion is only about the pros and cons of the various possible solutions. I.e. it should reveal the critical cases and consequences, that have to be considered and handled in every implementation.

The implementation can choose any model. Different models can be implemented as well, so that the final decision about the new standard can be delayed, until the models can be tested in real world applications.

One model has already been implemented: UTF-8. It may need some adds/improvements, like a *hard* separation of AnsiString from UTF8String, and nothing has to be thrown away.

DoDi

Oh come on -.-

There are some days on which I really dislike the developers of Delphi...

Regards,
Sven

WideString on Windows has no reference counting, thus everytime a WideString is assigned it needs to be copied.

Nearly all Windows API functions only allow single byte encodings or UTF-16. The only functions that I'm aware of, that can use UTF-8 encoding is the console input/output API (if the codepage is set to UTF-8) [and also file I/O APIs, but they don't assume any encoding].

Regards,
Sven

Sven Barth wrote

WideString on Windows has no reference counting, thus everytime a WideString is assigned it needs to be copied.

Not exactly true. widestring is com marshaled and thus has reference counting on the com level. afaik .
As long as your memorymanager is com marshaled too, that is. And since most pascal memory manager versions do not support com directly, it goes wrong in a big way.
I once wrote a simple com memory manager to test this. Performance stays sh*t, but strings seem to be counted, not copied.
If you use coTaskMemAlloc, coTaskMemFree,CoTaskMemRealloc in your memory manager you will see what I mean.
At least it comes close, but slow it will stay.

Another already implemented model is utf-16 UnicodeString in MSEgui. Needs no changes in Free Pascal compiler.

Martin

I see here these possible problems/disadvantages:
1. In many cases (dynamic build or ad-hoc queries) is creating persistent fields not very effective (or complicated)
2. allocation of space in record buffer is based on TFieldDef objects (see TCustomBufDataset.GetFieldSize) and TFieldDef object are created by TSQLConnectors in AddFieldDefs method, so setting Size in persistent field does not solve whole problem, because each SQLConnector must set Size also
3. in TStringField is Size used also to determine default DisplayWidth (for TDBGrid) and in Delphi also for setting MaxLength in TDBEdit (so here we can see, that Size is used like max.number of characters rather than bytes)
4. incompatibility of Delphi (if we reclasiffy Size as number of bytes not characters)

So I would prefer 1st way (increase buffer size, may be if we will support only BMP then 3*Size+1 will be sufficient) So Size remains as "character length"

Yes, but DataSize is defined only in TField not in TFieldDef class If we will add DataSize property also for TFieldDef class and overload TFieldDef.Create method with additional parameter DataSize, then SQLConnector would specify both information: "character size" (for displaying purposes) and "byte size" (for buffer purposes)

Yes I understand now.

Laco.

IMHO most of gui database applications use controls like TDBGrid or TDBEdit so they should display correct values by default without extra coding (or at least provide "some standardized support" ... )

Completely not, but only UCS-2 (no UTF-8)

Yes, but ODBC driver returns data in ANSI codepage (no possibility to force them return UTF-8)
This I can fix by patch in TODBCConnection LoadField like this: (so I convert to UTF-8 in connector method, when driver is unable return UTF-8)

    begin
      Res:=SQLGetData(ODBCCursor.FSTMTHandle, FieldDef.Index+1,

SQL_C_CHAR, buffer, FieldDef.Size, @StrLenOrInd);
+ if CharSet='ANSI' then //hack for Microsoft SQL Server
+ StrPLCopy(buffer, UTF8Encode(PChar(buffer)), FieldDef.Size);

    end;

Yes, in this case ODBC driver returns data in UCS-2, this data are written into "WideString buffer", which seems correct, but in DBGrid are displayed "?" instead of characters with diacritical marks (IMHO because widestringmanager in Windows converts WideString to ANSI string , not UTF-8 string).
This can be fixed by using OnGetText method of field: aText:=UTF8Encode(Sender.AsString);
Which is not user friendly, because requires "hacking in user code" in every TWideStringField in every TSQLQuery
It can be also fixed in fields.inc:
function TWideStringField.GetAsString: string;
begin
+{$IFDEF WINDOWS}
+ Result := UTF8Encode(GetAsWideString);
+{$ELSE}

  Result := GetAsWideString;

+{$ENDIF}
end;

So what is the expected encoding of data written into TWideStringField ... or is there way how to get correct results id DBGrid without above mentioned workarounds ?

Yes, but as I wrote such possibility does not exists with Microsoft SQL Server (and also I think Access)
(it seems, that Microsoft does not like UTF-8 and prefers UTF-16 (UCS-2))

And, luckily, you can instruct the Database-server which encoding to use when it's communicating with the outer world. So your problem is solved.

When it is possiblem then yes.

Now, if you also choose UTF-8 as the Database-server field encoding (the encoding the data is stored in) there's no conversion necessary at all.

Yes if DB supports UTF-8

-Laco.

I'm not so sure of that. AFAIR the field exists, but it's unused or reserved for shared memory management.

Of course the requirement, that a BSTR has to reside in shared memory, discourages the use of exactly that type for stringhandling inside an application.

I only wanted to prevent the introduction of another "UTF16String" type, in addition to WideString, BSTR (WinAPI) and UnicodeString (Delphi). Conversion-wise WideString/BSTR and (other) UTF-16 strings are equivalent.

Nearly all Windows API functions only allow single byte encodings or UTF-16. The only functions that I'm aware of, that can use UTF-8 encoding is the console input/output API (if the codepage is set to UTF-8) [and also file I/O APIs, but they don't assume any encoding].

And the conversion functions of course (MBCStoWStr...).

DoDi

Yes, if you use the set of memory allocators I mentioned the field *will* be used. COM marshalling.
No com, no count. simple as that. It is unused, because the memory manager doesn't use it.
com is not implemented, unless you use a com based memory manager. No com, no reference count.