Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-pascal / 2016-05

Summary of Unicode Strings Debate

4 messages · 3 people ·

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Mazo Winst

Hello,

A summary of what i learned from the Unicode String debate:

1 - FPC 3 introduces code page aware strings

2 - FPC 3 updates the RTL to provide a better support for code page aware strings

3 - The dynamic behavior of the string type regarding to the platform was not introduced by FPC 3. In the previous versions of FPC, the string type depends on the platform too.

4 - However, there is a key difference among FPC 3 and the previous versions: in the FPC 3, the compiler do perform automatic conversions in certain circumstances (two ansistring variables with diferent codepages). I think these automatic conversions can potentially be the source of unexpected data corruption.

5 - The automatic conversion is a desirable feature. IMHO, the definitive solution would be to stop the unpredictable dynamic behavior of the String type following the same path that Delphi followed: addopt the same unicode string code page on all platforms.

6 - A workaround when developing a cross-platform app is to use the Lazarus Unicode Support
<http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus>;

Best regards

Reply by email

Michael Van Canneyt
to Mazo Winst
Mazo Winst wrote 12 lines
Hello, A summary of what i learned from the Unicode String debate: 1 - FPC 3 introduces code page aware strings 2 - FPC 3 updates the RTL to provide a better support for code page aware strings 3 - The dynamic behavior of the string type regarding to the platform was not introduced by FPC 3. In the previous versions of FPC, the string type depends on the platform too.

The string type does not depend on the platform.

The only thing that depends on the platform is the default encoding used in an AnsiString.

But that was true in FPC 2.6.4 as well:
On windows that would have been CP_ACP, on unix UTF8. Now this is simply settable.

(Jonas can confirm/deny this)

Show quoted text 5 lines
4 - However, there is a key difference among FPC 3 and the previous versions: in the FPC 3, the compiler do perform automatic conversions in certain circumstances (two ansistring variables with diferent codepages). I think these automatic conversions can potentially be the source of unexpected data corruption.

In such cases the codepage is determined by the declared code page of the result string. (Jonas should be able to confirm/deny this).

If the codepages cannot be converted correctly, then the data may be corrupted.

Show quoted text 8 lines
5 - The automatic conversion is a desirable feature. IMHO, the definitive solution would be to stop the unpredictable dynamic behavior of the String type following the same path that Delphi followed: addopt the same unicode string code page on all platforms. 6 - A workaround when developing a cross-platform app is to use the Lazarus Unicode Support <http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus>;

Huh ? What kind of statement is this ?
What do you think Lazarus uses under the hood ?
Last I looked, it was FPC :-)

It seems to me that most people simply do not understand fully what is happening, and as a result, a lot of misunderstandings are abound.

Michael.

Reply by email

Jonas Maebe
to Michael Van Canneyt

Michael Van Canneyt wrote on Wed, 11 May 2016:

Show quoted text 7 lines
On Wed, 11 May 2016, Mazo Winst wrote: > 3 - The dynamic behavior of the string type regarding to the platform was > not introduced by FPC 3. In the previous versions of FPC, the string type > depends on the platform too. The string type does not depend on the platform.

Indeed, it only depends on the syntax mode and {$h+/-} setting (and, since FPC 3.0, {$modeswitch unicodestrings}

Show quoted text 5 lines
The only thing that depends on the platform is the default encoding used in an AnsiString. But that was true in FPC 2.6.4 as well: On windows that would have been CP_ACP, on unix UTF8. Now this is simply settable.

On Unix it was whatever is configured in the environment via the LC_ALL/LC_CTYPE/LANG environment variable, and only if a widestring manager is used (such as cwstring). Otherwise it defaulted to ASCII. None of this changed in FPC 3.0 indeed.

The rest of what you said is correct.

Jonas

Reply by email

Mazo Winst
to Michael Van Canneyt: The string type does not depend on the platform. The only thing that depends on the…

2016-05-11 11:44 GMT-03:00 Michael Van Canneyt :

Show quoted text 24 lines
On Wed, 11 May 2016, Mazo Winst wrote: Hello, > > A summary of what i learned from the Unicode String debate: > > 1 - FPC 3 introduces code page aware strings > > 2 - FPC 3 updates the RTL to provide a better support for code page aware > strings > > 3 - The dynamic behavior of the string type regarding to the platform was > not introduced by FPC 3. In the previous versions of FPC, the string type > depends on the platform too. > The string type does not depend on the platform. The only thing that depends on the platform is the default encoding used in an AnsiString.

When i said "dynamic String type behavior" i mean "dynamic definition of the String encoding".

Show quoted text 19 lines
5 - The automatic conversion is a desirable feature. IMHO, the definitive > solution would be to stop the unpredictable dynamic behavior of the String > type following the same path that Delphi followed: addopt the same unicode > string code page on all platforms. > > 6 - A workaround when developing a cross-platform app is to use the > Lazarus > Unicode Support > <http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus>; > Huh ? What kind of statement is this ? What do you think Lazarus uses under the hood ? Last I looked, it was FPC :-) It seems to me that most people simply do not understand fully what is happening, and as a result, a lot of misunderstandings are abound.

The Lazarus support for Unicode provide some units which allows us to adopt the same encoding on all platforms (UTF-8). When i say "allow us to adopt the same encoding on all platforms" i mean "they take care of many issues that arises when we adopt the UTF-8 on all platforms. I suppose that adopt UTF-8 on Windows is not as simple as set the system code page through SetMultiByteConversionCodePage.
I suppose there are several side effects when we do that. In this regard, the Lazarus unicode support helps a lot."

Best regards

Reply by email