Hi,

Large parts of the returning discussions about code page-aware AnsiStrings are related to the fact that many people don't how they work. For this reason I've created an overview that explains the rules that are followed by the RTL/compiler at http://wiki.freepascal.org/FPC_Unicode_support (only Sections 1 to 3; 4 and later are older and mostly either incomplete or wishful thinking).

Note that this is not a discussion document, and apart from some notes about potential future changes, errors or unknown major Delphi- incompatibilities, the behaviour as described there will not change. The behaviour has been defined to maximise both backward compatibility with existing FPC code (written for FPC 2.6.x and earlier) and with Delphi 2009+. Automagic string types that solve world hunger are outside the scope.

Jonas

This in fact is in the paragraph " Old/obsolete sections", but it does not seem to be mentioned in any "current" paragraph:

"Roadmap of RTL Unicode support with UnicodeString":
- TStrings Not implemented There is no UnicodeString version of TStrings
- TStringList Not implemented There is no UnicodeString version of TStringList

This is something just discussed and IMHO very critical, as (IMHO) a Delphi XE compatible implementation is not appropriate for flexible use and portability for multiple targets.

-Michael

That's because it is a description of what already exists and how it behaves, as explained in the introduction on that page and in the message I sent to this list. It is not the umpteenth iteration of a proposal of what a code page-aware classes unit should look like, nor an attempt to address every person's wish list written down previously on that page.

Jonas

Jonas Maebe wrote

... nor an attempt to address every person's wish list written down previously on that page.

Yep. And as (pure) Delphi XE compatible behavior (at least in my opinion) is not what is desirable for a portable language/rtl, while generally Delphi compatibility is a major request, this in fact obviously is a problematic issue.

-Michael

Great!

What is this crap:
http://wiki.freepascal.org/FPC_Unicode_support#FPC_Unicode_support ?

;)

Mattias

Thanks :-)

The chapter numbers are missing from the headings?

On my Win98 VM this page is not accessible:

>

Error 403

We're sorry, but we could not fulfill your request for /FPC_Unicode_support on this server.

You do not have permission to access this server.

Your technical support key is: 02f1-94ac-17f4-e8c8 <<

What's wrong?

On my Win8 machine the page and server is accessible, of course.

DoDi

Mattias Gaertner wrote

What is this crap: http://wiki.freepascal.org/FPC_Unicode_support#FPC_Unicode_support ?

It's under the header "Old/obsolete sections" and as mentioned above, that's "incomplete or wishful thinking". I didn't want to delete any existing content for now, but yes, what's mentioned there but not in the current Sections 1 to 3 is not going to happen. Well, there is an UCS4String type in the RTL (like in FPC 2.6.x), but it's just a dynamic array of 0..$10ffff and not part of the regular string handling.

Jonas

Jonas Maebe wrote

For this reason I've created an overview that explains the rules that are followed by the RTL/compiler at http://wiki.freepascal.org/FPC_Unicode_support

"it is best to save the source code in UTF-8 with a BOM."

Is this recommendation is compatible with Delphi7, Delphi 2007 ? the idea is to be able to share same code, same units between Delphi and FPC.

This is the full quote: "it is best to include either an explicit {$codepage xxx} directive or save the source code in UTF-8 with a BOM" (which I've now clarified a bit on the wiki page). Both are equally good as far as FPC is concerned.

Is this recommendation is compatible with Delphi7, Delphi 2007 ?

Delphi doesn't know the {$codepage xxx} directive, but you can put it between {$ifdef fpc}. I don't know whether it automatically detects UTF-8-encoded files (or even handles them at all), especially these older versions. If it doesn't, you can still save your source code in any code page and add a {$codepage xxx} statement for FPC. Additionally, I forgot to mention that you can also specify the used code page on the command line with FPC, via the -Fc parameter.

Jonas

Maybe it could be moved to the discussions page?

BigChimp (Reinier?) has already done that in the mean time.

Jonas

http://wiki.freepascal.org/FPC_Unicode_support (only Sections 1 to 3; 4 and later are older and mostly either incomplete or wishful thinking).

Just a note on RawByteString concatenation:

Delphi concatenates RawByteStrings to the dynamic encoding of the *first* string, the appended strings eventually are converted before concatenation. Special handling of strings with the same encoding is not required.
I.e. the result is *not* always a CP_ACP string, as documented in the wiki.

Please adjust the implementation accordingly, it makes the RawByteStrings much more useful. The handling of automatic conversions may be unified in general, when concatenated strings are assigned to a target of a known encoding; in this case the target encoding can be used for the result, instead of the encoding of the first string, the remaining concatenation process can be the same.

On OEMString:

CP_OEM (=1) works differently from CP_ACP (=0). Variables of type AnsiString(CP_OEM) will always have dynamic encoding CP_OEM, no substitution to a specific OEM codepage. CP_ACP strings instead have a dynamic encoding of the current DefaultSystemCodepage.

DoDi

"Hans-Peter Diettrich" wrote

Delphi concatenates RawByteStrings to the dynamic encoding of the *first*

string, the appended strings eventually are converted before concatenation. Special handling of strings with the same encoding is not required.

I.e. the result is *not* always a CP_ACP string, as documented in the

wiki.

Would you be so kind to provide a simple test case for this? :)

Regards,
Sven

function test(a,b: RawByteString): RawByteString;
begin
   Result := a+b;
   WriteLn(StringCodePage(Result));
end;
var
   u: UTF8String;
   a: AnsiString;
begin
   a := 'äöü';
   u := 'üöä';
   test(a,u);  //CP_ACP
   test(u,a);  //UTF-8
end;

It looks to me, however, that no conversion occurs at all! The strings are only concatenated as they are.
Same for a concatenation of (global) RawByteString variables.

This of course were not a desireable implementation :-(

DoDi

I'm inclined to say "of course", because in your "test" function you are concatenating two RawByteStrings which - by definition - don't do any conversion.

Regards,
Sven

I cited what I've been told in the EMBT groups - that a conversion is made when required. Everything else doesn't make sense.

DoDi

And your example proves otherwise...

Regards,
Sven