Hi,

In the tiOPF project the v3 branch is a Delphi Unicode String enabled version of tiOPF, thus it only supports Delphi 2009/2010 and newer. A while back I had some discussion with Michael van Canneyt, and he believes FPC is sufficiently compatible to Delphi 2010 and later, so the tiOPF3 port should be possible.

So today I started taking a look to see what would be involved in getting tiOPF3 compatible with FPC. Off the bat, I have a lot of compiler errors with latest FPC 2.7.1 but I'll push ahead and see how far I get.

I haven't been following the latest 2.7.1 development in detail, so I have a couple of questions... all relating to FPC 2.7.1 (trunk) only:

0) Am I jumping the gun here, and FPC is not nearly compatible enough to port a Delphi 2010, XE, XE2 project to FPC 2.7.1?

1) Is it correct that String AnsiString any more?

2) If true, what is String an alias of? UnicodeString, WideString, something else?

3) If false, what must I enable/toggle to say that String must now be a Delphi unicode-type string.

4) What Unicode encoding is used? UTF-8 or UTF-16?

5) Is it only the compiler that has unicode string type support. Has anything been done or started in the RTL?

Regards,
- Graeme -

In our previous episode, Graeme Geldenhuys said:

0) Am I jumping the gun here, and FPC is not nearly compatible enough to port a Delphi 2010, XE, XE2 project to FPC 2.7.1?

If it relies heavily on unicodestring, IMHO yes.

My guess is that you mistook a comment that confirmed the base implementation of the variable stringtype to be compatible for a confirmation that the whole project is ready.

2) If true, what is String an alias of? UnicodeString, WideString, something else?

ansistring or shortstring, depending on {$H+}. What to do with the D2009+ change in this regard has been discussed, but opinions varied.

4) What Unicode encoding is used? UTF-8 or UTF-16?

As said, undecided.

If I had to guess, it will end up D2009 compatible (UTF16 everywhere), causing a similar compatibility break as Delphi somewhere.

5) Is it only the compiler that has unicode string type support. Has anything been done or started in the RTL?

The helpers for the types have been done mostly. Nothing else.

The few things I know about are:

Graeme Geldenhuys wrote

1) Is it correct that String AnsiString any more?

Well, it never was. At least "string" could also be a shortstring, maybe other strings too meanwhile (I don't know).

3) If false, what must I enable/toggle to say that String must now be a Delphi unicode-type string.

I don't know what a Delphi unicode-type string is exactly.

4) What Unicode encoding is used? UTF-8 or UTF-16?

AFAIK UTF-8 is the prefered unicode string as it is used in the IDE and also many libraries (but not all). I am not sure what is planed for the future though.

5) Is it only the compiler that has unicode string type support. Has anything been done or started in the RTL?

The RTL still uses the ancient 255 char one-byte array. This works ok if paths are not longer than 255 chars and when you convert all strings to the (one-byte) string type that is used by the API of the OS (ANSI for Windows and I think UTF-8 for Linux).

Hi,

Marco van de Voort wrote

If it relies heavily on unicodestring, IMHO yes.

Well, there is a clear distinction being made between AnsiString and String in many classes.

For example:

  TtiCompressAbs = class(TObject)
  public
    ....
    function  CompressString(  const AFrom : string; var ATo :
string): Extended; overload; virtual; abstract;
    function  CompressString(  const AFrom : AnsiString; var ATo :
AnsiString)  : Extended; overload; virtual; abstract;
    ...
  end;

This gives a compiler error in 2.7.1

My guess is that you mistook a comment that confirmed the base implementation of the variable stringtype to be compatible for a confirmation that the whole project is ready.

The conversation was about many D2009+ features, but I think the main feature tiOPF v3 uses is the Unicode string.

OK, all projects I work on always have {$H+} defined, so for me (in 2.6.1 and earlier), String always means AnsiString.

I guess from the rest of your reply, I am jumping the gun here... Clearly 2.7.1 is not nearly ready yet for such a Delphi port.

Thanks for all the info.

Hi,

I guess I was a bit vague. All projects I work on and see these days use {$H+} by default. In fact I don't know why FPC's hasn't changed {$mode objfpc} to default to String = AnsiString a long time ago, but that is another battle to fight another day.

> 3) If false, what must I enable/toggle to say that String must now be a > Delphi unicode-type string. I don't know what a Delphi unicode-type string is exactly.

Please read the Delphi 2009 announcements. It was the biggest code breaking change in Delphi in years! I have no idea how you could have missed that. Anyway, String = Unicode UTF-16 in Delphi 2009 and newer. If you want the old behaviour, you have to explicitly define each string variable as AnsiString.

AFAIK UTF-8 is the prefered unicode string as it is used in the IDE and also

This is the FPC mailing list. The only IDE FPC has is the Free Pascal Text IDE, and that definitely doesn't use UTF-8. ;-)

Anyway, thanks for your response. I'll hold off on the tiOPF v3 port for now. There are too many undecided factors in FPC 2.7.1 at the moment.

Graeme Geldenhuys wrote on za, 18 aug 2012:

1) Is it correct that String AnsiString any more?

Only in {$mode delphiunicode} (which is what you should use if you want to compile code written for a Delphi version in which string=unicodestring).

2) If true, what is String an alias of? UnicodeString, WideString, something else?

The same as in Delphi: unicodestring.

4) What Unicode encoding is used? UTF-8 or UTF-16?

The same as in Delphi: utf-16

5) Is it only the compiler that has unicode string type support. Has anything been done or started in the RTL?

Nothing other than what's required for compiler support, and possibly some small extra's.

Jonas

Only in {$mode delphiunicode} (which is what you should use if you want to compile code written for a Delphi version in which string=unicodestring).

Is this hidden gem announced or documented somewhere? Is there a -M<x> compiler switch to set it as default language mode? FPC doesn't list it yet.

Thanks, Ludo

Hi,

OK, that would help. I'll try that now in tiOPF v3. When specifying that mode, do I still need to specify {$H+}? What happens when I don't specify {$H+}?

> 4) What Unicode encoding is used? UTF-8 or UTF-16? The same as in Delphi: utf-16

Wasn't there lots of "votes" from many that string is UTF-8 encode under Linux, Unix, MacOSX, and UTF-16 under Windows? Thus avoiding any conversion speed penalties on all non-Windows platforms. Or is defaulting to UTF-16 just the first step of implementing Unicode support, and UTF-8 will follow for all non-Windows platforms later?

I've enabled that new compiler mode... Any idea what the following warning means?

-----------------
/home/graemeg/devel/tiOPF3/src/Core/tiDefines.inc(42,4) Warning: Current system codepage "0" is not available for the compiler. Switching default codepage back to "28591".

-----------------

I don't know whether Michael documented it already, but it's not announced, because there's no guarantee yet, that it will work as advertised.

E.g. consider the following example:

{$mode delphiunicode}
type
   TMyStringList = class(TStringList)
     function Add(const aText: String): Integer; override;
   end;

This will currently give a compile error, because TStringList is compiled with "String = AnsiString" and thus the override is incorrect.

AFAIK there isn't any switch for that mode yet. You might want to create a bug report...

Regards,
Sven

In the Delphi modes (both $mode delphi and $mode delphiunicode) $H is set by default. You'd only need to define $H+ if you'd want to use the corresponding modeswitch "unicodestrings" in one of the other modes.

As Delphi now supports Mac OS X and in the future Linux as well that gains some weight...

Regards,
Sven

"String" can mean either "ShortString", "AnsiString" or "UnicodeString" depending on the compiler settings:

Non-Delphi modes and $H- (default): ShortString
Delphi mode: AnsiString
DelphiUnicode mode: UnicodeString
Non-Delphi modes and $H+: AnsiString
Non-Delphi modes and $H- and modeswitch "unicodestrings": (AFAIK) ShortString
Non—Delphi modes and $H+ and modeswitch "unicodestrings": UnicodeString

If the target is Delphi compatibility then the default string type will be UnicodeString and thus the encoding will be UTF-16 (or UCS-2 to be more correct...).

The RTL mostly uses PChar to not be restricted to 255 characters (exceptions are ancient compatibility units like DOS, Objects, etc.). There are often overloads for ShortString and AnsiString though.

Regards,
Sven

Graeme Geldenhuys wrote

Wasn't there lots of "votes" from many that string is UTF-8 encode under Linux, Unix, MacOSX, and UTF-16 under Windows?

The mode is called "delphiunicode" and in Delphi unicode versions, string = unicodestring. Maybe one day another mode or mode switch will be added that allows defining string to something else, but that's separate from adding support for the Delphi-with-string-is-unicodestring dialects.

Jonas

Delphi unicode versions parse all string constants in source code according to the current system code page (so if you compile a source file on two different systems without explicitly specifying the source code page, you can get different results if the source code contains non-ASCII strings).

We don't have support yet for determining the current system code page on Unix platforms without linking to libc, and the compiler cannot depend on libc. The warning you see above is the consequence of that.

If you don't want this code page behaviour anyway, you can use the following

{$mode delphiunicode}
{$modeswitch systemcodepage-}

Jonas

A bug report for that is not useful at this time, because this is the part on which there is still discussion. It will depend on how the RTL is handled whether any switch will even be required for that (other than {$mode delphiunicode}).

Jonas

@Michael van Canneyt
Now there is a good table to add to the Lang Ref or some other FPC documentation.

[...thinking out loudly...]
Oh how I yearn for the Java string handling simplicity - compared to the mess Delphi and FPC is in.

Hi

Jonas Maebe wrote

The mode is called "delphiunicode" and in Delphi unicode versions, string = unicodestring.

Not to get this thread into one of those heated unicode discussions again, but couldn't FPC at least do one better that Delphi. "Unicode" means a string in UTF-8 or UTF-16 or UTF-32 encoding - this is how the unicode standard is defined. Why must FPC fall into the same shit habit of Delphi where "unicode" means _only_ UTF-16 (which by definition is wrong!)

Maybe "UnicodeString" should be reserved for the true meaning. And for DelphiUnicode mode do something like String = DelphiString where DelphiString is a UTF-16 only string [or String =
CodePageString(utf-16) or whatever you want to call it].

Anyway, just my 2¢ worth..... no need for any reply.

Hi,

Jonas Maebe wrote

If you don't want this code page behaviour anyway, you can use the following {$mode delphiunicode} {$modeswitch systemcodepage-}

OK, I think I got my final answer on this.... FPC is definitely not ready to port any Delphi 2009+ projects. I fix one error, and get 30+ new errors.... over and over... The RTL _must_ also be unicode aware for any delphi unicode project to be ported. For example, you can't currently do something as simple as defining your own exception classes based on the RTL's Exception class, because the FPC 2.7.1 RTL defines the Message property as AnsiString at the moment, and doing a UnicodeString to AnsiString conversion will give you data loss. This is a critical issue, and appears all over the RTL at the moment.

So for anybody thinking of trying what I did...... DON'T BOTHER AT THIS TIME.

But thanks to everybody that offered information on this... it was educational. :-)

Sven Barth wrote

The RTL mostly uses PChar to not be restricted to 255 characters (exceptions are ancient compatibility units like DOS, Objects, etc.). There are often overloads for ShortString and AnsiString though.

AFAIK all file handling routines use the file record definitons from \lazarus\fpc\source\rtl\inc\filerec.inc with the following definitions:

------------------------------------------
const
   filerecnamelength = 255;
type
   FileRec = Packed Record
     Handle    : THandle;
     Mode      : longint;
     RecSize   : SizeInt;
     _private  : array[1..3 * SizeOf(SizeInt) + 5 * SizeOf (pointer)] of byte;
     UserData  : array[1..32] of byte;
     name      : array[0..filerecnamelength] of char;
   End;
------------------------------------------

So whenever you want to open a file the "name" can be no longer than 255 characters.

+1

Maybe "UnicodeString" should be reserved for the true meaning. And for DelphiUnicode mode do something like String = DelphiString where DelphiString is a UTF-16 only string [or String = CodePageString(utf-16) or whatever you want to call it].

Have sense. This is a good idea.

Anyway, just my 2¢ worth..... no need for any reply.

Now has 4¢. =)

Marcos Douglas