Hi,
In 2008 and 2011 there were threads about FPC and Unicode resource strings with the conclusion that FPC does not support them. Are Unicode resource strings implemented in FPC now? I did not find it in documentation.
Is there a replacement or supplement for
http://wiki.freepascal.org/FPC_Unicode_support
which answers such questions?

Thanks, Martin

Martin Schreiber wrote

In 2008 and 2011 there were threads about FPC and Unicode resource strings with the conclusion that FPC does not support them. Are Unicode resource strings implemented in FPC now?

No, not yet.

Jonas

At the moment FPC resource strings are still ansi strings even in delphi unicode mode.

Best regards,
Paul Ishenin

Thanks Jonas and Paul.
Please permit a little comment. All what was missing for good multi language Unicode support in FPC after Florian and Peter implemented widestrings (which were reference counted on all platforms at the beginning!) are Unicode resource strings. Now, many years later, we still don't have Unicode resource strings but we have codepage aware 8-bit strings? I don't understand the priorities.

Martin

Just to avoid confusion: The reference counted 2-byte string type on all platforms is UnicodeString, not WideString (the latter is not reference counted on Windows platforms).

The codepage aware string type was added to 2.7.1, because there already existed a branch for this and "just" needed to be merged. There does not yet exist any code for Unicode resource strings.

Regards,
Sven

WideString was reference counted on Windows when it was introduced in FPC. The FPC revision was 1.9.x IIRC. The first bug report I made in "wide" area is from 30.4.2004 and revision 1.9.3.
http://bugs.freepascal.org/view.php?id=5084

Later it has been changed to OLE string on Windows because of Delphi compatibility.
http://www.mail-archive.com/fpc-devel%40lists.freepascal.org/msg07520.html http://www.mail-archive.com/fpc-devel%40lists.freepascal.org/msg03010.html http://www.mail-archive.com/fpc-devel%40lists.freepascal.org/msg03018.html

Martin

Sven Barth wrote

Just to avoid confusion: The reference counted 2-byte string type on all platforms is UnicodeString, not WideString (the latter is not reference counted on Windows platforms).

Please correct me if I am wrong, but I think WideString was reference counted an all platforms "in the beginning" - like Martin mentioned. Later it was changed, and the new UnicodeString become the "reference counted on all platforms" type.

The codepage aware string type was added to 2.7.1, because there already existed a branch for this and "just" needed to be merged. There does not yet exist any code for Unicode resource strings.

FPC's Unicode support is still in its infancy. It is not just resource strings that are missing. As my recent message from the fpc-users mailing list shows.

Vital decisions of how Unicode should be implemented are still not decided by the FPC team. There is a major problem in the FPC project though. The FPC team seems to be dead-locked on how to implement Unicode features. Nobody can agree on anything. Thus no work can be started on the RTL and FCL.

In the meant time many projects keep implementing there own Unicode workarounds. Not a good sign, but all we can do.

So the BIG question remains: When will the FPC team sit down and hash out the details of implementing Unicode support? Please note, I'm not saying "implement it", just saying... "agree on how it should be implemented". If the FPC team stays in a dead-lock, then maybe the better option would be to allow the public to vote on it.

Regards,
- Graeme -

I guess most people would say that "good multi language Unicode support in FPC" requires a Unicode supporting RTL.

Mattias

Graeme Geldenhuys wrote

Nobody can agree on anything.

Which is rather understandable (as several a long winding discussion in this forum showed). FPC is always eager to provide as full Delphi compatibility as possible, and the Delphi Unicode support is not very "nice". "Decent" Unicode is an extremely complex issue, anyway (trying to provide as well user friendly usage, and and versatility, and good backwards compatibility, and good portability, and good performance). If necessary compatibility to the seemingly abysmal current Delphi and the necessity to reuse the code of the existing (seemingly rather good) implementation comes into play, a common denominator seems to be very hard to find.

If the FPC team stays in a dead-lock, then maybe the better option would be to allow the public to vote on it.

When thinking about Lazarus now exiting the v0.xxx state not a good omen.

-Michael

Not necessarily. Lazarus, fpGUI and MSEgui have their own implementations which don't need to be strictly Delphi compatible and therefore can better suit the users needs.

Martin

In our previous episode, Martin Schreiber said:

Not necessarily. Lazarus, fpGUI and MSEgui have their own implementations which don't need to be strictly Delphi compatible and therefore can better suit the users needs.

IMHO it is more a question if a hard break like Delphi is necessary or that there is a way to switch basetypes. (like what to do with overrides?)

I'm starting to believe the hard break is the only serious option.

My opinion: Do what you want with RTL and FCL but optimize the compiler so it is possible to build optimal frameworks for different requirements.

Martin

So could somebody make a definitive statement: for basic two-byte Unicode handling, what types should be used?

I've got a couple of terminal emulators using WideChar and WideString for internal manipulation, what /should/ I be using? and where does it leave things like Sorokin's regex unit, which similarly use WideChar and WideString?

Mark Morgan Lloyd wrote on Mon, 20 Aug 2012:

So could somebody make a definitive statement: for basic two-byte Unicode handling, what types should be used?

unicodestring, because it is generally somewhat faster than widestring on Windows. On non-Windows platforms, widestring=unicodestring in any case.

I've got a couple of terminal emulators using WideChar and WideString for internal manipulation, what /should/ I be using? and where does it leave things like Sorokin's regex unit, which similarly use WideChar and WideString?

widechar = unicodechar on all platforms. On Windows, where widestring is not the same as unicodestring, one can be converted to the other without any dataloss. Again, you may get some minor speed penalty on Windows, but it's unlikely you'll notice it.

Jonas

Yes, specializations have advantages.

I only think that "good Unicode support in FPC" sounds as if all the important things of Unicode are in FPC and you don't need extra libs.

Mattias

Now I understand what you ment with "which
were reference counted on all platforms at the beginning!" ;) [I came to FPC during the 2.2 series] Thank you for clarification.

Nevertheless to avoid confusion one should pay attention to which string type one mentions.

Regards,
Sven

True. I actually meant the possibilities of the compiler.

Martin

Thanks Jonas, noted.

Mark Morgan Lloyd wrote

for basic two-byte Unicode handling, what types should be used?

:-) *That* is part of the problem :-)

-Michael

Michael Schnell wrote

long winding discussion in this forum

.. and on the Lazarus forum ...

-Michael

+1

Get the compiler to decently support Unicode. At least the public can then build on that however they want. The FPC RTL can then later be mangled to be a delphi clone - or leave it to other projects to implement there own RTL's.

At the moment the Delphi-compatibility arguments is holding FPC back in the stone ages. All the advantages of FPC is starting to fall away. Delphi hasn't done anything major in years (Embarcadero simply buy out other projects and launch it as their own in the next Delphi version - which is a disaster too). And with all this "delphi compatibility" arguments, the FPC project wants to be one step behind that. Crazy! Like Martin with MSEgui, I would rather push fpGUI & tiOPF forward by implementing my own "unicode rtl" or required functions/classes if I must.

The "delphi compatibility" is purely a myth too. FPC isn't compatible with a single Delphi version (except maybe the 10 year old Delphi 7). For any newer Delphi versions, it has bits of this, bits of that... I wouldn't call that "compatible".

So lets try and keep the "delphi compatibly" out of this discussion of the FPC Compiler and Unicode support.

Regards,
- Graeme -

In our previous episode, Graeme Geldenhuys said:

The "delphi compatibility" is purely a myth too. FPC isn't compatible with a single Delphi version (except maybe the 10 year old Delphi 7). For any newer Delphi versions, it has bits of this, bits of that... I wouldn't call that "compatible".

That is an opinion of you and maybe Martin (S), and just that.

I don't agree, and frankly I think it is ridiculous. Compatibility is what brought the projects so far (and fact is that you yourself were asking about compatibility only a few days ago)

Not to least because all the anti-Delphians can't seem to agree on anything but their common anti-Delphi drum.

Dear FPC-all!

Since the beginning of my FPC use I maintain the following FPC/Delphi compatible code.

to get unicode string from resource i use

function _W(const pRec: PResStringRec):WideString;
begin
   try
     Result:=LoadResStringW(pRec);
   except
     Result:='';
   end;
end;

example of use

// this may be placed in separate includes for Delphi and FPC
{$ifdef FPC}
resourcsestring
     sSomeConstant='This is UTF-8 encoded resource string';
{$else}
resourcsestring
     sSomeConstant='This is winXXXX encoded resource string';
{$endif}
var ws:WideString;
ws:=_W(@sSomeConstant); // this is X-platform way and Delphi/FPC compatible
// FPC version of string loader
function LoadResStringW(const pRec: PResStringRec): WideString;inline;
begin
   if pRec=nil then
   Result:=''
   else
   begin
     Result:=UTF8Decode(PAnsiString(pRec)^);
   end;
end;
// Delphi version uses WinAPI call
function LoadResStringW(const pRec: PResStringRec): WideString;
var
   hInst:cardinal;
   Buffer: array [0..1023] of char;
begin^M
   if pRec = nil then Exit;
   if pRec^.Identifier < 64*1024 then
   begin
     hInst:=FindResourceHInstance(pRec^.Module^);
     // try wide
     if LoadStringW(hInst,pRec^.Identifier, @Buffer, SizeOf(Buffer))>0 then
       Result:=PWideChar(@Buffer)
     else // try narrow
     if LoadString(hInst,pRec^.Identifier, @Buffer, SizeOf(Buffer))>0 then
       Result:=PChar(@Buffer)
     else
       Result:='-lost-';
   end
   else
     Result := PWideChar(pRec^.Identifier);
end;

regards

Graeme Geldenhuys wrote on Mon, 20 Aug 2012:

Stop with the drama already, seriously. The Delphi-compatibility arguments are not about the compiler, but about the RTL/FCL. The compiler already fully supports unicode, except for resourcestrings. That has nothing to do with Delphi compatibility or not, but is simply a matter of nobody caring enough to implement it. The bug report requesting such support already dates from 2005:
http://bugs.freepascal.org/view.php?id=6477

Jonas

Anton Kavalenka wrote

Since the beginning of my FPC use I maintain the following FPC/Delphi compatible code.

Does it work if the source has been compiled with -Fcutf8 on Windows? IIRC FPC converts the source utf8 constants to the system encoding before building the resource string tables. Another question is the encoding of the *.rst files we need in the MSEi18n tool.

Martin

Marco van de Voort wrote

I don't agree, and frankly I think it is ridiculous. Compatibility is what brought the projects so far (and fact is that you yourself were asking about compatibility only a few days ago)

Yes, I gave FPC 2.7.1 the benefit of the doubt... And what did my little experiment prove? That FPC is not nearly compatible enough with even a D2009 project - that's a Delphi version that is 4 years old, and within a month superseded by yet another new Delphi version. Year-on-Year the "delphi compatibility" is getting more and more outdated. So stop drumming about the "delphi compatibility" when it doesn't exist, other to a 10 year old obsolete Delphi version.

And even with your "delphi compatibility" drumming, nobody is moving to FPC from Delphi any more, now that Delphi is again seeing some development work being done by its latest owner. Yes and that even includes you.

Anyway, I would have loved to take this for a few more rounds in fpc-other, but I'm a bit busy at the moment.

I use separate include files for resource strings in UTF-8 and in win1251.

{$ifdef FPC}
     {$i resources-utf8.inc}
{$else}
     {$i resources-cp1251.inc}
{$endif}

IIRC FPC does NOT convert resource strings into system encoding.

Besides my applications use FPC runtime to extract own resorcestrings into external files (for translation).

procedure ExtractStringResources;
var
   os:TFileStream;
begin
   if not FileExists(DefLocalesPath) then CreateDir(DefLocalesPath);
os:=TFileStream.Create(DefLocalesPath+PathDelim+ExtractFileName(ParamStr(0))+'.fpcres',fmCreate);
   try
     SetResourceStrings(@DoSaveStringResource,os);
   finally
     os.Free;
   end;
end;
function
DoSaveStringResource(Name,Value:AnsiString;Hash:Longint;arg:pointer):AnsiString;
var
   os:TStream absolute arg;
begin
   Result:='';
   os.WriteBuffer(Hash,sizeof(Hash));
   os.WriteAnsiString(Value);
end;

Resources from both Windows and Linux EXE are extracted in UTF-8 (not win1251).

Anton

Anton Kavalenka wrote

Resources from both Windows and Linux EXE are extracted in UTF-8 (not win1251).

Do you compile with -Fcutf8 or {$codepage utf8}?

Martin

I use default implicit settings (I suppose it is UTF8). None of my sources contain codepage controlling directive.

Anton

This probably explains the difference. MSEgui uses UnicodeString and is compiled with -Fcutf8 for assignement of Unicode string constants to UnicodeString variables and transparent conversion of Unicode string constants to system -Fcutf8 or {$codepage utf8} FPC copies the 8 bit strings "as is" into the executable.

Martin

In our previous episode, Graeme Geldenhuys said:

A nice example of Creative measuring.

FPC 2.6.0 of last januari has mostly 2006-2007 features. And D2009 is a landmark, version, the first one to major break compatibility in a decade with many very major features. (unicode is just one of them), so it is logical that will take more than a year.

(2.6.0 branched off just over an year ago)

And even with your "delphi compatibility" drumming, nobody is moving to FPC from Delphi any more, now that Delphi is again seeing some development work being done by its latest owner

I have no idea where you base this on.

Fine with me: let's kick all those complicated utf-8 and utf-16 encodings and just use utf-32 for the unicodestring type, it's very simple to use, no platform is prefered over others, normal PCs have 8 GB+ today so memory is not an issue anymore neither is CPU power. Clean, simple, easy.

If the FPC team stays in a dead-lock, then maybe the better option would be to allow the public to vote on it.

And who implements it? The public votes as well?

Fine with me: let's kick all those complicated utf-8 and utf-16 =====================
Isn't UCS-2 enough for all possible coverage on the Earth planet ? Even chineese people use "64K-" dictionary (the sipmplified alphabet) for computer typing.

PS:
UCS-4 (and UTF-8 BTW) looks appropriate for future contacts with aliens with their unknown alphabets, though :) Is "to be ready for contacts with them" the purpose ? Then UCS-2 with surrogate pairs is a solution as well & doesn't impose double memory allocation.

Florian Klämpfl wrote

And who implements it? The public votes as well?

Why are you complaining? You have your hobby projects (FPC etc.), and I have mine (fpGUI, tiOPF, OnGuard, MPP etc). I have no problem contributing to something I actually know or (have the time to) understand. In the case of FPC that would translated to the RTL, fpdoc, various FCL packages. But then many disagree with my past suggestions for RTL changes - so howe successful my attempts will be, I don't know. Anyway, I'll rather leave the Compiler internals in the more capable hands like you (just in case you didn't know, that was a compliment).

Ivanko B wrote

Isn't UCS-2 enough for all possible coverage on the Earth planet ?

No, Unicode is large than UCS-2. Also about your "alien language" comment. Unicode includes code points not just for spoken languages, but for anything that can be written. eg: musical notes, scientific symbols, advanced math symbols, map glyphs, smiley faces (can be used in in email's, IM programs etc without embedded images), braille etc

This probably explains the difference. MSEgui uses UnicodeString and is compiled with -Fcutf8 for assignement of Unicode string constants to UnicodeString variables and transparent conversion of Unicode string constants to system encoding at runtime.
=============
Really. MSEgui doesn't expose any problems with unicode - neither in runtime not for string constants from source files.

No, Unicode is large than UCS-2. Also about your "alien language" comment. Unicode includes code points not just for spoken languages, but for anything that can be written. eg: musical notes, scientific symbols, advanced math symbols, map glyphs, smiley faces (can be used in in email's, IM programs etc without embedded images), braille etc ========
They all have already been present in UCS-2, AFAIK. Do You means possible extensions in the future ? Are they really possible ?

This is exactly the problem: everybody has its own opinion.

so howe successful my attempts will be, I don't know. Anyway, I'll rather leave the Compiler internals in the more capable hands like you (just in case you didn't know, that was a compliment).

Besides the resourcestrings I'am not aware of any unicodestring issues in the *compiler*. If there are, please report them to the issue tracker.

Hi,

Florian Klämpfl wrote

Besides the resourcestrings I'am not aware of any unicodestring issues in the *compiler*. If there are, please report them to the issue tracker.

I have heard various back-and-forth comments about UnicodeString and others, which makes me believe the compiler+unicode is still incomplete, or at least not set in stone (understood as: don't really use, because it could still change).

For example:

* new mode 'delphiunicode' was introduced, but what about developers wanting Unicode,
but the project is a pure FPC project (it has no history of Delphi)?

* Only in 'delphiunicode' mode is String = UnicodeString (I guess the same issue as above)

* All other modes String = AnsiString or ShortString based on {$H+} or not. String is the most
used type, and so too is the objfpc mode. So shouldn't String = UnicodeString in FPC 2.8.0 release
for mode objfpc too?

* UnicodeString is always UTF-16 (so everything but Windows takes a conversion penalty)!
As far as I remember, that was good feedback from developers like the idea of having
UnicodeString as UTF-16 on windows and under Linux, MacOSX, Unix etc it is UTF-8.

* The above point would also alleviate the codepage compiler warning under Linux/Unix/MacOS when
delphiunicode mode (the only mode where String = UnicodeString). Then again ALL text files I have
even seen use UTF-8 as encoding, so even under Windows, source code should be assumed UTF-8
encoded. Java, C#, W3C (HTML & XML) etc all do that too.

* Unicode & resource strings as Martin mentioned is not working yet.

There are lot of uncertanties, thus developers are reluctant to start using (and testing it), because they might end up having to change 100's of thousands of lines of code later again.

True, it's not yet released but in 2.7.1 aka trunk for development, testing and bug fixing.

Then use {$mode objfpc} {$modeswitch unicodestrings}

{$mode delphiunicode} is for people who try to compile delphi code with
least as possible effort.

* Only in 'delphiunicode' mode is String = UnicodeString (I guess the same issue as above)

See above.

No, it breaks old code for nothing. We can add an objfpcunicode mode or whatever you like.

* UnicodeString is always UTF-16 (so everything but Windows takes a conversion penalty)!

A decision has been made and you are not happy with it. Fine. But before you called the fpc team being in a deadlock? Overcoming such deadlocks does not make everybody happy.

Indeed, this can be fixed.

Yes, this certanty is only given when we

WideStrings on Windows platforms are allocated in *system* space, so that they can be used across processes. Reference counting can occur only according to the Windows (COM) rules. Delphi UnicodeStrings are stored in the (local) program space instead, so that local reference counting can be used. Dunno about passing such strings to other processes, though.

IMO UTF-8 is supported by all platforms, so that there exists no urgent need for adding UTF-16 support. More problematic is the default "String" type break between older (AnsiString) and newer (UnicodeString) Delphi versions. The consequence of following *that* decision were incompatible FCL (and LCL) classes, resulting in double maintenance efforts. This duplication can be avoided by using the implicit string conversions, offered by the new string types. This applies also to the handling of resource strings. The runtime impact depends on the string model used in a *program*, where the use of UTF-16 strings would require many conversions in *interfacing* UTF-8 components/libraries.

It's unclear whether UTF-16 strings really allow for faster string handling, since *full* Unicode support still has to take into account UTF-16 *surrogate pairs*, no real difference vs. handling of UTF-8 multibyte sequences.

What special support do you expect?
Which of these features are essentially different for UTF-8 and UTF-16?

DoDi

I've got a couple of terminal emulators using WideChar and WideString for internal manipulation, what /should/ I be using? and where does it leave things like Sorokin's regex unit, which similarly use WideChar and WideString?

Depends on which libraries you use. AFAIK SBCS RegEx works for both Ansi and UTF-8 strings, so that an UTF-16 library is optional. For the terminal emulators I'd think that it's sufficient to introduce an internal string type that allows to switch between UTF-8 and UTF-16, so that the (different?) behaviour can be tested. When there exist differences, this indicates that the WideString emulators *only* handle Unicode BMP characters, not surrogate pairs, and you have to decide whether this restriction is okay for you.

DoDi

I guess most people would say that "good multi language Unicode support in FPC" requires a Unicode supporting RTL.

Please clarify: *Unicode* or UTF-16 support?

Unicode is covered by both UTF-8 and UTF-16, so it's not really important which encoding is used in the supporting procedures.

DoDi

The "delphi compatibility" is purely a myth too. FPC isn't compatible with a single Delphi version (except maybe the 10 year old Delphi 7). For any newer Delphi versions, it has bits of this, bits of that... I wouldn't call that "compatible".

+1

Since the Delphi versions are incompatible amongst each other, how can a single RTL and compiler mode switch support *all* these versions?

The same argument for the LCL, which is (and can not be) compatible with any specific VCL version, on non-Win32 platforms/widgetsets.

So lets try and keep the "delphi compatibly" out of this discussion of the FPC Compiler and Unicode support.

+1

DoDi

Hi,

Florian Klämpfl wrote

True, it's not yet released but in 2.7.1 aka trunk for development, testing and bug fixing.

Understood. I'm normally reluctant to try FPC Trunk, but thought that I better try this time round.

Ah, I guess I didn't fully understand the modeswitch option. I'll experiment with this a bit more.

Now that I know a bit more about the modeswitch option, I could simply modify my unit templates, so a new compiler mode is probably not needed then.

Many things by the FPC team get discussed or debated behind closed doors, so how are we even in the fpc-devel mailing list to know what is decided. Last I read in fpc-devel it was still heavily debated and no decision was in sight. I guess now we know, the decision is UTF-16 everywhere where UnicodeString is used.

Hans-Peter Diettrich wrote

*only* handle Unicode BMP characters, not surrogate pairs, and you have to decide whether this restriction is okay for you.

Such a restriction should NEVER be okay! Otherwise Unicode support is incomplete. That is what makes UTF-8 so great. It's perfect for (byte) streaming, no little- or big-endian to worry about, already the correct encoding for HTML, XML and in fact 99.99% of all text files, and NO SURROGATE PAIRS.

Since the Delphi versions are incompatible amongst each other, how can a single RTL and compiler mode switch support *all* these versions? ====================
Really the team seems to fights to FPC + Lazarus be capable of building thousands of Delphi based components - archivers, cyphers, audio processors etc things which people mostly like Delphi for and which seldom use specific Delphi features causing problems to FPC. That's not to write everything from scratch like with fpGUI, MSEgui etc libs far way from Delphi. Though from me experience, these thousands of components are difficult & time consuming to test [thoroughly! ] & choose amongst - in many cases writing from scratch is easier. Synapse vs Indy is a great example :)

Actually, you overestimate what the "FPC team" can decide. The "decision" was more like: somebody popped up and proposed to implement Unicodestring support in a D2009+ compatible way without breaking too much legacy stuff. So we could say, ok, we merge this into fpc's svn. The alternative would have been to say no, we leave things as they are: "deadlock".

The current situation is:
- either somebody starts to implement support for unicodestring being utf-8 (or whatever) on linux in a compatible way with the current approach, then 2.8.0 will use this
- nobody works on it, then 2.8.0 comes with unicodestring=utf-16 always.

MAm 20.08.2012 20:27 schrieb "Hans-Peter Diettrich" :

counted an all platforms "in the beginning" - like Martin mentioned. Later it was changed, and the new UnicodeString become the "reference counted on all platforms" type.

WideStrings on Windows platforms are allocated in *system* space, so that

they can be used across processes. Reference counting can occur only according to the Windows (COM) rules. Delphi UnicodeStrings are stored in the (local) program space instead, so that local reference counting can be used. Dunno about passing such strings to other processes, though.

The point that Martin and Graeme made was that at the beginning of WideString support in the compiler (1.9.x or so...) WideString was implemented in a refcounted way like UnicodeString is today. Only later on WideString on Windows was changed to be the COM compatible string type we know from Delphi.

Regards,
Sven

I think I need to clarify. The terminal emulators are not for a standard coding such as UTF-8, but accept a non-standard byte sequence over e.g. a telnet or serial connection and convert that to a particular set of characters to emulate e.g. an IBM Selectric APL golfball.

Sorokin's regex unit is a separate issue, and applies to FPC's regexpr package which uses WideChar: I don't know whether this would be problematic on Windows.

UCS2 is nowadays known as the BMP (Basic Multilingual Plane) of full Unicode. Have a look at the full Unicode codepages, what is and what is not part of the BMP.

DoDi

Ivanko B wrote

Really the team seems to fights to FPC + Lazarus be capable of building thousands of Delphi based components - archivers, cyphers, audio processors etc things which people mostly like Delphi for and which seldom use specific Delphi features causing problems to FPC.

ASYNCPRO (http://sourceforge.net/projects/tpapro/ )

PLEASE !!!!

I doubt that it will be possible to just compile it (e.g. for Linux) but with optimum compatibility of the compiler, porting the source code should be rather easy.

- Michel

Graeme Geldenhuys wrote

Such a restriction should NEVER be okay!

How _can_ it be OK regarding comparing strings, when all Unicode variants allow for multiple codings for the same single printable "character" (and moreover what "character" do the users regard as "equal").

-Michael

Graeme Geldenhuys wrote

* UnicodeString is always UTF-16 (so everything but Windows takes a conversion penalty)!

This is true of course,

But does that really suggest taking the effort to support other Unicode variants ?

The conversion is done only when entering and exiting the OS / GUI framework calls. I understand this does not happen too often.

Of course it is really nice to provide support for any Unicode encoding, but I don't think it does not harm to use UTF-16 as a default.

-Michael (who doe not usually advocate Windows-centric development)

Sorry:
I do think it would not harm to use UTF-16 as a default.

-Michael

HI,

Hans-Peter Diettrich wrote

UCS2 is nowadays known as the BMP (Basic Multilingual Plane) of full Unicode.

The UCS2 is considered obsolete! Nothing else needs to be said. :)

Have a look at the full Unicode codepages, what is and what is not part of the BMP.

Download the Unicode eBook chapters and look at the section "Details of Allocation" normally in Chapter 2. There are lots of useful things in Plane 1-16 (BMP being Plane 0).

Like I said, outside the BMP, less is used for the spoken languages, but there are some CJK characters in Plane 2. Plane 1 has many interesting things for "modern applications" like Domino Tiles, Advanced Math symbols (which we use in our company), Map Symbols often seen on GPS units, Mahjong Tiles, Musical symbols, Smiley Face symbols used in emails and IM programs, System of Divination symbols, Large Historic Script area, Large private areas (any apps that ship custom fonts could use those private areas - MacOSX uses the private areas for the "apple" symbol in their fonts) and many more.

The reluctance to support Planes 1-16 is often seen by UTF-16 or UCS-2 users, because they are just lazy! That is just wrong. I'll say it again, that is what makes UTF-8 so great. NO special handling or implementation is required to handle the _whole_ Unicode character set. Implement UTF-8 to handle the BMP (Plane 0), and you get handling for Plane 1-16 for free.

I know.

I'm only saying that for most people an utf-8 and/or utf-16 string type is not enough. For example common file function currently support only 8-bit windows codepage.
Users expect to handle files x-platform without using external libraries like LazUtils.

Mattias

Ehm, I did both. In the beginning MSEgui switched from Widestring to utf-8 encoded Ansistring because of the buggy FPC widestring implementation (MSEgui started with Delphi/Kylix). Some weeks later I switched back to widestring and bite the bullet to write FPC bug reports until it reached usable stability.

But if you are such a UTF-16 (actually UCS-2 as

that is what MSEgui supports) fan, why isn't MSEgui source code stored in UTF-16 encoding either? ;-)

Sure, MSEgui uses utf-8 for external storing and exchanging text data. Internal all is 16 bit UnicodeString. Use the best encoding for the task. ;-)

There is good reason why UTF-8 is so

popular. And by the way, a UTF-8 codepoint is only 1-4 bytes in size.

It depends on the specification, seen the parentheses?.

Martin

100% agreed.

Martin

For non-fixed char length there's nothing better than UTF8 (default ASCII compatible, ready for any future alphabets,..). For fixed-char length (fast string operations etc) also there's nothing better than UCS-2 (the Earth coverage ) & UCS-4 (the galaxy coverage). The non-fixed char length UTF-16 (UCS-2 + surrogate pairs) looks less efficient than UTF-8 almost from any look point.

For non-fixed char length there's nothing better than UTF8 (default ASCII compatible, ready for any future alphabets,..). For fixed-char length (fast string operations etc) also there's nothing better than UCS-2 (the Earth coverage ) & UCS-4 (the galaxy coverage). The non-fixed char length UTF-16 (UCS-2 + surrogate pairs) looks less efficient than UTF-8 almost from any look point.
==============
It assumes having FPC RTL optimized string functions for UTF-8 ( rudimental nowadays - via conversion procedures), UCS-2 (not all functions nowadays) & UCS-4 (none nowadays).

I disagree. Handling 1..4(6) bytes is less efficient than handling surrogate *pairs*.

Martin

Michael Schnell wrote

I doubt that it will be possible to just compile it (e.g. for Linux) but with optimum compatibility of the compiler, porting the source code should be rather easy.

You're in for a surprise... With a statement that reads "It provides direct access to serial ports, TAPI, and the Microsoft Speech API." it should start sounding alarm bells for Linux developers. TAPI and MS Speech API doesn't exist under Linux, so requires completely new implementations from the ground up. Direct serial port access is probably very different under Linux too.

I have ported (or fixed previous port attempts) of Turbo Power products to work x-platform. Turbo Power loved to use Windows API calls etc, and that makes porting a non-trivial task. But with lots of effort, anything is obviously possible. Not to mention that sometimes things are just easier when you start an implementation from scratch in a x-platform manner.

Anyway, I don't know how this relates to the current message thread, so I'll stop here.

Handling 1..4(6) bytes is less efficient than handling surrogate *pairs*.
===============
But surrogate pairs break array-like fast char access anyway, isn't it ? And there's a lot of room for optimizing utf-8 operation for instance http://bjoern.hoehrmann.de/utf-8/decoder/dfa/.
Also a publication at http://www.utf8everywhere.org/.

Hi,

Michael Schnell wrote

How can it be OK regarding comparing strings, when all Unicode variants allow for multiple codings for the same single printable "character" (and moreover what "character" do the users regard as "equal").

The Unicode Standard covers all this. It short, if you want to do string comparisons, one option is to normalise the text before you do a compare.

Hi,

Michael Schnell wrote

But does that really suggest taking the effort to support other Unicode variants ?

Yes, if you want to to make the statement "FPC fully supports Unicode"

The conversion is done only when entering and exiting the OS / GUI framework calls. I understand this does not happen too often.

I beg to differ.

Of course it is really nice to provide support for any Unicode encoding, but I don't think it does not harm to use UTF-16 as a default.

As Florian said... UTF-16 was implemented first in the FPC Compiler. Others are welcome to add UTF-8 as the default for other FPC platforms like Linux. There are a lot more Unix-type platforms than the one Windows platform, so why must all the other platforms take a conversion hit. I know nothing about compiler internals, but I hope to rise to the challenge Florian set.

Martin Schreiber wrote

I disagree. Handling 1..4(6) bytes is less efficient than handling surrogate *pairs*.

Yet another myth.... But if you are such a UTF-16 (actually UCS-2 as that is what MSEgui supports) fan, why isn't MSEgui source code stored in UTF-16 encoding either? ;-) There is good reason why UTF-8 is so popular. And by the way, a UTF-8 codepoint is only 1-4 bytes in size.

Anyway, this message thread is not about a UTF-16 vs UTF-8 pissing match, it is about FPC boasting good all-round Unicode support in the most efficient manner on all supported platforms.

IMO unicodestring should be the same on all platforms, because otherwise the character size switches per platform, which is hard to test and asking for trouble.

The compiler already supports an UTF8String, right? If yes, then some functions can use UTF8String, some UnicodeString (=UTF-16) and the compiler magic will convert automatically.

The difficult decision is what functions and types should use UTF-8 and what UTF-16. This may depend on the platform.

One problem is that an UTF-8/16 string can contain invalid characters making it impossible to convert.
For example under Linux file names are treated as UTF-8 but are only bytes. They can and they do contain invalid UTF-8 characters. If your program should support this, you must use a FindFirst with UTF-8. To be clear: I don't say the default FindFirst under Linux must be UTF-8, I only say, there must be one version with UTF-8, e.g. FindFirstU8 and that must directly use the Linux file functions without conversions.

I guess there is no good solution for TStrings. Whatever string type is chosen, some programs will suffer.

Mattias

But if you are such a UTF-16 (actually UCS-2 as
that is what MSEgui supports) fan
=================
If Martin can implement UTF-16 (with surrogate pair) support in MSEgui string units (and these units fully cover absenting code of FPC RTL ) then the things are excellent.

PS:
UTF-8 is very-very slow compared to UCS-2 as to string manipulations so its best usage is encoding source files (as done in MSEide).

Ivanko, please stop this "slow" non sense.
Performance heavily depends on what you do and you can find good examples for almost any Unicode encoding.
At those places where speed matters, you are free to use better functions in your application.

Mattias

It's also "broken" in UTF8 in the same way - so none of them gets +1 on this. UCS4 is the only real winner here (one dword for each character).

_______________________________________________ fpc-devel maillist - http://lists.freepascal.org/mailman/listinfo/fpc-devel

Maybe you can name some example. Concrete problems can be solved, abstract UTF-8 vs UTF-16 can not.

Mattias

Graeme Geldenhuys wrote

You're in for a surprise... With a statement that reads "It provides direct access to serial ports, TAPI, and the Microsoft Speech API." it should start sounding alarm bells for Linux developers.

Of course you are very right and silly me did not take these additional uses into account.

I only used AsyncPro for avoiding explicit Thread programming when using serial interfaces. And just this is a demand I heard very often in several FPC related forums.

So I should have stated that these additonal windows-centric uses of AsyncPro need to me {$if ed out when not appropriate.

Direct serial port access is probably very different under Linux too.

I don't think so. Accessing the port always is similar to accessing a file, and I understand a blocking access (which is used in the Threads of AsyncPro is always provided.
Supposedly setting the serial port parameters needs some porting.

I have ported (or fixed previous port attempts) of Turbo Power products to work x-platform. Turbo Power loved to use Windows API calls etc, and that makes porting a non-trivial task.

I suppose you are right. That is why I did not start porting (the said parts of) AsyncPro.

Not to mention that sometimes things are just easier when you start an implementation from scratch

You may be right on this.

-Michael

Graeme Geldenhuys wrote

if you want to do string comparisons, one option is to normalise the text before you do a compare.

Other than the conversion necessary with system-calls when a different encoding is used internally, comparing strings happens very often within the user code. So the compiler can't force using complex stuff like that all over the place.

-Michael

Mattias Gaertner wrote

IMO unicodestring should be the same on all platforms, because otherwise the character size switches per platform, which is hard to test and asking for trouble.

This does seem appropriate. But right now Delphi comparability forces 16 Bits and Lazarus forces 8 Bits :( .

-Michael

Hi,

Mattias Gaertner wrote

IMO unicodestring should be the same on all platforms, because otherwise the character size switches per platform,

Please define "character" in your sentence above. Are you referring to a Unicode codepoint, or a "printable character"? If the first, then 4 bytes is always sufficient on all platforms.

The compiler already supports an UTF8String, right? If yes, then some functions can use UTF8String, some UnicodeString (=UTF-16) and the compiler magic will convert automatically.

How I would wish for FPC to stop this ridiculous ambiguity that Delphi enforces. Can't we just introduce UTF8String and UTF16String types. By the name they clearly state what encoding the hold. A UnicodeString type should mean any Unicode encoding, and defaults to UTF-8 under *nix type systems and UTF-16 under Windows. Thus no performance loss on any platform. After all the name "Unicode String" does not imply UTF-16 only - as per the Unicode Standards.

The difficult decision is what functions and types should use UTF-8 and what UTF-16. This may depend on the platform.

As I said, if you use the correct default encoding on each platform for the UnicodeString type, the problem you mention will not be a problem any more. Linux will use UTF-8 by default, so file handling and API was will work without any conversion.

The whole RTL should use UnicodeString type, where the encoding is as I described above.

I guess there is no good solution for TStrings. Whatever string type is chosen, some programs will suffer.

Why will some suffer? Simply default UnicodeString to the correct encoding on each platform, and no performance issues and no unnecessary conversions will occur.

Ivanko B wrote

UTF-8 is very-very slow compared to UCS-2 as to string manipulations so its best usage is encoding source files (as done in MSEide).

Please supply a test program that proves this. I don't believe you are correct.

I have implemented multiple text edit/display widgets that do plenty of string manipulation... all based on the UTF-8 encoding. I have suffered NO speed penalties.

Just because you had a bad experience doesn't doom the utf-8 encoding forever. Maybe you just had a buggy implementation. No coder is perfect. ;-)

Performance heavily depends on what you do and you can find good examples
==================
Hmm.. are there implementations of UTF-8 substringing, string comparision etc - but not using intermediate HEAVY normalizations from/to fixed char length type for BOTH input arguments ? Though me'm sure that latin people don't suffer from slowliness of utf-8 where utf-8 = ansistring.

+1

There are lot of encodings around, but for different areas of application:
- external text assets could be in any encoding (system-locale encoding, UTF8, UTF16 both BE and LE - for example, MS Excel export UTF16 text file)
- Windows system calls are UTF16, on (most of) other platforms UTF8
- input translation (physical keyboard to Unicode character)
- internal application representation is choice of developer

The problem here is that libraries floating around (including RTL and FCL) use different string types (UnicodeString, UTF8String, AnsiString), so the question is - is it possible to (re)write those libraries in a generic way (RawByteString?), so they can work with any string type?

In my experience, only about 1% of applications requires handling of individual Unicode characters (input, rendering, GUI text editing). Other parts of application can happily without that knowledge :-)

Aleksa

I have implemented multiple text edit/display widgets that do plenty of string manipulation... all based on the UTF-8 encoding. I have suffered NO speed penalties.
================
Sure no problems for GUI. But how about processing large texts ?

Lazarus does not force "unicodestring" to anything for the simple reason, that it does not use it. It only provides some functions for converting UTF-8 to/from unicodestring.

At the moment Lazarus does not even use UTF8String, because the RTL does not use it.

Mattias

OK, lets assume I'm under Linux and fpGUI uses UTF-8 internally (ideally I would like fpGUI to use the correct encoding on each platform).

TfpgMemo,LoadFromFile(). TfpgMemo uses a TStringList internally, which LoadFromFile maps too. Under Linux, filenames are UTF-8 encoded, so conversion occurs because TStringList uses UTF-16 (as per the FPC 2.7.1 suggestions) for the filename variable, and for all the file access routines. The file contains UTF-8 encoded text. They are now converted to UTF-16 text to be stored internally in the TStringList. fpGUI now needs to display that text, so calls the UTF-8 X11 API's, so another conversion is required.

This is a simple example, but look at all the conversions already. Now if UnicodeString uses the correct encoding on each platform, the conversions would be zero!

Ivanko B wrote

Though me'm sure that latin people don't suffer from slowliness of utf-8 where utf-8 = ansistring.

And I gather you base your assumptions on MSEgui. MSEgui uses UCS-2, *not* UTF-16. I also believe MSEgui doesn't bother with surrogate pairs (please correct me if I am wrong). UTF-16 will also slow down if UTF-16LE and UTF-16BE is taken into account, and surrogate pairs added to the mix. How well will your "access char via index" code perform on that? Or will your code simply be broken then. In my apps where I use UTF-8 internally, I don't have to worry about any of that, and my programs will continue ticking as normal, no matter if you I use Plane 0 or Planes 1-16 of the Unicode codepoints.

As I said multiple times, developers love to take shortcuts when it comes to UCS-2 or UTF-16. They only think BMP and nothing further. That's not what I consider "supporting Unicode".

In our previous episode, Mattias Gaertner said:

IMO unicodestring should be the same on all platforms, because otherwise the character size switches per platform, which is hard to test and asking for trouble.

I think the big issue is more about what "string" will be when the FPC is compiled in modes that are now objfpc h+ and delphi.

And then specially anything you override or pass VAR strings too.

The compiler already supports an UTF8String, right? If yes, then some functions can use UTF8String, some UnicodeString (=UTF-16) and the compiler magic will convert automatically.

rawbytestring and unicodestring overloaded. See thread in fpc-pascal of a few days back with subject "rawbytestring".

The difficult decision is what functions and types should use UTF-8 and what UTF-16. This may depend on the platform.

The question is if you fixate the classes hierarchy to a certain type on all platforms, (to avoid problems with virtual/override and VAR) does it make sense to finely grain divide the RTL over both encoding types.

That stringtype will be so dominant in practice, doing the RTL in a different stringtype depending on platform won't be as useful.

That's ugly indeed. Since that doesn't mean just an utf8 overload, but that the entire internal trajectory behind that (searchrec inclusive) must be 1-byte without conversion. Or the 1-byte to utf16 and back conversion must be stable. (invF(F(x))=x

I guess there is no good solution for TStrings. Whatever string type is chosen, some programs will suffer.

tstrings will be "string". So whatever "string" is chosen for the OOP FPC code (see first paraphraph), that will be the declaration of tstrings.

But D2009 changes many streaming related routines (load/save file/stream) to add a encoding parameter with some default value. This decouples tstrings disk format from memory format. Maybe that fixes your worry ?

Ivanko B wrote

Sure no problems for GUI. But how about processing large texts ?

Same experience as before. I must add "processing large text" is a vague statement.

Theoretically you could rewrite the FCL to support UTF8String, UnicodeString and AnsiString. But not at the same time. In an application there is always be only one of them. So you have to ship for each flavor a whole FCL plus all packages that depends on it. I guess the FPC team wants to support at most one legacy and one Unicode version. And eventually only the Unicode version.

In my experience, only about 1% of applications requires handling of individual Unicode characters (input, rendering, GUI text editing). Other parts of application can happily without that knowledge :-)

True.
But that 1% may be scattered around the whole application and there are no compiler warnings, so it is hard to find all places.

Mattias

In our previous episode, Mattias Gaertner said:

Which reminds me to point you at
http://bugs.freepascal.org/view.php?id=22501

It does not.

Mattias

How well will your "access char via index" code perform on that?
=================
It'll mean "now is the time to switch to UCS-4" :)

marcov wrote on Tue, 21 Aug 2012:

Since it's just raw bytes, it's actually as much utf-8 as it is Windows Latin-1.

but that the entire internal trajectory behind that (searchrec inclusive) must be 1-byte without conversion. Or the 1-byte to utf16 and back conversion must be stable. (invF(F(x))=x

Other frameworks also have to deal with this, and generally have a particular default and allow the programmer (and sometimes the end user) to override the default behaviour. E.g., glib assumes all file names are UTF-8, but you can change this to "assume file names are encoded in the current user's locale" or to "assume file names are encoded using encoded XYZ" (either programmatically or via an environment variable). Qt assumes they are encoded in the current user's locale, but the programmer can change this to a different code page (no environment variable). In practice, the default Qt and glib behaviour is almost always the same on Linux nowadays, since UTF-8 locales are the default.

I'm not aware of a framework that allows you to say that file names are just random bytes. It would probably be possible to implement this in FPC by adding "support" for the invalid $FFFF code page (both in ansistring and in unicodestring) and never converting anything if that one is used (basically overwrite the destination string's codepage with $FFFF if it's used by the source). Other options are not supporting invalid file names in the cross-platform RTL interface (have to use platform-specific APIs to deal with them on platforms that "support" such file names, like with glib and Qt), optionally adding "raw" overloads of such functions that possibly even accept and return arrays of byte rather than strings in order to avoid any accidental conversions and to make it clear what you're dealing with.

Jonas

Another idea is to use String all over the place as a generic string which can hold string in any encoding (probably another modeswitch for this?). So, (in theory) TStrings could store both UTF8 and UTF16 strings at the same time, and compiler magic would do necessary conversions when needed. Now, since I don't really know much about compiler internals, I'll try to give it some time and see if above idea is applicable at all.

Yes, they will most probably be scattered all around, but then - it's developer-related organizational challenge, not compiler one.

Aleksa

Especially on large texts UTF-8 can be better, because it needs less memory and fetching memory pages is expensive.
I ported the widestring XML units of FPC to UTF-8
because I had to handle thousands of xml files with about 400 MB. Because these documents are in UTF-8 parsing is about 2-3 times faster on these documents, searching is about 20 to 50% faster, which is pretty much the saved memory pages.
The UTF-8 overhead is not measurable.

Another example are the codetools.

Mattias

Because these documents are in UTF-8 parsing is about 2-3 times faster on these documents, searching is about 20 to 50% faster =============
Because You name is latin ANSISTRING "Mattias Gaertner" :) But Imagine gigabytes of 4 bytes/char UTF-8 text.

In our previous episode, Graeme Geldenhuys said:

I think unicode or not is a bigger performance hit than utf8 vs utf16.

All routines like capitalization (routinely used for case insensitve comparison) get a lot more complicated. Many routines must forfeit their simple charset loops and will do a call for any set test.

utf8<->utf16<- any 256 char (ansi) charset conversion operations are fairly simple and mechanical operations that don't need much context. They are probably much cheaper than a single uppercase that we routinely for case insensitive comparisons.

utf8/16 -> ansi are a bit more involved. (since mapping many chars to few, naieve implementation requiring large lookupsets)

The conversion of the file names is negligible compared to the overhead of the OS itself. Even on Linux.

Converting a text from UTF-8 to UTF-16 does some overhead, but compared to loading it from disk the overhead is not that big. Normally an application does not simply load a text, it scans or parses the text. So for most programs the conversion makes only a few percent.

Painting a string is much more expensive than the conversion.

I agree that TStringList can easily create a performance problem, but afaik loading a text into a GUI is not a good example to show conversion overhead.

This is a simple example, but look at all the conversions already. Now if UnicodeString uses the correct encoding on each platform, the conversions would be zero!

No. On Windows you have to open UTF-8 files too.

Mattias

Marco van de Voort wrote

All routines like capitalization (routinely used for case insensitve comparison) get a lot more complicated.

Obviously Unicode is a lot more complicated, because it is design for _all_ spoken and non-spoken languages. ASCII is minute in comparison.

Good news is that Unicode can do lots of other things too which could help performance.
eg: You don't need to attempt to capitalise characters if they are outside a Unicode category "Letter" and "Uppercase". Or for a Unicode StrToInt() implementation, simply do a simple "Number" or "Decimal Digit" category check before you attempt the conversion.

Unicode categories are very useful, and they are simple lookup tables
- and already available for FPC to use (implemented in Object Pascal).

Because these documents are in UTF-8 parsing is about 2-3 times faster on these documents, searching is about 20 to 50% faster ============= Because You name is latin ANSISTRING "Mattias Gaertner" :)

Actually my name is Gärtner.
The texts are mostly Latin, Arabic and Hebrew. They are smaller in UTF-8 because xml has lots of English tags.

But Imagine gigabytes of 4 bytes/char UTF-8 text.

I gave you an example where UTF-8 is better. I already wrote there are good examples for every Unicode encoding, including UTF-16.

Please stop telling UTF-16 is always
faster/better/smaller/safer/whatever than UTF-8.

Mattias

Hi,

Mattias Gaertner wrote

I agree that TStringList can easily create a performance problem, but afaik loading a text into a GUI is not a good example to show conversion overhead.

Maybe so, but it does debunk the statement "does not happen too often".

OK, so zero is maybe incorrect. Let change it to 1 conversion (the file contents only, seeing that just about nobody stores files in UTF-16 encoding). Now compare 1 conversion to the multiple conversions under Linux if the RTL is only UTF-16 based.

And as you so clearly stated in a prior message, it depends on what your application does. Some programs will be heavily penalised by so many conversions.

Depending on the language, ligatures etc. still can span multiple codepoints. IMO everybody should decide whether he wants to do text processing for full Unicode, or whether simple stringhandling (as used till now) is sufficient.

I never heard that non-canoncial text has caused problems in character sets with accents or umlauts - except in (MacOS, Linux) filenames. Since file searches have to use the platform API, all required special handling can be encapsulated in the RTL.

Breaking strings into substrings can be done on specific delimiters (spaces...), which are all ASCII, again no complication with UTF. A comparison or search for given patterns also is insensitive to the encoding. Where would one really need indexed access to single characters?

DoDi

RawByteStrings are not a solution, because the specific encoding of every parameter has to be checked inside such a subroutine, and specific handling of all encodings has to be implemented there. It's easier to expect strings of a specific encoding, and let the compiler insert eventual conversions automatically. Then overloaded subroutines can be provided, for the 3 common encodings (CP_ACP, CP_UTF8 and UTF-16).

What I'm missing is a true binary string encoding, which never is subject to automatic conversions. Why implement TBytes or similar classes for data buffers, with zero comfort compared to standard stringhangling operators/functions?

DoDi

True. But let's be realistic. Some conversions are not measurable and are ok. Some can be avoided by simple changes to the application.

And as you so clearly stated in a prior message, it depends on what your application does. Some programs will be heavily penalised by so many conversions.

Yes. But maybe these applications can be adapted easily. This discussion should be about the issues where the conversions matter and there is no simple workaround.
It would be good if everyone who knows such a problem comes up with it now, so the FPC team can give an advice and/or consider it in the Unicode RTL.

Mattias

Graeme Geldenhuys wrote

Can't we just introduce UTF8String and UTF16String types. By the name they clearly state what encoding the hold.

It does make sense to (optionally) provide dynamically encoded strings, so that it is possible to do library functions that work with any encoding (and can be called with any encoding) without the need to do overloaded functions or do conversions under the hood.

-Michael

Mattias Gaertner wrote on Tue, 21 Aug 2012:

But let's be realistic. Some conversions are not measurable and are ok.

Case in point: the FPC Win32 RTL until now. It always uses the ansistring versions of OS interface functions, while NT-based Windows OSes internally all work with UTF-16. This obviously has caused problems due to unrepresentable file names, but in terms of performance?

Jonas

Graeme Geldenhuys wrote

Maybe so, but it does debunk the statement "does not happen too often".

With "not so often" I meant program runtime: it is usually not called in a close long running loop.

-Michael

There is the large category of network apps. Most protocols are utf8 or have a clear preference for utf8 (json for example). Databases are an extension of that and have the additional complication that they can mix codepages at any level. These apps can be quite sensitive to conversion overhead.

Ludo

Aleksa Todorovic wrote

Yes, they will most probably be scattered all around, but then - it's developer-related organizational challenge, not compiler one.

The compiler should not in a large area produce code that does not work as a former version (that did not use Unicode). So maybe it should not compile myString[i] at all which will not work as expected in most cases (at least with UTF8 and Unicode agnostic users).

-Michael

Michael Schnell wrote

So maybe it should not compile myString[i] at all

... and provide a decent enumerator syntax instead.

-Michael

AFAIK, all the GUI related LCL calls use UTF8 in the parameters and results. OTOH all Delphi VCL GUI calls use 16 bit string encoding in the parameters and results.

-Michael

The VCL uses the same string as the Delphi RTL classes. Formerly the encoding was MS Windows codepage, now it is UTF-16.

The LCL uses the same string as the FCL classes.
The FCL uses 8-bit strings without forcing an encoding, except for file names. The LCL expects UTF-8 and provides UTF-8 file functions. If the FCL moves to another string or starts enforcing an encoding the LCL has to be adapted.

Mattias

Michael Schnell wrote

With "not so often" I meant program runtime: it is usually not called in a close long running loop.

I have a program that does exactly that... Loads files to do CRC checking to see what changed. It's a recursive find-all that goes through 100k or more files. It's already a slow process on non-SSD drives processing 12GB or more of data, so adding the multiple unnecessary string conversions which will be enforced on Linux users would make that even worse. For me, every optimisation counts.

With some simple trials in various projects I can clearly see a Unicode RTL with one string type, and native encoding on each platform as very plausible. So why the resistance to NOT implement something as efficient as that??? Such resistance by the FPC project is what baffles my mind, and why I sometime agree with Martin Schreiber's ideas.... fork everything and just implement your own damn RTL.

[just finished reading a new Embarcadero blog post about post-XE3 development] Embarcadero is pretty much dropping the desktop scene (no future there according to them) and will primarily cater only for mobile development (iOS and Android). A new compiler, new linker, new toolkit, new RTL, new debugger. They know full well it will be a "complete code breaking change again". Oooh, I wonder how FPC and Lazarus is going to "clone" that? Does everybody still want to keep following Delphi?

http://blogs.embarcadero.com/jtembarcadero/2012/08/20/xe3-and-beyond/

Well, without more details the advice is probably to use UTF8String.

Mattias

Then you would not use TStrings in the first place.

With some simple trials in various projects I can clearly see a Unicode RTL with one string type, and native encoding on each platform as very plausible.

One string type and native encoding. Do you mean the current AnsiString?

I guess you mean UTF-16/UTF-8 depending on platform. That would be different character sizes, which means lots of IFDEFs in users code.

Mattias

Make much sense and AFAIK so far no one has said why this approach would not work.

Marcos Douglas

A more detailed example then. A web application that fills in HTML templates with variable data coming from fe. a database or whatever. HTML is all ASCII. So parsing an iso-8859-1 or UTF8 template and making ASCII tag substitutions in both CP is exactly the same. The ascii uppercase works nice in both and tags are case insensitive at virtually no cost. The problem starts when a string is supposed to have a codepage and conversions are made before functions like concatinating strings, uppercase, pos, etc. See http://bugs.freepascal.org/view.php?id=22501. Detecting code page of the template and setting the string cp accordingly? Detecting code pages can be quite expensive.
Even in the utf8 only case, converting all to utf16 to do some basic string manipulations as suggested can lead quickly to bottlenecks for such basic string manipulations in high volume web servers. I understand one can not make an rtl for every code page but the question was to list application areas where string conversions could be important or critical. I'm not pushing one or the other solution;)

Ludo

Mattias Gaertner wrote

One string type and native encoding. Do you mean the current AnsiString?

I meant a string type that changes it's encoding based on the platform it is compiled for. UTF-16 under Windows, UTF-8 under others. The RTL then uses that sinle string type throughout.

The Char type would be defined as String[4] (max size in bytes of a unicode codepoint)

I guess you mean UTF-16/UTF-8 depending on platform. That would be different character sizes, which means lots of IFDEFs in users code.

Why? Can you give an example where IFDEF's would be required?

In our previous episode, Graeme Geldenhuys said:

The Char type would be defined as String[4] (max size in bytes of a unicode codepoint)

Doesn't sound wise. length(stringtype)=n should mean that the string takes sizeof(char)*n bytes. (give or take the #0#0)

Marco van de Voort wrote

Doesn't sound wise. length(stringtype)=n should mean that the string takes sizeof(char)*n bytes. (give or take the #0#0)

I'm not sure what you are trying to accomplish? Give me sample code that will cause a problem.

In fpGUI I have UTF8Length(mystring) which returns the actual number of code points used - not bytes used. If you want the number of bytes used, simply use Length(mystring). Use each of those at appropriate times based on what you want to accomplish.

The RTL Length() function has been the source of lots of confusion to Delphi and FPC developers. So without an actual use-case I don't know what you are trying to do.

First of all: How do you define the "native encoding"? The encoding of the file system? Ext4 does not have one, HFS+ knows only normalized UTF-8.
The encoding of readln/writeln? That depends on environment options rather than OS.
The Unicode functions of the system libraries? Most applications rarely use them, but rather use frameworks like fpGui, MSEGui or the LCL. Linux does not have such functions.
Make a poll? Then you should not call it "native encoding".

Second: Why is it bad to have a platform dependent string type?
- More test work. It's not sufficient any more to test a string function on one platform, you have to test it on all platforms. That's especially hard for projects where some developers have only access to a few OS.
- Harder to optimize. It's easy to write a few optimized functions for one encoding. With multiple encodings you have to write multiple versions.
- Loss of simplification. Some things are easy in one encoding, some are not. Because you have to support all encodings, you have to always implement the difficult encoding too.

Mattias

length returns the number of characters.
UTF8Length the number of codepoints.
There must also be a function to return the number of bytes. Does someone know the name?

The RTL Length() function has been the source of lots of confusion to Delphi and FPC developers. So without an actual use-case I don't know what you are trying to do.

Mattias

Bug 22501 is about string constants and mismatch of CPs. Note that this is a different beast than dynamic data coming from files, sockets or db.

Detecting code page of the template and setting the string cp accordingly? Detecting code pages can be quite expensive.

And it is often impossible.

It's about string conversions that are critical and hard to fix. I have not doubt that changing the string type means that some functions will become slow. But it does not mean they are hard to fix. For example you could change the string type of the time critical strings to UTF8String to make sure that the big strings are never converted.

Mattias

Length(s) * SizeOf(s[1])

Regards,
Sven

21.08.12, 23:21, Sven Barth пишет:

> There must also be a function to return the number of bytes. > Does someone know the name? Length(s) * SizeOf(s[1])

It has the name ByteLength()

Best regards,
Paul Ishenin

Cheater. ;)

Mattias

O.o

Again what learned...

Regards,
Sven

utf8/16 -> ansi are a bit more involved. (since mapping many chars to few, naieve implementation requiring large lookupsets)

A single 256 element array can be used for both directions. In Ansi to Unicode the char value is used to index the array of Unicode values, otherwise the given Unicode value is searched in the array.

DoDi

length returns the number of characters.

the number of elements, which can be of any size (in arrays in general).

UTF8Length the number of codepoints. There must also be a function to return the number of bytes. Does someone know the name?

Length(s)*sizeof(s[1])

DoDi

IMO string conversion and CRC are mutually exclusive.

DoDi

In our previous episode, Hans-Peter Diettrich said:

That is an option also of course, but O(n). Probably the better solution is what was mentioned before, have a set of ranges and smaller lookuptables for these ranges.

This lowers the set size at the expense of a few (constant time) comparisons.

Hans-Peter Diettrich wrote

IMO string conversion and CRC are mutually exclusive.

Accessing a 100k of files (filenames to be exact) in a UTF-8 environment (Linux), which must all be stored in a UTF-16 string type. That's lots and lots of encoding conversions right there - in a tight loop. The point is, your apps might not require lots of encoding conversions, but there are lots of different types of apps out there - it all depends on what the application does.

Mattias Gaertner wrote

The LCL uses the same string as the FCL classes.

Yep:

type
   TCaption = TTranslateString;
...
   TTranslateString = type String;

The FCL uses 8-bit strings ...

Isn't this exactly what I tried to point out ? AFAIK in newer Delphi TCaption is String, and String is 16 Bit, so catch 21.

-Michael

Mattias Gaertner wrote

If the FCL moves to another string or starts enforcing an encoding the LCL has to be adapted.

I believe if "String" becomes a sequence of 16 bit entities instead of 8 bit entities, the LCL needs a really thorough rework.

In the Lazarus form some time ago I learned that there are supposed to be decent arguments for keeping the LCL in 8 bit mode (speed with GUI transactions in Linux) so the a move to 16 bit (or dynamically encoded) strings is not intended.

- Michael

The UTF-8 optimized functions needs UTF-16 versions. But why do you mean it needs a "really thorough rework"?

In the Lazarus form some time ago I learned that there are supposed to be decent arguments for keeping the LCL in 8 bit mode (speed with GUI transactions in Linux) so the a move to 16 bit (or dynamically encoded) strings is not intended.

The LCL itself already has some widgetsets using UTF-16. Strings speed is almost no issue for the LCL itself.
But many LCL applications and packages would suffer.

Mattias

Graeme Geldenhuys wrote

http://blogs.embarcadero.com/jtembarcadero/2012/08/20/xe3-and-beyond/

Other than politics, the big news regarding technology seems to be that Objects (or whatever) seem to get reference counted and thus I understand ".Free" gets obsolete (like with Prism). Without assessment - this is a huge move for a native code compiler. If FPC will follow, this sounds like a lot of work.

-Michael

Depending on how they implement it this might indeed be an interesting feature that we could implement (cherry picking Delphi features ^^).

Regards,
Sven

Mattias Gaertner wrote

The UTF-8 optimized functions needs UTF-16 versions. But why do you mean it needs a "really thorough rework"?

Guesswork :-)

The LCL itself already has some widgetsets using UTF-16.

Yep. So there the conversion needs to be dropped, while with the other a conversion needs to be implemented.
Thus Lazarus would get "compatible" with MSEGUI on that behalf.

Strings speed is almost no issue for the LCL itself.

I do believe that you are right here (Martin very often offered this argument when explaining why he built MSEGUI the way it is.

But many LCL applications and packages would suffer.

Same issue as when Delphi made this move.

-Michael

Sven Barth wrote

Depending on how they implement it this might indeed be an interesting feature that we could implement (cherry picking Delphi features ^^).

It will be interesting to watch if they might implement other Prism goodies as well (e.g. parallel loops and futures, which would also be a really advantageous feature and supposedly unique with a native code compiler)

-Michael

In our previous episode, Sven Barth said:

Probably it will remain limited to firemonkey. Note that the announcements says it is for mobile.

Using the same string type is far from "compatible".

> Strings speed is almost no issue for the LCL itself. I do believe that you are right here (Martin very often offered this argument when explaining why he built MSEGUI the way it is.

I never saw Martin saying that he chose Widestring/Unicodestring because speed does not matter.
AFAIK he choose widestring, because UCS2 is simpler than UTF-8 and was completely sufficient for all his projects.

> But many LCL applications and packages would suffer. Same issue as when Delphi made this move.

It is not the same.
Delphi moved from Windows CP to Unicode. That is a necessary step the LCL has already taken years ago. Now the RTL+FCL is moving to Unicode, which is necessary too. The LCL is just one package from many that needs to adapt.

Mattias

Filenames typically deserve platform specific handling, so that a special type and related classes (TFileNameList?) may be a good idea.

TFileName already exists BTW. It might be a good idea to also *use* it in the RTL/LCL filehandling support.

That's lots and lots of encoding conversions right there - in a tight loop.

Well, what do you intend to do with the filename list? Reading the files certainly requires more time than the conversions of the filenames.

The point is, your apps might not require lots of encoding conversions, but there are lots of different types of apps out there - it all depends on what the application does.

When I look at the Unicode introduction in VB4 and Delphi, there was much rumor before, many people expected slow programs with high memory requirements. Half a year later the move was accepted, and nobody complained any more. I don't think that this will be different for FPC/Lazarus :-)

BTW VB4 offered both an 16 bit (Ansi) and 32 bit (Unicode) compiler, so that everybody could evaluate and compare both versions. It turned out that the 32 bit Unicode versions at least were not slower than the 16 bit Ansi versions. VB5 came with 32 bit support only, and nobody asked for an Ansi version any more.

FPC could go the same route, offering the old (Ansi/UTF-8) version in parallel to a new Unicode (UTF-16) version. This step is required anyway. Then everybody can find out the real pro's and con's, and whether it's really recommended to continue supporting both versions.

DoDi

I don't see much work here. The code for handling interface references exists, it only has to be applied to the new TObject type, too.

More work is required in applications, because the sequence of destructor calls may change. I'm not sure whether it *really* will be possible to remove all ".Free" calls. In a GC environment it turned out that explicit finalization of objects frequently *still* is required, just to prevent trouble caused by random desctruction sequences. See Wikipedia...

DoDi

They specifically mention this is context of the frontend (see below quotation), so I'd say this will in general become part of the Delphi language (I don't think that they'll implement "target specific" language feature).

=== quote begin ===

The Delphi language front-end is being enhanced to deliver apps suited for mobile devices by adding memory management features such as automatic reference counting.

=== quote end ===

Regards,
Sven

I'm not sure whether this is a valid argument here. A constant n=256 is equivalent to O(1) - it may be a single machine instruction. Effectively the array size is only 128, because ASCII maps 1:1 to Unicode.

P.S.: Above applies to SBCS only, MBCS require more complex solutions.

See above :-)

In any case a single lookup table, for both directions, reduces memory requirements and error prone implementation efforts.

DoDi

Graeme Geldenhuys wrote

I have a program that does exactly that... Loads files to do CRC checking to see what changed.

Hmm. I feel that reading files takes a lot m,ore CPU time than converting the stings at the border of the LCL.

This of course does not include converting the file content itself.

Here of course the use of the "default" string type might not be appropriate, but - depending on the calculations - it might be useful to use a string type that keeps the format the files are in or to convert the files after reading and then do the internal work with the most appropriate encoding.

BTW.: if the files are done in many different encodings, even using a String type with fully dynamical encoding might be the most appropriate solution. Another argument not completely forget about an (optional) fully dynamically encoded string type.

-Michael

Graeme Geldenhuys wrote on Wed, 22 Aug 2012:

Accessing a 100k of files (filenames to be exact) in a UTF-8 environment (Linux), which must all be stored in a UTF-16 string type. That's lots and lots of encoding conversions right there - in a tight loop.

It's nevertheless a bad example, because even with the fastest SSD all those conversions will probably be hidden by I/O and memory latency. Even if you would implement something like the Unix "find" or "ls" programs, they would be more likely to be limited by I/O and all sorts of file/directory attribute lookups than code page conversions of file names.

Jonas

In our previous episode, Hans-Peter Diettrich said:

I'm not sure but afaik the current situation doesn't have any support for breaking cycles. Heavier use will probably need that.

Cycles are one of the reasons for using Free or other explicit finalizers, regardless of the GC model.

Mark-sweep collectors can remove also clusters with cyclic references, but they can/should not call the destructors of these (any) objects, because this could result in infinite recursion or other unwanted effects. Clearing the references before calling the destructors is not a solution, because it can prevent proper operation of the destructors.

DoDi

Even if you would implement something like the Unix "find" or "ls" programs, they would be more likely to be limited by I/O and all sorts of file/directory attribute lookups than code page conversions of file names.
============
1) I/O is heavily cached on modern a-lot-of-RAM machines & 2) conversion eat CPU & L2/3 caches & 3) conversion penalties grow rapidly on servers because of exausting resources mentioned in 2).

Although you are right that nowadays OS have highly optimized file systems, Jonas is right too.
File functions are so slow, even on Linux, that both the compiler and Lazarus uses its own cache for file functions. These caches require each file name to normalize and doing binary searches with case insensitive comparisons - a lot of string operations. The speed gain of such a cache is noticeable on Linux, while on Windows it is dramatic.
That's why I doubt that some extra string conversions are measurable.

Can we now return to the real problems?
Marco gave some examples.

Mattias

Sven Barth wrote

Depending on how they implement it this might indeed be an interesting feature that we could implement (cherry picking Delphi features ^^).

It's already possible, just use IInterface (and TInterfacedObject) everywhere. :)

On 08/22/2012 07:30 PM, Ivanko B wrote:
...

You need to stop your mailer from all the time replying to the wrong message. This makes the forum rather unreadable.

-Michael

You need to work with variables of type IInterface then so that the reference counting mechanism works correctly. The compiler does not generate reference managament code if you work with a variable of type TInterfacedObject or decendant. E.g.:

var
   i: IInterface;
begin
   i := TSomeInterfacedObjectDescendant.Create;
   (i as TSomeInterfacedObjectDescendant).SomeMethodOfThatClass;
end;

This just looks ugly...

The other possibility is to define an interface for the class you want to use, but that's duplication I'd like to avoid...

Regards,
Sven