Hi,

"official" Unicode support in FPC has been discussed for so long (years), that I came up with my own solution for fpGUI since 2007 (LCL did the similar) and it still works 100% and well tested. But now I want to learn what the FPC team finally came up with in FPC 3.0.

Is there some white paper, blog post, awesome MvC article that explains it all. And how it compares to Delphi's Unicode support (if this is important - though I don't use Delphi at all any more).

For example, I looked at a few recent messages here and in the Lazarus mailing list. I also looked at a length discussion in the Lazarus Forum which had some example code. I tried out a few things with various String CodePage valuse and writeln() statements using a UTF-8 encoded .pas unit, and I got horrible results on my FreeBSD 10.1 system. The Unicode (UTF-8) output to console that I expected to work simply didn't. I need to figure out why.

There seems to be a million and one tweaks (options) one can now apply to the FPC compiler, the RTL, different modes can be set in each unit etc. It is all very confusing.

So basically what I'm asking, is there a Unicode (or is that the wrong term) chapter or something in the FPC documentation that describes all this in easy to understand English. The do's and don't, what works and what doesn't in FPC 3.0, is the RTL fully Unicode enabled now etc.

Regards,
- Graeme -

Graeme Geldenhuys wrote on Thu, 10 Dec 2015:

Is there some white paper, blog post, awesome MvC article that explains it all. And how it compares to Delphi's Unicode support (if this is important - though I don't use Delphi at all any more).

http://wiki.freepascal.org/FPC_New_Features_3.0#Support_for_codepage-aware_strings

Jonas

Thanks, but here's the problem. The wiki says FPC 3.0 is now Delphi 2009 string compatible. So I starting reading the "Delphi_and_Unicode.pdf" white paper. I tried the first example shown on page 7 (code below):

var
  ch1: Char;
  ch2: AnsiChar;
begin
  ch1 := 'ù';   // error
  Memo1.Lines.Add ('WideChar');
  Memo1.Lines.Add ('UpCase ù: ' + UpCase(ch1));
  Memo1.Lines.Add ('ToUpper ù: ' + ToUpper (ch1));  // warning
  ch2 := 'ù'; // error
  Memo1.Lines.Add ('AnsiChar');
  Memo1.Lines.Add ('UpCase ù: ' + UpCase(ch2));
  Memo1.Lines.Add ('ToUpper ù: ' + ToUpper(ch2)); // warning
end;

Tried to compile that with FPC (yes, I included the new "Character" unit in uses clause) and I get a couple of compiler errors. I tried first with mode OBJFPC and then with mode DELPHI. Same compiler errors.

Compile Project, Target: project1: Exit code 1, Errors: 3, Warnings: 2
project1.lpr(35,10) Error: Incompatible types: got "Constant String"
expected "Char"
project1.lpr(38,50) Warning: Implicit string type conversion with
potential data loss from "WideString" to "AnsiString"
project1.lpr(39,10) Error: Incompatible types: got "Constant String"
expected "Char"
project1.lpr(42,50) Warning: Implicit string type conversion with
potential data loss from "WideString" to "AnsiString"
project1.lpr(54,13) Error: Variable identifier expected

So already it seems information is missing for FPC usage, or FPC clearly isn't as "delphi compatible" as the wiki makes out to be. Or something else went astray. This is exactly what I'm talking about. It clearly doesn't "just work". I'll read the FPC_Unicode_support wiki from top to bottom. Maybe that sheds some light.

My test project was created using Lazarus (Project -> New Application). I tried the same using a fpGUI toolkit project too.

Regards,
- Graeme -

2015-12-10 11:18 GMT+01:00 Graeme Geldenhuys
:

And mode DelphiUnicode?

Vincent

Graeme Geldenhuys wrote on Thu, 10 Dec 2015:

The language feature is implemented in a Delphi 2009-compatible way (although some Delphi-specific behaviour is only activated in {$mode delphiunicode}). Not all RTL (let alone Lazarus, like for TMemo) support is ready yet. You even explicitly added an example at http://wiki.freepascal.org/FPC_Unicode_support#RTL_todos

So already it seems information is missing for FPC usage,

It's all documented on the wiki page you mention below.

It is the "white paper [or] blog post" you were asking about.

Jonas

Vincent Snijders wrote

And mode DelphiUnicode?

Oh, yet another compiler mode. Okay, just tried, and got these warnings, but the project at least compiled and generated a binary.

Compile Project, Target: project1: Success, Warnings: 5 project1.lpr(37,47) Warning: Implicit string type conversion with potential data loss from "UnicodeString" to "AnsiString" project1.lpr(38,50) Warning: Implicit string type conversion with potential data loss from "UnicodeString" to "AnsiString" project1.lpr(41,36) Warning: Implicit string type conversion from "Char" to "UnicodeString"
project1.lpr(41,47) Warning: Implicit string type conversion with potential data loss from "UnicodeString" to "AnsiString" project1.lpr(42,50) Warning: Implicit string type conversion with potential data loss from "UnicodeString" to "AnsiString"

Now I try and run the binary and get the following error.

[t2]$ ./project1
This binary has no unicodestrings support compiled in. Recompile the application with a unicodestrings-manager in the program uses clause.
An unhandled exception occurred at $000000000044354D: ENoWideStringSupport: SIGQUIT signal received.
$000000000044354D

Okay, so I add the "unicodestrings-manager" unit to the program unit's uses clause, as the above message suggests.

Now I can't compile the project...

project1.lpr(10,43) Fatal: Cannot find unicodestrings used by project1
of the Project Inspector.

Yeah, my experience with FPC Unicode is going really well. ;-)

Regards,
- Graeme -

Jonas Maebe wrote

The language feature is implemented in a Delphi 2009-compatible way (although some Delphi-specific behaviour is only activated in {$mode delphiunicode}).

Okay, so forget Delphi 2009 compatibility for the moment (I don't use Delphi). So does this mean to use "official" FPC Unicode, I can't use {$mode objfpc} any more?

Not all RTL (let alone Lazarus, like for TMemo) support is ready yet.

No problem, I changed the example to a console program and used WriteLn() instead.

Does WriteLn() work correctly with Unicode text? Will it detect my console encoding used in mate-terminal or xterm under FreeBSD? If not, what else must I do to test this Unicode support?

> So already it seems information is missing for FPC usage, It's all documented on the wiki page you mention below.

As past experience has shown me, the wiki is a very unreliable source - often things are way out of date, or "concepts" was documented but never removed when something else was decided etc.

But I'll spend my lunch time reading the wiki page in question and see if anything helps. But like I said, the experience thus far is anything but ideal or straight forward.

It is the "white paper [or] blog post" you were asking about.

Both. I can't follow the examples shown in the Delphi_and_Unicode.pdf, because they simply don't compiler or don't run.

The wiki pages doesn't have a single compilable "test program" example that can get you going.

Here is the Lazarus Forum discussion on FPC's new String support in 3.0. I tried that example on my FreeBSD system too, and the console output was total rubbish no matter what I tried.

http://forum.lazarus.freepascal.org/index.php/topic,30200.msg191799.html#msg191799

Regards,
- Graeme -

Okay, I changed my program to a console program. After much searching I found that "unicodestrings-manager" is not actually a unit. I tried "cwstrings" instead - the wiki page didn't mention this - and could compile and run my program.

[t2]$ ./project1
WideChar
UpCase ù: Ù
ToUpper ù: Ù
AnsiChar
UpCase ù: ?
ToUpper ù: ?

The output is not the same as the output from the
"Delphi_and_Unicode.pdf" example.

Source code:
The .pas unit is stored as a UTF-8 encoded file, and the program was run in mate-terminal and xterm, showing the same output as above.

===========================
program project1;
{$mode delphiunicode}{$H+}

uses
cwstring,

  Classes, SysUtils, Character;

var
  ch1: Char;
  ch2: AnsiChar;
begin
  ch1 := 'ù';
  Writeln('WideChar');
  Writeln('UpCase ù: ' + UpCase(ch1));
  Writeln('ToUpper ù: ' + ToUpper (ch1));
  ch2 := 'ù';
  Writeln('AnsiChar');
  Writeln('UpCase ù: ' + UpCase(ch2));
  Writeln('ToUpper ù: ' +  ToUpper(ch2));
end.
===========================

Regards,
- Graeme -

Try {$codepage utf8}.

Martin

Graeme Geldenhuys wrote on Thu, 10 Dec 2015:

There is no such thing as "official FPC Unicode". We added support for codepage-aware ansistrings. We added unicodestring overloads for a number of RTL routines, and made sure that those same routines also correctly handle the ansistring codepage information. This functionality is always available everywhere, and behaves the same everywhere.

We also added the {$mode delphiunicode} syntax mode, which changes some type definitions ((p)char = (p)widechar, string in {$h+} = unicodestring), and the default source file code page in a Delphi 2009 and later compatible way. It does not modify the above functionlaty, nor is it required.

No problem, I changed the example to a console program and used WriteLn() instead.

The test program you posted later does not work completely because it uses routines that have not yet been adapted (UpCase and ToUpper are not mentioned in
http://wiki.freepascal.org/FPC_Unicode_support#RTL_changes ).

Does WriteLn() work correctly with Unicode text?

Yes.

Will it detect my console encoding used in mate-terminal or xterm under FreeBSD?

It will use whatever code page has been (correctly) configured for your terminal, yes.

As past experience has shown me, the wiki is a very unreliable source -

Wiki articles that are linked from official release notes are by definition supported and guaranteed to be reliable by us.

But I'll spend my lunch time reading the wiki page in question and see if anything helps. But like I said, the experience thus far is anything but ideal or straight forward.

You seem to be looking for a tutorial. We have written no such document. There is only a technical description of what has been added and how it behaves.

Jonas

Martin Schreiber wrote on Thu, 10 Dec 2015:

That won't change anything, because {$mode delphiunicode} implies "use the default system code page to interpret the source code", and that's UTF-8 on his system.

Jonas

Does that mean a {$mode delphiunicode} resets a -Fc parameter?

Mattias

Martin Schreiber wrote

Try {$codepage utf8}.

It made no difference to the console output.

Regards,
- Graeme -

Jonas Maebe wrote

There is no such thing as "official FPC Unicode". We added support for codepage-aware ansistrings. We added unicodestring overloads for a

Does the type UnicodeString always mean UTF-16 encoded text (like it's Delphi counterpart)? If so, that unfortunate.

Can the encoding of UnicodeString be changed to say UTF-8? And yes, I know there is a UTF8String type too, but I'm specifically referring to UnicodeString (as in the Unicode standard supporting multiple encodings), as many RTL functions seem to use that specific type.

The test program you posted later does not work completely because it uses routines that have not yet been adapted (UpCase and ToUpper are

OK, so FPC unicode is still far from usable, as some RTL function will work, and others won't. :-/ With that conclusion, I'll then keep fpGUI's Unicode support exactly as it currently is.

It will use whatever code page has been (correctly) configured for your terminal, yes.

Excellent, thanks.

> As past experience has shown me, the wiki is a very unreliable source - Wiki articles that are linked from official release notes are by definition supported and guaranteed to be reliable by us.

Very good to know.

You seem to be looking for a tutorial. We have written no such document.

Umm, with all the hype around FPC + CodePage-aware AnsiStrings, I would have thought such a tutorial would be a given with the FPC 3.0 release. It sure would reduce a lot of questions and confusion. But as I mentioned above, maybe it's because the RTL is not really ready for the new FPC feature, so hence the sparse information out there. [just a thought]

Regards,
- Graeme -

My public PGP key: http://tinyurl.com/graeme-pgp

Mattias Gaertner wrote on Thu, 10 Dec 2015:

Yes, similar to how a {$codepage XXX} directive does.

Jonas

Yes.

Can the encoding of UnicodeString be changed to say UTF-8?

No.

And yes, I know there is a UTF8String type too, but I'm specifically referring to UnicodeString (as in the Unicode standard supporting multiple encodings), as many RTL functions seem to use that specific type.

Many RTL functions support UTF8String, AnsiString too, as Jonas already pointed out.

If you have a working Unicode solution there is no reason to abandon it. The FPC team did a great job keeping compatibility (contrary to Delphi).

Mattias

Mattias Gaertner wrote on Thu, 10 Dec 2015:

If you have a working Unicode solution there is no reason to abandon it.

Exactly. There is absolutely no point in changing your code to unicodestring in reaction to these changes (nor in reaction to future changes an sich, if we can help it). There's not even any point to changing your code at all in many cases. And all functionality that has been added is indeed also available when using single byte strings, as explained on the FPC_Unicode_support page.

The main reason so few RTL-level features are supported yet is exactly because it takes a lot of planning and energy to keep functionality as backward compatible as possible. If we had just abandoned backward compatibility and made everything UTF-16 only, we would probably have finished a long time ago.

Jonas

Mattias Gaertner wrote

If you have a working Unicode solution there is no reason to abandon it. The FPC team did a great job keeping compatibility (contrary to Delphi).

Indeed - from what I have tested and seen using FPC 3.0 with fpGUI. So kudos to the FPC team for that!

Regards,
- Graeme -

My public PGP key: http://tinyurl.com/graeme-pgp