Hello,

I'm currently working for a company (Tixeo.com) that is preparing to use FPC to recompile their flagship product (written in Delphi) for certain targets.

As part of this work, we're striving to make the Free Pascal RTL and Packages more compatible with recent Delphis.

That means:

- String = UnicodeString, Char=WideChar
- Dotted units.

Before you all get a heart attack:

Because backwards compatibility is important, FPC will of course continue to distribute a backwards-compatible RTL and packages, with single-byte string. All FPC modes will continue to function.

To make this possible a feature called 'subtargets' has been implemented. This can be used for other things than the unicode RTL, but will be used for the unicode RTL.

The RTL is compiled 'normally' and with subtarget=unicodertl;
this will result in 2 separate sets of .o/.ppu files:

The normal CPU-OS directory and a CPU-OS-unicodertl directory.

The first part of the work has been pushed to gitlab in a branch called unicodertl (NOT the svn/unicodertl branch):

- subtarget support is there
- The rtl compiles with make SUB_TARGET=unicodertl with the compiler as it is in that branch.

Needless to say, this is a major change that will need thorough testing.

The earlier testing starts, the better.

For those that wish to help in testing:

- Update your git clone

- switch to branch unicodertl.

- in the toplevel FPC directory, do a

make all

- if that went well, next to your fpc.cfg, create a file fpc-unicodertl.cfg with the following contents:

----
-dUNICODERTL
-Municodestrings
----

- to create a Unicode RTL, in the rtl directory do a

make clean all SUB_TARGET=unicodertl PP=path/to/the/new/compiler

- if that worked, you can try then a

make install SUB_TARGET=unicodertl

Note that this is NOT ready for production use:

- The packages have not yet been converted (working on this now) and will certainly fail...
- a private testsuite has been run and gives no failures, but the complete compiler testsuite still needs to be examined.
- Dotted names will be created after the unicode transition is deemed complete.

if you do wish to test and spot errors in the RTL or compiler, please file a bugreport.

You can use a tag 'UnicodeRTL' to report bugs for this.

Michael.

Michael Van Canneyt wrote

- String = UnicodeString, Char=WideChar

UnicodeString ? ? I don't know what that is supposed to be. We have UCS-2, UCS-4, UTF-16 and UTF-8 Some marketing people, not understanding anything about computers, talk about "Unicode" to mean "UTF-16-treated-as-UCS-2".

For me, UTF-16 is the dumbest thing ever invented in computing. So I shiver at the thought of a "Unicode" RTL.

We really should use UTF-8, as most open source projects do.

Regards,

Adriaan van Os

There will be two RTLs: the current one (which supports any single-byte code page, including utf-8), and a unicode/utf-16 one for compatibility with recent Delphi versions.

Jonas

It is named after Microsofts use of the word unicode. Which was originally UCS2,  but morphed into UTF16 in later versions.

The reason however is Embarcadero's adaptation of it in Delphi.

For the encoding advocacy, these discussions were already held in 2009-2010. Let's not have them again.

That’s what it’s going to be, if I understood correctly. “Unicode” string (UTF-16) will be an option in addition to UTF-8.

Seems my warning to prevent heart attacks was on its place but maybe not entirely effective :-)

Relax, no-one will force you to use UTF16.

But there are people that do not have the luxury of choice, and we should be kind to them too.

Michael.

On Fri, 6 Jan 2023 18:05:43 +0100 (CET)
Michael Van Canneyt via fpc-devel

[...] For those that wish to help in testing: - Update your git clone

I used a fresh clone.

- switch to branch unicodertl.

I used git checkout unicodertl

- in the toplevel FPC directory, do a make all

I get:

Compiling .../packages/fcl-process/src/process.pp
Compiling .../packages/fcl-process/src/pipes.pp
process.inc(94,23) Error: Incompatible type for arg no. 1: Got "PChar", expected "PWideChar"

Mattias

Relax, no-one will force you to use UTF16. But there are people that do not have the luxury of choice, and we should be kind to them too.

I really appreciate this mindset. Thank you.

Lesson:
Always run a toplevel "make all" after even the smallest (no matter how insignificant) change in RTL.

Fixed. Make all works.

Thanks for testing!

Michael.

Michael Van Canneyt via fpc-devel schrieb
am Sa., 7. Jan. 2023, 12:46:

That's my usual modus operandi when working on the RTL or some compiler feature: first make sure that make inside the rtl directory works and then ensure that it also works in the top level.
There have often enough been subtle errors further down the line (good candidate, especially when working with generics, is the rtl-generics package 🙄).

Regards,
Sven

On Fri, 6 Jan 2023 18:05:43 +0100 (CET)
Michael Van Canneyt via fpc-devel

[...] - to create a Unicode RTL, in the rtl directory do a make clean all SUB_TARGET=unicodertl PP=path/to/the/new/compiler

The "make clean" deletes the new compiler.

Storing the compiler/ppcx64 somewhere else and using that gives:

*** The only supported starting compiler version is 3.2.2. You are trying to build with 3.3.1.

Trying to build using fpc 3.2.2 gives:
Error: Illegal parameter: -tunicodertl

Mattias

You probably executed this in the main directory ?

This command must be executed in the rtl directory (see: "in the rtl directory do a"), to create the unicode RTL.

Michael.

On Fri, 6 Jan 2023 18:05:43 +0100 (CET)
Michael Van Canneyt via fpc-devel

It installed under
/usr/lib/fpc/3.2.2/units/x86_64-linux-unicodertl

Instead of
/usr/lib/fpc/3.3.1/units/x86_64-linux-unicodertl

Mattias

That is because I gave a wrong command-line :/

Should have been

make install SUB_TARGET=unicodertl PP=path/to/the/new/compiler

Sorry about that.

Michael.

OK. It reminds me of the Turbo assembler, which had a mode to be compatible with the bugs in the Microsoft assembler 😃 😃 .

Regards,

Adriaan van Os

On Sun, 8 Jan 2023 08:46:37 +0100 (CET)
Michael Van Canneyt via fpc-devel

[...] Should have been make install SUB_TARGET=unicodertl PP=path/to/the/new/compiler

This does not install the new compiler, so I used the compiler/ppcx64 exe directly to compile a program.

I see in the -va output:
[0.008] Hint: Start of reading config file /etc/fpc-unicodertl.cfg [0.008] Handling option "-dUNICODERTL"
[0.008] Interpreting option "-dUNICODERTL"
[0.008] Macro defined: UNICODERTL
[0.008] Handling option "-Municodestrings"
[0.008] Interpreting option "-Municodestrings"
[0.008] Macro defined: FPC_UNICODESTRINGS
[0.008] Macro defined: UNICODE
[0.008] Hint: End of reading config file /etc/fpc-unicodertl.cfg

But then it uses the wrong unit paths:
[0.012] Using unit path: /usr/lib/fpc/3.3.1/units/x86_64-linux/httpd22/

It should be:

[0.012] Using unit path:
/usr/lib/fpc/3.3.1/units/x86_64-linux-unicodertl/rtl/

My /etc/fpc.cfg contains:

# searchpath for units and other system dependent things -Fu/usr/lib/fpc/$fpcversion/units/$fpctarget
-Fu/usr/lib/fpc/$fpcversion/units/$fpctarget/*
-Fu/usr/lib/fpc/$fpcversion/units/$fpctarget/rtl

Mattias

What does your /etc/fpc-unicodertl.cfg contain? The correct paths should be there...

Tomas

I am aware of this one.

This is fixed since some time but not yet pushed.

I will push this change together with many others later today.

I'm busy fixing pas2js resolver failures :-)

All packages compile using unicode meanwhile, json/xml work fine, pas2js for the most part as well, just the resolver needs some work. Down to 3 errors in the testsuite. Coming from over 2000 failures initially :/

Michael.

Nono, there was an error in calculating $fpctarget. It is fixed.

Michael.

Michael wrote it should contain:

----
-dUNICODERTL
-Municodestrings
----

Adding unitpaths without removing/disabling the wrong ones seems wrong to me.

Mattias