It months ago I realized that Lazarus was saving the sources like that:

1- If there is only ANSI chars, save it as ANSI encode;
2. If there is more than ANSI chars, save it as UTF8 encode;

(correct me if I'm wrong about that)

I think this was changed when I updated my FPC to 3.* and Lazarus (trunk). Everything continues works. This new behavior was completly transparent to me... but I have a doubt:

If Delphi sources don't use UTF8, how is the best way to mantain sources that need to work in both compilers?

I mean, I would like to work with FPC and Lazarus as I am used to doing, without take concerns about "remember to change the encode of these files", "remember to not put accented chars on code", or something like that.

Is there some wiki page that explains this?
Are you working on projects that have these problems? Could you tell me which is the best approach to deal with it?
Thank you.

Best regards,
Marcos Douglas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20170430/46749465/attachment.html>

Marcos Douglas B. Santos wrote

Are you working on projects that have these problems? Could you tell me which is the best approach to deal with it?

Speaking from my experience, the only approach (not only the best one but the only one) is not to use characters above #255.

Ondrej

On Sun, Apr 30, 2017 at 4:30 PM, Ondrej Pokorny via Lazarus <

I see...
Well, this is not the best approach for me, as you've said... I'm brazilian and is common to use accents here. I can forget this 'rule' easily. :(

But is this the only problem that you got?

Marcos Douglas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20170430/81841827/attachment-0001.html>

Marcos,

When I originally created the Firebird Pascal API package, I wrote it with FPC as the only intended target. However, I then got feedback asking for Delphi compatibility and I couldn't resist the challenge of seeing how easy it would be to convert the source code. In the end, it was not that difficult and the following summarises the changes needed (apart from the minor syntax changes):

1. Standard set of defines for all modules:

{$IFDEF MSWINDOWS}
{$DEFINE WINDOWS}
{$ENDIF}
{$IFDEF FPC}
{$mode delphi}
{$ENDIF}

The first one is needed to ensure that the "WINDOWS" defined symbol works with Delphi as well as FPC, while "mode delphi" is needed for FPC.

2. All strings declared explicitly as AnsiString expect where they are used with system calls (e.g. FileNames).

This rule may vary between applications. However, as FPC defaults to AnsiString and Delphi to WideString, it is usually a good idea to be explicit in your string types. In my case, most strings were the ASCII subset of UTF8 and AnsiString worked for me.

Similarly AnsiChar instead of Char.

3. PByte instead of PChar.

Most uses of PChar in FPC/Lazarus aren't to character strings but are instead used to point to untyped buffers. However, in Delphi, PChar now means widestring and you will be caught out if you try and use PChar for an untyped buffer. PByte has the same semantic in both FPC and Delphi and is a pointer to an unsigned byte.

4. PAnsiChar instead of PChar.

When PChar is used to genuinely point to a character string then as in point 2, you should be explicit about which type of string. PWideString may also be appropriate for some cases.

5. Take care when using string literals.

I added

{$IFDEF FPC}
{$codepage UTF8}
{$ENDIF}

to sources with string literals just to make sure that FPC interprets all strings as UTF8. However, you need to be careful with Delphi. In some of the test scripts I ended up with source code such as:

     {$IFDEF DCC}
     ByName('title').AsString := UTF8Encode('Blob Test ©€');
     ByName('Notes').AsString := UTF8Encode('Écoute moi');
     {$ELSE}
     ByName('title').AsString := 'Blob Test ©€';
     ByName('Notes').AsString := 'Écoute moi';
     {$ENDIF}

The first case (DCC defined) is Delphi and as my strings were all UTF8, I added an explicit UTF8Encode to force the literal to UTF8 before assigning to the string. Note the above is a test case with literals chosen to show up differences between character sets.

6. Varargs

Delphi is much fussier than FPC about varargs constructs. If you have to use them then the only way this seems to work is e.g.

function blah(...): sometype; cdecl varargs;

7. Generics

If you don't have to go here then don't. Unfortunately I had dug a hole for myself and made extensive use of generics. Some of the differences are minor syntax, but others... Delphi and FPC just don't work the same way. Examples of problems are:

- calling a constructor from a function defined using generics.

- extracting an interface from an object defined using generics.

Both are possible, but with very different syntax.

8.TSystemCodePage does not exist in Delphi

You may need to add

{$IF not declared(TSystemCodePage)}
   TSystemCodePage = word; {not defined in Delphi}
{$IFEND}

if you are using any Ansistring code page functions.

9. $IFEND or $ENDIF

As illustrated in point 8, older versions of Delphi demand $IFEND rather than $ENDIF to terminate conditional compilation.

I hope you find this a useful checklist.

Regards

Tony Whyman

MWA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20170501/5e52a5b2/attachment-0001.html>

On Sun, 30 Apr 2017 13:37:17 -0300
"Marcos Douglas B. Santos via Lazarus"

It months ago I realized that Lazarus was saving the sources like that: 1- If there is only ANSI chars, save it as ANSI encode; 2. If there is more than ANSI chars, save it as UTF8 encode;

Lazarus saves source file by default as UTF-8 without BOM.

Option a) You can use English in sources and load all non ASCII constants via resourcestrings or similar. Then the codepage is irrelevant.
Option b) You can store all files as UTF-8 with BOM. Then FPC will store all non ASCII string constants as unicodestrings. Be careful when using PChar with them. This adds implicit conversions, so it might be slower.

Mattias

Tony Whyman wrote

When I originally created the Firebird Pascal API package, I wrote it with FPC as the only intended target ...

I guess you used the old Unicode support with explicit UTF-8 conversion functions.
Things have improved a lot since then. You should read this page: http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus

2. All strings declared explicitly as AnsiString expect where they are used with system calls (e.g. FileNames).

No no no!
AnsiString is not compatible with Delphi any more. It is an unfortunate side-effect of the new Unicode support. However using "String" type works like magic in a Delphi compatible way at source level, despite their different encodings. So, do NOT use AnsiString explicitly.

However, as FPC defaults to AnsiString and Delphi to WideString

No, Delphi defaults to UnicodeString, not WideString. Lazarus with its Unicode support defaults to AnsiString with encoding changed to UTF-8. Thus it is NOT compatible with AnsiString in Delphi.

it is usually a good idea to be explicit in your string types.

No! The good idea is to use "String".

3. PByte instead of PChar. Most uses of PChar in FPC/Lazarus aren't to character strings but are instead used to point to untyped buffers.

"Some uses", not "most uses". In both Delphi and FPC strings and PChar have been misused for binary data but it was never recommended. Such code must be fixed anyway.

Yes, string literals are tricky but usually you should NOT use {$codepage UTF8}. It is explained in the wiki page. I will not repeat it here.

I added an explicit UTF8Encode to force the literal to UTF8 before assigning to the string.

You don't need UTF8Encode any more. That also means your code can be 100% Delphi compatible at source level despite their different encodings.

7. Generics

For Delphi compatible generics you can use FPC trunk and the Generics Collection lib made by Maciej.

8.TSystemCodePage does not exist in Delphi ... if you are using any Ansistring code page functions.

Windows system codepages are not supported by the new Unicode system. You must convert such data explicitly by SetCodepage() or such. This is the only thing not compatible with neither Delphi nor the old Lazarus solution. :(
However the world is moving to Unicode and this is less and less of a problem.

I hope you find this a useful checklist.

It contained so much false information that it only confuses people. :(

I want to repeat that it is possible to write code dealing with Unicode that is fully compatible with Delphi at source level. It will be compatible with a future UTF-16 solution in Lazarus as well. Encoding agnostic (UTF-8 / UTF-16) code is possible even if you must iterate individual codepoints. See the wiki page for details.

Remember these to keep your code compatible:
1. Normally use type "String".
1. Assign a constant always to a type String variable.
2. Use type UnicodeString explicitly for API calls that need it.

Juha

Tony Whyman wrote

When I originally created the Firebird Pascal API package,

Now I realize your code may have been for FPC but not for Lazarus. Even then the solution provided by LazUtils (2 files there) is good because it allows compatible and portable code. Later when FPC's UTF-16 support is ready, such code can be ported easily.

Juha

Am 01.05.2017 11:31 schrieb "Tony Whyman via Lazarus" <

: 7. Generics If you don't have to go here then don't. Unfortunately I had dug a hole

for myself and made extensive use of generics. Some of the differences are minor syntax, but others... Delphi and FPC just don't work the same way. Examples of problems are:

Would you please elaborate on these and which FPC version you targeted?

Regards,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20170501/20ee070f/attachment.html>

I assume that you mean that my code is non-visual which is indeed where I am coming from. If you want to write an application that is LCL/VCL compatible then that is another can of worms.

Your concluding remarks in your other post were:

I am not sure how much your second post rows back from this but I do think that false is a bit harsh.

You seem to be coming from a view that strings are strings and the compiler should be allowed to work out what is the appropriate string encoding for the local environment. All the programmer has to do is declare the type as "string" and all will be good. I guess that is your definition of portable code: it is agnostic as regards the string encoding.

I am coming from a much messier perspective that says a portable program has to deal with whatever string encoding is may be valid criticism to say that I was taking a particularly messy example and deriving generic rules from it - but few programs work in a vacuum and it is worth being aware of real world problems.

I my case, the real world problem is Firebird. Firebird will expect or give you a string encoded not according to the local environment but that which was specified for the database connection and it is the API user that decides this and not the API. Ideally, the user specifies UTF8, but Firebird supports many other string encodings - but not UTF16 or the original version of the library, the API
was defined using the "string" type as were the internal structures. When I looked at moving to Delphi support, there was no way that this would work if "string" suddenly became "UnicodeString". All over the place I had assumed that "string" meant "AnsiString" including checking and setting the code page in order to match the connection character set with whatever code page was being used by the API user.

Could I have written the API without being aware of the character encoding? I doubt it. The connection character set is not something that the compiler can be aware of. Part of the role of the API library is to manage the character encoding on behalf of the user. On the other hand, by defining the API using the explicit AnsiString type, it should mean that if the API user uses the "string" type, then the compiler can automatically transliterate from the API to the API user's string types when string means "UnicodeString".

So is my messy example typical or atypical? Am I correct in offering it as a source of rules. Ideally, it is atypical. However, I would observe that few programs exist in isolation. They have to deal with external objects such as files, GUIs and TCP connections. The compiler cannot work out the character encoding for itself in these cases and either your program or some intermediate library has to be character coding aware in order to deal with these objects.

The bottom line is that it would be great if we never needed to be aware of the character encoding behind the string type. However, all too often you do and, because of that, when you are writing code that is portable between platforms and compilers, you either needed to be explicit in the string type throughout your program, or at least in the modules that deal with external interfaces.

Tony Whyman

I am working with FPC 3.0.x and an example of the solution I used follows. Note that with FPC, there is no problem calling a constructor from a placeholder, while with delphi I found it necessary to add an explicit coercion. I also wanted to return the interface provided by the object. Again with FPC, this is easy. With Delphi, I found it necessary to use QueryInterface to do the same.

The difference between the two seems to be that FPC waits until the generic is instantiated before type checking while Delphi performs type checking when compiling the generic itself - or thereabouts. IMHO, FPC is much superior to Delphi in this area.

{$IFDEF FPC}
   TOutputBlockItemGroup<_TItem,_IItem> = class(TOutputBlockItem)
{$ELSE}
   TOutputBlockItemGroup<_TItem: TOutputBlockItem; _IItem: IUnknown> =
class(TOutputBlockItem)
{$ENDIF}
   public
     function GetItem(index: integer): _IItem;
     function Find(ItemType: byte): _IItem;
     property Items[index: integer]: _IItem read getItem; default;
   end;

....

{$IFDEF FPC}
{ TOutputBlockItemGroup }
function TOutputBlockItemGroup<_TItem,_IItem>.GetItem(index: integer):
_IItem;
var P: POutputBlockItemData;
begin
   P := inherited getItem(index);
   Result := _TItem.Create(self.Owner,P);
end;
function TOutputBlockItemGroup<_TItem,_IItem>.Find(ItemType: byte): _IItem;
var P: POutputBlockItemData;
begin
   P := inherited Find(ItemType);
   Result := _TItem.Create(self.Owner,P);
end;
{$ELSE}

{ TOutputBlockItemGroup }

function TOutputBlockItemGroup<_TItem,_IItem>.GetItem(index: integer):
_IItem;
var P: POutputBlockItemData;
     Obj: TOutputBlockItem;
begin
   P := inherited getItem(index);
   Obj := TOutputBlockItemClass(_TItem).Create(self.Owner,P);
   if Obj.QueryInterface(GetTypeData(TypeInfo(_IItem))^.Guid,Result)
0 then
IBError(ibxeInterfaceNotSupported,[GuidToString(GetTypeData(TypeInfo(_IItem))^.Guid)]);
end;
function TOutputBlockItemGroup<_TItem,_IItem>.Find(ItemType: byte): _IItem;
var P: POutputBlockItemData;
     Obj: TOutputBlockItem;
begin
   P := inherited Find(ItemType);
   Obj := TOutputBlockItemClass(_TItem).Create(self.Owner,P);
   if Obj.QueryInterface(GetTypeData(TypeInfo(_IItem))^.Guid,Result)
0 then
IBError(ibxeInterfaceNotSupported,[GuidToString(GetTypeData(TypeInfo(_IItem))^.Guid)]);
end;

Am 01.05.2017 18:50 schrieb "Tony Whyman via Lazarus" <

follows. Note that with FPC, there is no problem calling a constructor from a placeholder, while with delphi I found it necessary to add an explicit coercion. I also wanted to return the interface provided by the object. Again with FPC, this is easy. With Delphi, I found it necessary to use QueryInterface to do the same.

The difference between the two seems to be that FPC waits until the

generic is instantiated before type checking while Delphi performs type checking when compiling the generic itself - or thereabouts. IMHO, FPC is much superior to Delphi in this area.

Your assumption about the way FPC and Delphi do things is indeed correct.

Comments of mine follow inside your example.

{$IFDEF FPC} TOutputBlockItemGroup<_TItem,_IItem> = class(TOutputBlockItem) {$ELSE} TOutputBlockItemGroup<_TItem: TOutputBlockItem; _IItem: IUnknown> =

class(TOutputBlockItem)

{$ENDIF}

You should be able to use the second variant with FPC as well. It would allow the compiler to do more type checking when parsing the generic instead of when specializing it.

_IItem;

_IItem;

_IItem;

Is this typecast really needed?

if Obj.QueryInterface(GetTypeData(TypeInfo(_IItem))^.Guid,Result) 0

then

IBError(ibxeInterfaceNotSupported,[GuidToString(GetTypeData(TypeInfo(_IItem))^.Guid)]);

end;

"Result := Obj;" does not work? What about "Result := Obj as _IItem;"?

function TOutputBlockItemGroup<_TItem,_IItem>.Find(ItemType: byte):

_IItem;

then

IBError(ibxeInterfaceNotSupported,[GuidToString(GetTypeData(TypeInfo(_IItem))^.Guid)]);

end;

Regards,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20170501/586a9bea/attachment.html>

Tony Whyman wrote

I am not sure how much your second post rows back from this but I do think that false is a bit harsh.

Yes, sorry, it was correct when using the default types in FPC 3. However making it compatible with Delphi requires some effort, as you noticed. My experience is that the Unicode support provided by LazUtils leads to more Delphi compatible code.
The Windows system codepages are a problem, yes. Then data must be converted in some I/O layer.

Everybody please experiment with it. It can be used also for non-GUI code.

Juha

Tony, thank you for all these tips.

Best regards,
Marcos Douglas

Maybe option A could be the best.
I did not remember to use resourcestrings... it is a good tip, thanks.

Marcos Douglas

Juha Manninen wrote

On Mon, May 1, 2017 at 12:30 PM, Tony Whyman via Lazarus ... No! The good idea is to use "String".

I agree.

So, as Mattias said, we should code using ANSI chars and everything will be Ok.

> 7. Generics For Delphi compatible generics you can use FPC trunk and the Generics Collection lib made by Maciej.

Is it part of FPC? If not, could you can post the official URL?

That is I wanted to read. Thanks.

Some doubts:

Remember these to keep your code compatible: 1. Normally use type "String". 1. Assign a constant always to a type String variable.

What do you mean? Instead of create a constant, is it better create a String variable and assign the string to it?

2. Use type UnicodeString explicitly for API calls that need it.

Best regards,
Marcos Douglas

Marcos Douglas B. Santos wrote

So, as Mattias said, we should code using ANSI chars and everything will be Ok.

No, you can use all the Unicode freely.
The source files are saved as UTF-8 by default. Delphi does the same, this detail is also compatible.

> For Delphi compatible generics you can use FPC trunk and the Generics > Collection lib made by Maciej. Is it part of FPC? If not, could you can post the official URL?

It is part of FPC trunk.

means that success in assigning string literals depends on the string type.
This:

  S := 'Have 🍷 for FPC 💓 Lazarus';

always works if "S" is a "String". It may not work with other string types. It is all explained in the wiki page.

When all your string data is Unicode then you can code in a Delphi compatible way.
Only the Windows system codepages impose a problem, but I got an impression you don't need them now.

Juha

Marcos Douglas B. Santos wrote

If Delphi sources don't use UTF8, how is the best way to mantain sources that need to work in both compilers?

I wonder if I have misunderstood something about your questions. What means "Delphi sources don't use UTF8"?
AFAIK they do use UTF8.

One more clarification about assigning string data: The potential problem is only with string literals, constants. Assignment between variables always goes right thanks to their dynamic encoding in FPC 3+.

Juha

I understood, thanks.

When all your string data is Unicode then you can code in a Delphi compatible way. Only the Windows system codepages impose a problem, but I got an impression you don't need them now.

I am develop on Windows. What problems do you mean?

Marcos Douglas

Sorry about that. I stopped using Delphi at version 7, that uses ANSI. I thought that Delphi nowadays was using UTF16.
I will install Delphi Tokyo Starter and discover these things.

One more clarification about assigning string data: The potential problem is only with string literals, constants. Assignment between variables always goes right thanks to their dynamic encoding in FPC 3+.

But if I put theses constants as resourcestrings, it's Ok as Mattias told me, right?

Thank you for these tips.

Best regards,
Marcos Douglas

Juha Manninen wrote

From FPC's point of view our UTF-8 solution is a hack.

FPC's point of view or Marco's point of view? Just curious - so what is FPC’s “correct” solution then for using UTF-8 and the preferred encoding? What's the alternative they offer?

[rant]
ps:
Both FPC and Delphi is in such a messed up state when it comes to string and character types. It is the laughing stock of programming languages at the moment. At least EMBT is heading in the right direction with their Linux Delphi compiler - they completely removed AnsiString.

FPC and Delphi can learn a huge lesson from Java and Qt in how to handle string and character types.
[/rant]

Regards,
Graeme

Marcos Douglas B. Santos wrote

Sorry about that. I stopped using Delphi at version 7, that uses ANSI. I thought that Delphi nowadays was using UTF16.

They (Delphi) loves to follow Microsoft. Files are stored in UTF-8 (this is the norm), but they use UTF-16 internally.

Lazarus stores files in UTF-8 and uses UTF-8 internally.

Some background info
====================
UTF-16 was the first encoding implementation for Unicode - at a time when they thought 2-bytes will be big enough for everything. They were wrong. So then they invented UTF-8 to solve the problem. But by that time Microsoft already standardised on UTF-16, so Delphi followed suite. Linux, FreeBSD etc saw the light and used UTF-8 instead.

Regards,
Graeme

Marcos Douglas B. Santos wrote

I am develop on Windows. What problems do you mean?

Unicode is recommended also on Windows. No worries. You don't need to use the old system codepages.
People who need them must convert them explicitly because the Unicode system of Lazarus does not support them directly.

Sorry about that. I stopped using Delphi at version 7, that uses ANSI. I thought that Delphi nowadays was using UTF16. I will install Delphi Tokyo Starter and discover these things.

You confuse separate things now.
The encoding of their "String" is now UTF-16.
Source files are saved as UTF-8.

But if I put theses constants as resourcestrings, it's Ok as Mattias told me, right?

I don't think it makes any difference. You can use the full Unicode in both cases.

---
I am puzzled why there were so many misleading and confusing replies, also from knowledgeable Lazarus developers.
Remember, the question was about making sources compatible with Delphi. The person (Marcos Douglas) did not know details of how strings work in Delphi and Lazarus.
Now we finally have a system that allows (more or less) compatible code when using Unicode. Why was it not even mentioned by you guys?

For example:

Ondrej Pokorny wrote

Speaking from my experience, the only approach (not only the best one but the only one) is not to use characters above #255.

Nonsense. Full Unicode is supported.

That must be very confusing. Why didn't you just tell him to use the default Unicode support in Lazarus which allows to write Delphi compatible code, just by remembering couple of rules.

Also Tony's advice to use AnsiString explicitly is quite irresponsible for a person who looks for Delphi compatibility.
AnsiString is not Delphi compatible any more in our system and it brings you back to stone-age in Delphi, to the horrors of system codepages.

Is it possible that people still don't know how Delphi compatible the Lazarus Unicode system is (unless you need the old system codepages obviously)?
For example Lazarus developer Werner (wp) didn't know the Ansi...() string functions, like AnsiCompareStr(), are compatible with Delphi. http://forum.lazarus.freepascal.org/index.php/topic,36664.msg244619.html#msg244619 Yes they are!

Please also look at program LazUnicodeTest in components/lazutils/test/. It does advanced Unicode stuff and works in both Delphi and Lazarus. It means any code dealing with Unicode can do it.

How could this thing be communicated so that people understand? Why other Lazarus developers don't want to mention it? I am puzzled. :(

Juha

Not if you need pre-unicode Delphi support :)

(Well, Marcos didn't specify what Delphi version he wants to target but he stated "If Delphi sources don't use UTF8 [...]", which applies to pre-unicode Delphi versions.)

Ondrej

Graeme Geldenhuys wrote

At least EMBT is heading in the right direction with their Linux Delphi compiler - they completely removed AnsiString.

I must agree with you. I hope it will be removed in (far) future when nobody uses the old Windows system codepages any more. Windows already supports Unicode in everything ... except for console output! Why is that?

Anyway, please let's leave out encoding supremacy issues now. My point has been that our Unicode solution makes the encoding issues irrelevent. It is almost compatible at source level despite the different encodings.
Think how improbable that is, yet it works.
See also the encoding agnostic support provided by LazUnicode.

Encoding does not matter any more, as long as it is Unicode.

Juha

Juha Manninen wrote

Windows already supports Unicode in everything ... except for console output! Why is that?

You can start the console with UTF-8 codepage:
http://stackoverflow.com/questions/14109024/how-to-make-unicode-charset-in-cmd-exe-by-default

Then you have full unicode (utf-8) support.

Ondrej

Ondrej Pokorny wrote

You can start the console with UTF-8 codepage: http://stackoverflow.com/questions/14109024/how-to-make-unicode-charset-in-cmd-exe-by-default Then you have full unicode (utf-8) support.

Or use the much better console alternatives. The Windows platform is finally catching on to what was been available in X11 environments for years - multiple choice on consoles.

For example, take a look at ConEmu for Windows.
* Tab support built-in
* Resizeable console windows
* User defined encoding per console window
* Font choice
* better mouse & clipboard support
* User defined "console engine" per window or tab. eg: I can have Bash run in one tab and the standard windows console in another.
* color customisation
* transparency support

https://sourceforge.net/projects/conemu/

I don't know why anybody would still want to run the standard Windows console - it is 20 years behind everybody else.

Regards,
Graeme

Ondrej Pokorny wrote

Not if you need pre-unicode Delphi support :)

Ok, true. IMO such old Delphi versions should not be used any more for new code. Maintenance tasks only I think.

Fortunately there is again a free Delphi Starter edition. It means anybody can use the latest version. Things are surely getting better!

Juha

Graeme Geldenhuys wrote

I don't know why anybody would still want to run the standard Windows console - it is 20 years behind everybody else.

The reason: It is available on every Windows machine. The alternatives need to be installed first
so scripts designed for them don't work out of the box.

The "default Unicode support in Lazarus" is not compatible with Delphi. For compatibility it needs the BOM.

Do you have a link where the couple of rules are listed?

Mattias

[...] Please also look at program LazUnicodeTest in components/lazutils/test/. It does advanced Unicode stuff and works in both Delphi and Lazarus.

I tried it:

First it did not compile in Delphi, because of an unsupported inline. I fix that with an IFDEF FPC.

Then it runs. The output is somewhat hard to interpret as the Windows console shows many chars as '?' and the writelns do not explain what it is supposed to show.
The good news is that it works.

The bad news is, that it only works because Delphi silently altered the source file and added the BOM.

Back in Lazarus compiling such a file gives the error:
LazUnicodeTest.lpr(28,10) Error: UTF-8 code greater than 65535 found

You are right, that using Unicode with Lazarus only needs a couple of rules to follow. Sharing code with Delphi adds a few more rules.

Mattias

Mattias Gaertner wrote

First it did not compile in Delphi, because of an unsupported inline. I fix that with an IFDEF FPC.

Right, I had added that after testing with Delphi. The inline looks good to me, don't know why Delphi does not like it.

Then it runs. The output is somewhat hard to interpret as the Windows console shows many chars as '?' and the writelns do not explain what it is supposed to show. The good news is that it works.

Yes, I guess it should be a GUI app. The console is the only place in Windows not supporting Unicode still. Damn MS, they have supported Unicode in other APIs and apps for nearly 20 years already.

The bad news is, that it only works because Delphi silently altered the source file and added the BOM. Back in Lazarus compiling such a file gives the error: LazUnicodeTest.lpr(28,10) Error: UTF-8 code greater than 65535 found

Äh, I did not test moving it back to Lazarus.
Hmmm... why does FPC not understand the BOM?

You are right, that using Unicode with Lazarus only needs a couple of rules to follow. Sharing code with Delphi adds a few more rules.

One valid choice is to edit in Lazarus and copy to Delphi only to be built. I understood Marcos Douglas planned something like that. If code really must be edited in both, how to solve it? This was another complication I did not think about. :(

About the couple of rules to follow, I had these in mind:
1. Normally use type "String".
2. Assign a constant always to a type String variable.
3. Use type UnicodeString explicitly for API calls that need it.

They are not listed as a short list in wiki. The wiki page is more detailed. Maybe they should be listed as a "Getting started" paragraph.

Juha

Ok, thank you for these informations.

best regards,
Marcos Douglas

Well, that's the point of sharing code, isn't it?

Hmmm... why does FPC not understand the BOM?

It does. And so does Delphi. But with and without BOM have different meanings.

That's what I tried to explain to Marcos and what you described as "That must be very confusing."

4. When sharing code with Delphi use BOM or use only ASCII constants. You can choose for each unit. Either way load texts using resourcestring or similar techniques.

5. What about Char and aString[]?
6. What about PChar?

They are not listed as a short list in wiki. The wiki page is more detailed. Maybe they should be listed as a "Getting started" paragraph.

Good idea.

Mattias

Is Delphi/FPC string handling that much worse than 'C'?

To me, the great thing about AnsiString is that it provides unified handling of UTF8 and legacy codepages in a single managed type by including the code page id as a dynamic property of the string.

On the other hand, AnsiString and UnicodeString are still separate types. Why? Why should there not be a single unified string type with (e.g.) ASCII, UTF8 and UTF-16 (or MS Unicode) being just another code page?

[...] On the other hand, AnsiString and UnicodeString are still separate types. Why? Why should there not be a single unified string type with (e.g.) ASCII, UTF8 and UTF-16 (or MS Unicode) being just another code page?

Many 8bit string functions work for any 8bit encoding, and so do many 16bit string functions for any 16bit encoding. But almost no string function works for both.

Mattias

Juha Manninen wrote

How could this thing be communicated so that people understand?

It would probably help if there weren't three different pages about "Unicode Support" on the wiki, all saying slightly different and conflicting things (because they talk about different things, but that's really not obvious unless you already know) and decidedly *not* saying what a user might want to know...

Maybe split the technical internals from a "simpler" user's guide?

Martok

Tony Whyman wrote

Is Delphi/FPC string handling that much worse than 'C'?

I can’t answer about C, but compared to Java and Qt’s solution, Delphi and FPC’s solutions are terrible and very confusing.

Regards,
Graeme

Am 03.05.2017 11:34 schrieb "Graeme Geldenhuys via Lazarus" <

: For example, take a look at ConEmu for Windows. * Tab support built-in * Resizeable console windows

While not point and click you can resize console windows (and the window buffer) without problems. And it's even remembered *per shortcut*!

* User defined encoding per console window

While not selectable upon start (I think) it can easily be changed with a command.

* Font choice

Supported since at least XP

* better mouse & clipboard support

If you use the PowerShell window that experience is already vastly improved. Could be that this is the case for the default console window in Windows 10 as well.

* User defined "console engine" per window or tab. eg: I can have Bash run in one tab and the standard windows console in another.

No one stops you from running a different shell in a command window (e.g. I often switch to cmd inside PowerShell when some program give me trouble with the latter).

* color customisation

Already supported since at least Windows 7 (can't currently check older versions).

I don't know why anybody would still want to run the standard Windows console - it is 20 years behind everybody else.

Because it does its job and its part of the OS without the need to install some 3rd party application.

Regards,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20170503/71349efe/attachment.html>

Am 03.05.2017 14:37 schrieb "Tony Whyman via Lazarus" <

: On the other hand, AnsiString and UnicodeString are still separate types.

Why? Why should there not be a single unified string type with (e.g.) ASCII, UTF8 and UTF-16 (or MS Unicode) being just another code page?

Because indexed access to the string data would slow down quite a bit as the RTL would need to determine whether the string is a 1-Byte, 2-Byte, 4-Byte or multi Byte String. Yes the compiler could do optimizations for this inside loops, but it would definitely slow down -O- code.

Regards,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20170503/2196a83c/attachment.html>

I stand corrected.
If you share and edit the sources between Delphi and Lazarus then you cannot use the full Unicode. Resourcestrings work for translated Unicode texts.
If you edit only with Lazarus and copy the files for Delphi compilation then you can use full Unicode.

Actually I now remember testing with UTF-8 BOM and noticing the same issue but I forgot already. This is an unfortunate side-effect of the solution being a hack.

It is so close to being easily portable.
Why Delphi adds the BOM? Why can't it just read the file and understand it is UTF-8?
If somebody is interested, this could be solved by removing the BOM from all files by a script before opening again in Lazarus. It could even be a plugin package in Lazarus. It would be relatively easy to implement.
How many people are editing their sources in both Delphi and Lazarus?

Juha

Juha Manninen wrote

How many people are editing their sources in both Delphi and Lazarus?

Me, but I keep the files ASCII-only because I need to target all Delphi versions down to D5 :/ My customers really demand it, unfortunately. I'd like to kill these dinosaurs, believe me - not the customers but the thousand different Delphi versions from ancient eras, during the best period even 2 of them in a year :)

I still feel that code should be in English, including comments - and I am a patriot, even if it is not popular nowadays :). The utf-8 bom seems to be a stupid issue, though.

Ondrej

Juha Manninen wrote

Why Delphi adds the BOM? Why can't it just read the file and understand it is UTF-8?

Probably for the same reason as FPC: the default code page if no BOM is available and no command line option is set and no $codepage directive is found is ISO-8859-1.
A BOM does the same as the command line option -FcUTF8 or the directive {$codepage utf8}, namely switching the source codepage to UTF-8. Only then string constants that contain UTF-8 characters are converted to UnicodeString constants.

Regards,
Sven

Juha Manninen wrote

If you share and edit the sources between Delphi and Lazarus then you cannot use the full Unicode.

Quite comical considering that the FPC team always makes such a big fuss about "we want Delphi compatibility", and now it seems to be worse than ever before.

Regards,
Graeme

Yeah, sorry. I said that because I didn't know which encoding Delphi is using nowadays.
But I would like to use only Delphi with Unicode support with "dot.unit.support".

Marcos Douglas

I got it.

Best regards,
Marcos Douglas

Well, the code + its string literals are compatible. If you copy/paste them, they compile and work. Now only this small invisible BOM screws up things.

Actually the IDE could have an option to remove the BOM. No scripts or packages are needed for that. It should be easy to implement. I will look at it later, hold on ...

Juha

I don't believe that string indexing even works for UTF8 strings at present - at least not in a simple s[i] way.

Is it really that much overhead to have a simple codepage check before calling the correct function to index a string? The obvious optimisation would be to check for UTF8, then UTF16 then the Default codepage and then the rest. Or perhaps UTF16 first for Windows. With register level code you are talking about very few actual machine level operations.

To me, a unified string type would have the advantage that:

- You would only have one managed string type "string" (and hence avoids the confusion that exists today).

- You would have standard string byte length and string character length functions (which yes, in the latter case, would have to have a codepage check as above).

- String indexing could be standardised as always returning the character at position 'i' (including UTF8 strings - albeit after having to "walk" the string).

- Automatic transliteration on string compare (with code page check of course) - and perhaps with the option to specific a non-standard collation.

- Readily portable code.

- The only time that a programmer has to think about the character encoding is when writing code that interacts directly with an external interface.

How often would that extra lookup be significant compared with the benefits that unified string handling would bring? And, there is no reason why you could not retain the UnicodeString type for cases where you really need to optimise UTF16 handling.

I see the unified string type as a further extension to AnsiString to include UTF16 and UCS2 code pages together with appropriate function support.

Tony

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20170504/3c7d70b0/attachment.html>

Tony Whyman wrote

I don't believe that string indexing even works for UTF8 strings at present - at least not in a simple s[i] way.

It's simple, STOP using index arrays into strings. It doesn't work for Unicode! Use specialised code-point iterators or something similar instead.

If you expect a Byte value from s[i] then fine, but if you expect a "character" (like something you see on the screen), then no it will never work. Why? See below:

* UTF-16 will return a 2-byte value which isn't big enough to cover the full Unicode range BMP and above.

* UTF-8 will return a 1-byte value which again isn't big enough to cover all possible code points in Unicode. For UTF-8 it could be anything from 1-4 bytes.

* A "character seen on the screen" could be made up of multiple code points. eg: U+0065 (e) + U+0302 (^) gives you ê. So it might look like one "character", it is *not*. How is arraying indexing into a string supposed to handle this? It can't, unless it first normalises all Unicode strings, but even that will not work in all cases - because not all combining code points can be normalised.

Regards,
Graeme

Graeme Geldenhuys wrote

* A "character seen on the screen" could be made up of multiple code points. eg: U+0065 (e) + U+0302 (^) gives you ê. So it might look like one "character", it is *not*.

Applying better typography to that representation would yield:

U+0065 (e) + U+0302 (̂◌) = ê.

The “DejaVu Sans” font will render the above correctly.

And here is a list of Combining Diacritics, and as I mentioned before, not all of them can be normalised into a single code-point “character”. Also some languages even use multiple combining diacritics for a single "character on the screen".

https://en.wikipedia.org/wiki/Combining_Diacritical_Marks

Regards,
Graeme

[...] I don't believe that string indexing even works for UTF8 strings at present - at least not in a simple s[i] way.

It exists the same as for UTF-16 strings.

The char type does not fit widechar. You would need widechar.

And in most cases the [] are used in loops. The compiler would have to add checks on each access. It would be faster to convert the string at the beginning to UnicodeString and back at the end. A technique that many RTL functions do to support any string type.

To me, a unified string type would have the advantage that: - You would only have one managed string type "string" (and hence avoids the confusion that exists today).

You can avoid the confusion by using only one string encoding, either UTF-8 or UTF-16. The problem is that existing libraries often support only one.

[...]> - The only time that a programmer has to think about the character encoding is when writing code that interacts directly with an external interface.

That's already possible. With LazUTF8.
The problem is legacy code and sharing code with Delphi.

[...]

Mattias

Juha Manninen wrote

Encoding does not matter any more, as long as it is Unicode.

reminds me of a saying that is attributed to Henry Ford...

Any customer can have a car painted any color that he wants so long as it is black.

:)

Ok, maybe my wording was not good.
I believe everybody is happy to get rid of the horrendous Windows system codepages and the question marks in text.
Unicode is a good thing!
The LazUnicode unit provides a solution for encoding agnostic code. Yes. It is no joke.

There has been many wars about UTF-8 UTF-16 encoding supremacy. They look ridiculous for anybody who knows the complexity of Unicode. The complexity is not related to encodings, it is at the abstract Unicode level where encodings have no effect.

Currently Delphi and Lazarus use a different encoding. Some people see it as a problem. LazUnicode proves it is not a problem. You can write code that works with both. Codepoints in both encodings have variable width, thus they must use the same fundamental concepts. It was easy to write functions to support them both.
However LazUnicode does not solve the problems at abstract Unicode level and nobody claims so.

Another fact is that lots of UTF-16 code out there is broken because people treat UTF-16 as fixed width encoding for some reason. Using LazUnicode unit improves the situation. The code will inevitably work right with both encodings.
I have seen comments saying that treating UTF-16 as fixed width encoding is OK because the characters outside BMP are so rare. It is like saying that a buggy spreadsheet app is OK because it calculates the sums wrong only sometimes.
IMO such people should not do programming.

I have not seen any feedback or comments about LazUnicode so far. I guess it means that nobody uses it. :(

Juha

i was making a joke... my apologies if it didn't make the great divide... you're good! :) :) :)

+1
I purchased a commercial text editor renowned for having excellent Unicode support - at least that is what ALL the reviews said. Umm yeah, to my disappointment it internally uses UTF-16 (because it is written in Delphi), and treats UTF-16 as 2-byte fixed width! WTF!

I have not seen any feedback or comments about LazUnicode so far. I guess it means that nobody uses it. :(

You made me curious, so I want to take a look. Hopefully it doesn’t depend too heavily on the rest of LCL, so I’ll be able to use it in other projects of mine.

Regards,
Graeme

Graeme Geldenhuys wrote

You made me curious, so I want to take a look. Hopefully it doesn’t depend too heavily on the rest of LCL, so I’ll be able to use it in other projects of mine.

It has no dependency for LCL, it is part of LazUtils package just like the other Unicode stuff.
It can be used also with Delphi by copying 2 unit files. It is briefly explained here:
http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus#CodePoint_functions_for_encoding_agnostic_code

Juha

Juha Manninen wrote

It is briefly explained here:

I haven't been following FPC 3.x development much because I think the Unicode changes are terribly confusing.

A case in point. Looking at the Wiki page you listed, I read the following:

"
Since FPC 3.0 you must add the flag -FcUTF8 or add {$codepage UTF8} at the beginning of the unit.

* Note: This changes all string literals to UTF-16, increasing the size of the binary and slowing it down. That's why Lazarus does not add it by default. "

That makes NO sense to me. My units are always saved as UTF-8 encoded text. By helping the compiler out by explicitly telling it my files are UTF-8 encoded using -FcUTF8 or adding {$codepage UTF8} or saving the unit with a BOM marker breaks writeln() output under Linux/FreeBSD. Who knows what else it breaks. Apparently the breakage is because of the "NOTE" quoted above. Why the hell does FPC consider string literals UTF-16 when I explicitly told it the whole unit is UTF-8 encoded? FPC is doing the opposite of what I told it!

I simply can't see myself moving past FPC 2.6.4 at this point. FPC 3.x just doesn't make any sense.

Regards,
Graeme

I added a FAQ:
http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus#What_happens_when_I_use_.24codepage_utf8.3F

I simply can't see myself moving past FPC 2.6.4 at this point. FPC 3.x just doesn't make any sense.

AFAIK you are using UTF-8 in AnsiString in FPC 2.6.4. That works in many cases, because of double fooling the compiler. This trick does not work on Windows with RTL file functions though.
The good news is that the same trick works in FPC 3.0. And with some new tricks it is now possible to make the RTL file functions support UTF-8. Of course it would be nicer, if we don't need tricks to get Unicode.

Mattias

What tricks do you still need in 3.0.x ?

As far as I know, you don't need any tricks to work with unicode filenames or output in 3.0.2. Maybe with exception of TStrings and TFileStream.

Michael.

On Thu, May 4, 2017 at 8:53 PM, Graeme Geldenhuys via Lazarus <

To play the devil's advocate, the fact that ALL reviews said that it has excellent support for Unicode means that characters outside the BMP *are* rare. After all, BMP does include practically all languages used today.

I mean, it isn't technically correct, it is just that in practice it is good enough for a very large number of tasks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20170505/c51bc27f/attachment-0001.html>

On Fri, 5 May 2017 11:31:00 +0300

Devil's advocate: The new emojis are outside BMP.

Mattias

Juha Manninen wrote

I believe everybody is happy to get rid of the horrendous Windows

If if this is true, there is a decent need for backwards compatibility. That is why, theoretically, code aware strings is a good idea. Unfortunately the implementation of those, IMHO, is abysmal, as well in Delphi, as in fpc. (Most obvious drawback: not flexibly typed TStrings.)

-Michael

Mattias Gaertner wrote

I added a FAQ: http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus#What_happens_when_I_use_.24codepage_utf8.3F

Ah, thanks for that explanation.

AFAIK you are using UTF-8 in AnsiString in FPC 2.6.4. That works in many cases, because of double fooling the compiler. This trick does not work on Windows with RTL file functions though.

Yes and true, but fpGUI supplies its own "wrapper" RTL file functions, thus it works 100% on all platforms for years. I believe LCL used to do the same.

RawByteString type (yet another string type in FPC & Delphi's arsenal) did not exist at the time, otherwise I would probably have defined...

  TfpgString = RawByteString;

and used that everywhere.

Of course it would be nicer, if we don't need tricks to get Unicode.

Indeed, and that is why I love solutions implemented by Java and Qt Framework. They are simple, it works and not confusing. Even Embarcadero is doing some string type clean-up. Their new Linux compiler completely removed AnsiString support. After all, why do you need any other string types when you support the Unicode standard.

Regards,
Graeme

Michael Van Canneyt wrote

As far as I know, you don't need any tricks to work with unicode filenames or output in 3.0.2. Maybe with exception of TStrings and TFileStream.

Again, I didn't have time to follow FPC 3.x development much, and I was too confused with all the Unicode changes.

With FPC 3.0.x, can you now load text files from disk using TStringList and specify the encoding of the file at load time?

Something like:

  sl.LoadFromFile('some_utf8_file.txt', CP_UTF8);
  sl.LoadFromFile('some_utf16_file.txt', CP_UTF16);
  sl.LoadFromFile('some_latin1_file.txt', CP_Latin1);

etc

Regards,
Graeme

Kostas Michalopoulos wrote

After all, BMP does include practically all languages used today.

The bottom line:

Unicode Standard BMP only!

If you think that, then rather promote your application as a UCS-2 compliant application, not a Unicode compliant application.

I can't remember my exact use case at the time, but the code-points I needed to work with (using a data dump text file) were outside the BMP range. I had to use a Java based text editor to correctly edit the files.

Also, as Mattias said, the Emoji's, Musical notes, Scientific symbols, Map symbols etc all fall outside the BMP.

Regards,
Graeme

I don't know about 3.0.x but you can do it in trunk 3.1.1. I posted a patch for it (r34475). I also extended TEncoding to support AnsiString, which was the requirement for TStrings encoding support.

Ondrej

Not yet. These are the exceptions I was talking about. But the FileOpen, Assign,Reset, Write of plain pascal do work with both Unicode and plain strings.

To fix the classes issues properly, we need a unicode RTL and a ANSI RTL if we wish to remain backwards compatible: The Strings[] property can have only 1 type.

Michael.

Michael Schnell wrote

(Most obvious drawback: not flexibly typed TStrings.)

I know not everybody likes Generics, but that is where I see Generics could come in very handy. A single TStrings implementation that supports multiple string types.

Or just implement a UTF-8 version. ;-)
On a side note:
I have implemented a UTF-8 version of TStrings & TStringList somewhere on my hard drive.

Regards,
Graeme

Yes, this somewhat alleviates the problem;
but this still is a single-byte TStrings, as opposed to the WideString TStrings of Delphi. It's also still a single-byte filename argument.

Michael.

That also means FPC 3.0.x is then seriously flawed. It supports Unicode, but it also doesn't support Unicode.

So what is the suggested work-around for FPC 3.0.2 to load various text encoding files into a TStringList? Hopefully the answer is not: "there is none" :-/

Because that will seriously impair/break INI usage too. The first example off the top of my head. XML and JSON probably too.

Regards,
Graeme

Current trunk 3.1.1 can do that since r34475 - you applied it :) I don't know if you ported it back to 3.0.x, though.

TFileStream can also open files with unicode names - at least on Windows (since 3.0.0 if I am not mistaken). See

Function FileCreate (Const FileName : UnicodeString; ShareMode :
Integer; Rights : Integer) : THandle;

in rtl/win/sysutils.pas

-->> There are absolutely no limitations whatsover, AFAIK. At least I don't know any and I don't experience any.

Ondrej

Ondrej Pokorny wrote

I don't know about 3.0.x but you can do it in trunk 3.1.1. I posted a patch for it (r34475).

Fantastic! Glad to see somebody was thinking in the same train of thought as I did. :)

Is that scheduled to be back-ported to FPC 3.0.x?

Regards,
Graeme

Use the plain pascal routines to read lines from a file, fill stringlist. You can write a class helper for it.

Because that will seriously impair/break INI usage too. The first example off the top of my head. XML and JSON probably too.

No. Those have been using widestring/UTF8string since day 1.

The main problem to switch the classes unit is backwards compatibility.

Michael.

As far as I know, it is not backported, but Marco would need to confirm it.

Michael.

Use "DefaultSystemCodePage := CP_UTF8" and you can load any text in any encoding into TStrings without character loss - the file will be converted to UTF-8 in LoadFrom* and converted back in SaveTo*. So your code can handle all encodings equally.

There are no limitations and no problems whatsoever. Yes, FPC is fully unicode-ready - in case you are fine with using UTF-8 internally!

Ondrej

Michael Van Canneyt wrote

Yes, this somewhat alleviates the problem; but this still is a single-byte TStrings, as opposed to the WideString TStrings of Delphi. It's also still a single-byte filename argument.

Yes but you forget that unicode is also single-byte UTF-8. And the greatest thing about FPC: it fully supports "DefaultSystemCodePage := CP_UTF8".

Therefore you don't need WideString/UnicodeString file arguments and UnicodeString-TStrings to have full unicode support in current FPC.

Just use "DefaultSystemCodePage := CP_UTF8" and every single-byte string
is unicode enabled.

Believe me, I use it in production without any problems: I have unicode-aware TStrings, I can read files with unicode names, I can do everything with plain FPC trunk.

I don't need a 100% UTF-16 Delphi-Compatible RTL for can do that with current UTF-8 FPC RTL as well. (Honestly I think it's better for FPC to stick with UTF-8 and don't overcomplicate the RTL with UTF-16 support.)

Ondrej

Yes, and with FPC 3.0 many of them are no longer needed.

How would that help?

> Of course it would be nicer, if we don't need tricks to get Unicode. Indeed, and that is why I love solutions implemented by Java and Qt Framework. They are simple, it works and not confusing.

IMO you are comparing apples and oranges.
The FP compiler provides a very easy Unicode solution - or even two (UTF-8 and UTF-16). The problem are the old RTL and libs, which are written for system encoding, not for Unicode.
You can design in FPC an Unicode RTL just like Java and QT. fpgui and LazUtils are kind of a start of that.
Or you can help FPC finishing the Unicode RTL. So stop complaining and help them.

Even Embarcadero is doing some string type clean-up. Their new Linux compiler completely removed AnsiString support. After all, why do you need any other string types when you support the Unicode standard.

That's true for most cases.
I wonder what they do when you need to access the raw 8-bit file names, or work with large amount of 8-bit strings.

Mattias

Mattias Gaertner wrote

I wonder what they do when you need to access the raw 8-bit file names,

OSX, iOS, Android and Linux all use UTF-8 as standard, so filename access is not going to be any problem. Windows is moving more and more towards UTF-16 everywhere, so that shouldn't be a problem either.

or work with large amount of 8-bit strings.

Why would you want to? Unicode supports all languages, there simply is no need for other non-Unicode encodings any more. If it is memory usage you are worried about, convert your 8-bit strings as UTF-8 encoded text (most Western countries text will all use low memory then - compared to UTF-16 as an alternative).

Java has only supported Unicode since its inception in 1995, and Java runs everywhere. It's never had a problem running on non-Unicode enabled platforms.

Regards,
Graeme

I am aware of this, I do it myself.
But I work on Linux, where UTF8 is the norm.

So I cannot vouch for other platforms...

I don't need a 100% UTF-16 Delphi-Compatible RTL for is the crux of the problem.

Is this wanted/needed or do we stick to UTF8 ?

We claim Delphi compatibility.
So IMHO we must provide a UTF-16 Delphi compatible RTL.

Michael.

[...] > or work with large amount of 8-bit strings. Why would you want to? Unicode supports all languages,

Maybe there is a misunderstanding. Let me rephrase my question: What string do you use in Linux Delphi when working with UTF-8 strings?

Mattias

Michael Van Canneyt wrote

We claim Delphi compatibility. So IMHO we must provide a UTF-16 Delphi compatible RTL.

In the end it’s about supporting Unicode. Does it really matter what internal encoding it is to achieve the “Unicode support” goal?

Regards,
Graeme

Embarcadero realized they made a mistake when they disabled (yes, only disabled not removed) 8-byte strings from NEXTGEN compilers. UTF8String and RawByteString are back for all NEXTGEN compilers since 10.1. You can use them in Linux Delphi as well.

http://andy.jgknet.de/blog/2016/05/system-bytestrings-for-10-1-berlin/

Ondrej

It worked on Linux since years.
Since FPC 3.0 it works on Windows too.

Mattias

Wow. I guess that means FPC lost the title of
"compiler with most confusing string types".

Mattias

Michael Van Canneyt wrote

What tricks do you still need in 3.0.x ?

The annoying tricky part with our UTF-8 solution is the assignment of Unicode string literals.
With UTF-8 BOM it does not work at all, as discussed here. Without BOM it depends on string type + compiler settings in an illogical way. We would need a more robust solution for that. Do you have ideas?

Juha

I propose to let the compiler observe the BOM.
But I don't think more is needed.

Michael.

Graeme Geldenhuys wrote

In the end it’s about supporting Unicode. Does it really matter what internal encoding it is to achieve the “Unicode support” goal?

From a performance perspective it may be unwanted
to convert string encodings back and forth all the time.

Although, in my file manager I use UTF-8 internally and convert to/from UTF-16 for all Windows API functions and I never found any problem with it.
The time that the API functions requires is so much longer than the time for string conversion that it does not fast API-functions like changing attributes only take a second for thousands of files.

A situation where it may be a problem is when reading (UTF-16 encoded) text files.
But I never stumbled over such a thing yet.

I would promote the use of UTF-8 whereever possible while converting to target encodings only when unavoidable. It makes life much easier if you only concentrate on one (the best) Unicode encoding (UTF-8).

Therefore I see no use of a UTF-16 bases RTL.
I don't think that you would notice any performance difference to the UTF-8 based RTL.
It would only waste valuable time that can be invested in other things.

For now I am only on Windows and I have to say loadly: IT WORKS GREAT :)

I write code that is compatible with FPC and Delphi 5 - 10.2 and it works fine. So you already have a Delphi-compatible RTL. The only (well documented) difference is that FPC uses single-byte string and Delphi uses 2-byte string.

The only place where you need to handle the difference is where you need the size of char (when you access string as buffer) - which is particularly low-level code:

MyStream.WriteBuffer(MyString[1], Length(MyString) * SizeOf(Char));

-> you need the extra SizeOf(Char) and not a constant (1 for fpc, 2 for unicode Delphi).

That's all. All high-level code is compatible already. Good job. I really do think it's not worth it to pollute FPC RTL with UnicodeString overloads of every function, class etc.

Better to keep 1 clean approach (UTF-8 RTL) and not confuse people with 2 approaches (UTF-8 vs UTF-16). E.g. how do you want to call the new UnicodeString-TStrings class? You have 2 options:
1.) Break compatibility to legacy FPC. (New TStrings will use UnicodeString.)
2.) Break compatibility to Delphi. (TStrings will stay with 8-byte string.)

There is no obvious solution for the problem :/

And then if you will introduce a compiler switch to change String from 1-byte to 2-bytes... Oh no, so much mess and so many variants to care about. Really, sometimes it's better to give people no options :) (Or have you already introduced the switch?)

Just stick with current utf8 approach that proved well :)

Ondrej

Jürgen Hestermann wrote

A situation where it may be a problem is when reading (UTF-16 encoded) text files.

No, you convert the file on the fly, there is almost 0 performance penalty.

Ondrej

Ondrej Pokorny wrote

Just use "DefaultSystemCodePage := CP_UTF8" and every single-byte string is unicode enabled.

So does that mean you don't have to also call the following two functions (which LCL does).

 SetMultiByteConversionCodePage(CP_UTF8);
 SetMultiByteRTLFileSystemCodePage(CP_UTF8);

So doing

   DefaultSystemCodePage := CP_UTF8;

is all you need to switch the RTL, FCL and the String data type to UTF-8?

If so, when why does LCL also call the above two functions?

Regards,
Graeme

Graeme Geldenhuys wrote

A case in point. Looking at the Wiki page you listed, I read the following: " Since FPC 3.0 you must add the flag -FcUTF8 or add {$codepage UTF8} at the beginning of the unit. ...

Uhhh, the same page in "String Literals" section says: "In most cases {$codepage utf8} / -FcUTF8 is not needed." which is the correct information.

Also this wiki page has become a mess when many people add stuff but nobody removes any.
For example Michl added the grid about how constant assignment works with and without {$codepage utf8}. It is nice but he didn't remove the other paragraphs explaining the same thing. It looks like an extremely complex topic for a new user, while in reality he should code like with Delphi + remember only few simple rules.
Then Mattias adds FAQs contradicting the earlier texts ...

The comment from Martok was valid. This page is not good for users who just want to get started quickly.
I will simplify the page. I will remove stuff and move the FAQ to a new page. Sorry in advance for people who's text will be removed.

Juha

Windows is also the only platform (of any modern OS) that doesn't use UTF-8 as standard - so I consider it the minority.

A situation where it may be a problem is when reading (UTF-16 encoded) text files.

I'm yet to find a UTF-16 encoded text file in the wild. I'm not saying they don't exist, I'm just saying they are extremely rare and more like an anomaly. UTF-8 seems to rule the roost and the Internet.

This graph should say it all:

https://en.wikipedia.org/wiki/File:Utf8webgrowth.svg

(source): https://en.wikipedia.org/wiki/UTF-8

Even so, a simple conversion to UTF-8 at load time should resolve all possible problems.

Regards,
Graeme

Graeme Geldenhuys wrote

In the end it’s about supporting Unicode. Does it really matter what internal encoding it is to achieve the “Unicode support” goal?

Yep it does.

There are ways around that issue (i.e. code aware strings) but in fact these trigger a new bunch of problems.

You might want to read ->
http://wiki.freepascal.org/not_Delphi_compatible_enhancement_for_Unicode_Support

-Michael

[...] I propose to let the compiler observe the BOM. But I don't think more is needed.

FPC observes the BOM. Same as Delphi.

I wonder if it would help if FPC would store UTF-8 string literals as UTF-8 and how much work that is.

Mattias

Then what is still the problem ?

I wonder if it would help if FPC would store UTF-8 string literals as UTF-8 and how much work that is.

Amount of work is probably not so much.

The question is whether it will cause problems for e.g. the JVM code generator.

But I still fail to see the actual problem, aside from a lot of confusion by users.

Confusion arises from a lack of clear information.

So in my opinion, we simply need to provide clear information, before we start changing things.

Michael.

Mattias Gaertner wrote

I wonder if it would help if FPC would store UTF-8 string literals as UTF-8

Yeah, that would be the logical thing to do. FPC not doing that is what really confused me.

Regards,
Graeme

SetMultiByteConversionCodePage does only one thing: it sets DefaultSystemCodePage :) So yes, if you set DefaultSystemCodePage you don't have to call SetMultiByteConversionCodePage.

You are right - I forgot about
SetMultiByteRTLFileSystemCodePage/DefaultRTLFileSystemCodePage.

BUT if I take a look into the RTL sources I see that it's used only in FindFirst/FindNext, FExpand and GetDir/do_GetDir. And only in the result strings. IMO it could be removed and replaced with DefaultSystemCodePage.

Ondrej

Michael Van Canneyt wrote

Then what is still the problem ?

With BOM you get:
Error: UTF-8 code greater than 65535 found
which is counter-intuitive when the file and the string literal are both UTF-8. It is related to changing the default codepage at run-time which is a hack from FPC's POV.
For the same reason we need this grid:
http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus#String_Literals_Overview So, it is not only a communication issue. It is truly messy. If only it could be improved...

Juha

Graeme Geldenhuys wrote

If so, when why does LCL also call the above two functions?

Graeme, they are called by LazUtils package, LazUTF8 unit, not by LCL. It is not limited to GUI programming.
A wrong information easily propagates, thus it is important to get this right.

Juha

Juha Manninen wrote

A wrong information easily propagates, thus it is important to get this right.

No worries, I agree. Thanks for correcting my terminology.

Regards,
Graeme

Am 05.05.2017 13:50 schrieb "Juha Manninen via Lazarus" <

UTF-8.

That is mainly due to the compiler not supporting surrogate pairs for the UTF-8 -> UTF-16 conversion. If it would support them, then there wouldn't be a problem anymore... (though of course it would need to be ensured that other parts of the RTL support surrogate pairs as well)

Regards,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20170505/6da03152/attachment.html>

[...] Then Mattias adds FAQs contradicting the earlier texts ...

Oops. Which one?

Mattias

Sven Barth wrote

That is mainly due to the compiler not supporting surrogate pairs for the UTF-8 -> UTF-16 conversion. If it would support them, then there wouldn't be a problem anymore...

That is a serious bug. Getting codepoints right is the absolute minimum requirement for Unicode support. Surrogate pairs are the UTF-16 equivalent of multi-byte codepoints in UTF-8.

Now I understand this was not caused by our UTF-8 run-time switch "hack". It is a plain bug in FPC.
Is there a plan to fix it?

Juha

Mattias Gaertner wrote

Oops. Which one?

The FAQ says:
"Since FPC 3.0 you must add the flag -FcUTF8 or add {$codepage UTF8} at the beginning of the unit."

The same page in "String Literals" section says:
"In most cases {$codepage utf8} / -FcUTF8 is not needed." which is the correct information.

Actually I don't know if that FAQ entry is yours. Many people have added stuff there. The page is intimidating for a user who just wants to support Unicode without a fuss.

Juha

I improved it a bit.

Mattias

Why? What makes a string literal UTF-8?

FPC not doing that is what really confused me.

You have to distinuish between source encoding and string encoding.

Here: $codepage utf-8 and non ASCII.

> FPC not doing that is what > really confused me. You have to distinuish between source encoding and string encoding.

Yes, but sometimes the string encoding is not obvious:

{$codepage utf8}
const s = 'äöüالعَرَبِيَّة';
begin
  writeln(s); // needs widestringmanager
end.

Mattias

Yes. Which is good imo. The compiler should call the unicode writeln in this case and this requires a widestringmanager (where I agree, its name is chosen wrong, it should be named unicodestringmanager) anyways. If the string were encoded UTF-8 and the ansistring writeln is called, your example would work only on a system with an utf-8 console. But you can achieve the same (broken output) by just leaving away the codepage directive then. It will not work on non utf-8 consoles either.

Florian Klaempfl wrote

> Yeah, that would be the logical thing to do. Why? What makes a string literal UTF-8?

As Mattias said, the fact that the source unit is UTF-8 encoded. Defined by a BOM marker, or -Fcutf8 or {$codepage utf8}. If the source unit is UTF-8 encoded, the literal string constant can't (and shouldn't) be in any other encoding.

I would say the same if the source unit was stored in UTF-16 encoding. Then string literals would be treated as UTF-16.

It's perfectly logical to me.

Regards,
Graeme

And if a ISO/Ansi codepage is given? Things would probably fail.

The point is: FPC is consistent in this regard: also sources with a given iso/ansi codepage are handled the same way. If there is a string literal with non-ascii chars, it is converted to UTF-16 using the codepage of the source. Very simple, very logical. It is a matter of preference if UTF-8, -16, -32 are chosen at this point, but FPC uses UTF-16. If it uses UTF-8, the problem would occur the other way around.

If no codepage is given (by directive, command line, BOM), string literals are handled byte-wise as raw strings.

It's perfectly logical to me.

It is logical only in a limited view.

Am 07.05.2017 12:07 schrieb "Florian Klaempfl via Lazarus" <

Small correction: FPC only does this conversion if the codepage is UTF-8, no other.

Regards,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20170507/f64c6a11/attachment.html>

Then something is wrong/broken :)

Florian Klaempfl wrote

Then something is wrong/broken :)

I rest my case. :-P

Regards,
Graeme

Well, the code in tscannerfile.readtoken() only does the conversion to UTF-16 if the source codepage is UTF-8, otherwise it only converts to UTF-16 if the string is already an UTF-16 string.
So probably not broken as it seems rather on purpose; if at all it's wrong...

Regards,
Sven