Hi,

I'm using Lazarus (trunk) and FPC 2.6.2 to programming on Windows (XP, 7 and 8). As everyone knows, Lazarus is UTF8 but FPC 2.6.2 is ANSI.

I would like to know how do you working (on Windows) when uses Lazarus.

How I work:
1. I use [string] to represent any type string. But some libs (DLL,
ActiveX, etc) uses WideString;
2. If I have to create a file, I use UTF8ToSys(FileName)... and if I
have a TStringList I use SS.Text := UTF8ToSys(Text)... and at the end
SS.SaveToFile(UTF8ToSys(FileName));

I don't know about you think, but this is VERY frustating. If you forget only one call to UTF8ToSys... errors can happen, and the same to SysToUTF8.

So, I searched for a good solution but the Wiki stay the same: http://wiki.freepascal.org/LCL_Unicode_Support#Dealing_with_directory_and_filenames

...use SysToUTF8, UTF8ToSys.

Then I searched in Lazarus' sources and -- as you can see in image attachment -- Lazarus uses the same "technique".

I know the FPC team are working in a version of FPC Unicode but I don't know when it will be available -- and if the problem will persist.

I have many systems coded in FPC+Lazarus only to run on Windows so I ask you: Is there some trick to make the FPC+Lazarus to use only ANSI?

Marcos Douglas
-------------- next part --------------
A non-text attachment was scrubbed...
Name: screenshot.PNG
Type: image/png
Size: 46757 bytes
Desc: not available
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20131215/f73a44d3/attachment-0001.png>

How I work: 1. I use [string] to represent any type string. But some libs (DLL, ActiveX, etc) uses WideString;

That's Windows specific, not portable.

2. If I have to create a file, I use UTF8ToSys(FileName)...

Okay for filenames, even if IMO it should not be necessary.

and if I have a TStringList I use SS.Text := UTF8ToSys(Text)... and at the end SS.SaveToFile(UTF8ToSys(FileName));

Why that?

I have many systems coded in FPC+Lazarus only to run on Windows so I ask you: Is there some trick to make the FPC+Lazarus to use only ANSI?

Why that? Lazarus is using UTF-8 throughout, so that writing and reading files will work the same on all targets.

DoDi

The Tstringlist.save* routines that Marcos mentions are FPC.

We have TStringListUtf8 and TFileStreamUTF8 in LazUtf8Classes unit.

Furthermore all file routines in FileUtil and LazFileUtils do all this conversion automatically for you. They excpest UTF8 strings as their parameters (all LCL strings are UTF8) and (on NT based platforms) use WideString API to implement this.

Mutatis mutandis for ParamStrUtf8.

So for basic stuff all this is already taken care of.

As for DLL, ActiveX, well that is platform specific, and you will have to convert.

Bart

Yes and this is not a problem.

> 2. If I have to create a file, I use UTF8ToSys(FileName)... Okay for filenames, even if IMO it should not be necessary.

Not Ok for me. Because this I wrote this mail...
It is not necessary ONLY if you uses ANSI path names.

As Marco van de Voort said, SaveToFile is FPC.

That is the problem! Lazarus write files in UTF-8 but I can't write my (all) Windows files in UTF-8.

Marcos Douglas

Only in Lazarus' context... but I have some components that is only FPC's context.
These components do not use Lazarus' routines and that is the BIG problem. I need to "remember" in pass only ANSI strings for these components as remember to convert the component's output string results to use in Lazarus.

As for DLL, ActiveX, well that is platform specific, and you will have to convert.

Forget that, this is not a problem. I'm using many DLL and ActiveX and I know this is not portable... that's Ok. Do not need any conversion.

Thanks,
Marcos Douglas

Yes.

I know you use Delphi too and, of course, Windows. Could you tell me how you work with this problem?

Thanks,
Marcos Douglas

Why not just include a project reference to LCLBase (IIRC that should be enough) and just always use the LCL units until FPC catches up?

You propose include LCL in packages that not have LCL references and change all code of these packages to use UTF8 functions??

I think didn't understand what you proposed...

Best regards,
Marcos Douglas

I think this was also meant in context of string encoding and not that ActiveX itself is not portable...

Regards,
Sven

I think I don't understand what you're after? You said yourself you don't want to pass ANSI strings!?!?

My suggestion is to replace calls to the FPC (ANSI) RTL with LCL UTF8 equivalents in your code where possible.

Apart from that there's not much else you can do except contribute patches to help "unicode-ise" the FPC RTL...

Note:
The Lazarus UTF8 functions are in the package LazUtils. You don't need to use the LCL (nor LCLBase) if you only need the UTF8 functions.

Mattias

I said "a trick to use only ANSI".

My suggestion is to replace calls to the FPC (ANSI) RTL with LCL UTF8 equivalents in your code where possible.

But FPC still ANSI and all read/write in FPC is ANSI. We can not override these internal FPC's routines... and, some packages are used by others non-GUI program and that not have dependencies to LCL.

Apart from that there's not much else you can do except contribute patches to help "unicode-ise" the FPC RTL...

Ok... But first I would like to know who have these same problems, like me, working on Windows and what these people do to bypass these problems.

Regards,
Marcos Douglas

Sigh. I don't think you understood what I wrote.
And LCLBase doesn't pull in GUI stuff, IIUC...

As is customary whenever somebody mentions the magic "U" word, I'm sure you'll get a lot of responses....

You're right.
IMHO, the real problem is not to use LCL/LazUtils but having multiple calls to SysToUTF8..UTF8ToSys in all code.

Question:
After the new FPC version is released, how Lazarus will work together "FPC Unicode"?

Marcos Douglas

So explain better, please.

And LCLBase doesn't pull in GUI stuff, IIUC...

Sigh. I know.
I think you did not understand is:
Some packages do not work (internally) using UTF8 strings. I can change all code to use UTF8 in all package and all projects... but this would be a massive work.

Sorry, I didn't understand... (sarcasm?)

Thank you,
Marcos Douglas

I tried 2 times already, sorry. It's not important anyway.

And my point is (unfortunately also... sigh...) that you cannot expect that to change magically.
Conversion is going to be required unless and until... well I'll stop here, see the many U* threads on this list, the FPC one and fpc-devel.

> As is customary whenever somebody mentions the magic "U" word, I'm sure > you'll get a lot of responses.... Sorry, I didn't understand... (sarcasm?)

Yes, but not directed at you - at the reactions of this list whenever that word with U is mentioned ;)

The problem is not specific to the LCL. The problem is using libraries with different encodings.

Question: After the new FPC version is released, how Lazarus will work together "FPC Unicode"?

What do you mean with "FPC Unicode". The new compiler feature of strings with encoding and various changes to the RTL or the idea to release a FPC flavor with dotted unitnames and String=UnicodeString?

Mattias

Ok, no problem.

Ok, thank you.

Hm... Ok... thanks again.

Marcos Douglas

Of course!
But I'm using Lazarus and... LCL... so, I'm searching a better way to continue programming without concern about these problems -- at least decrease.

Just for clarify: I'm not judging the LCL or whatever. I'm only searching a solution to my problems. If I can help the Lazarus team, in a near future, would be great.

I mean "new compiler feature of strings with encoding and various changes to the RTL".
Lazarus will change something? Will use "FPC new feature of strings" or continues using UTF8?

Best regards,
Marcos Douglas

You are not alone.

Just for clarify: I'm not judging the LCL or whatever. I'm only searching a solution to my problems. If I can help the Lazarus team, in a near future, would be great.

You are welcome.
The LCL uses one encoding on all platforms, while still using native widgetsets and native file handles. That was already a big step to decrease the amount of conversions. And many packages using the LCL followed that approach. That further decreased conversions. The big missing piece in the puzzle is the RTL. Now FPC has extended 'string' and is adapting the RTL for unicode.
Of course this does not magically solve the general problem. All libraries must be adapted including the LCL.

One of the new FPC strings is UTF8String. So for compatibility the first step is to use that. That is not yet complete.

Mattias

I think use one encoding on all platforms is good, however Windows uses UTF-16. All string from/to Windows needs to be converted, right? Is this not a penalty for Windows platform?
What coding FPC's team chose to use on Windows in the next release of the compiler, UTF-16?

Of course this does not magically solve the general problem. All libraries must be adapted including the LCL.

Of course.

Lazarus uses only 'string', not UTF8String, UnicodeString, RawByteString, etc. This will change?

Marcos Douglas

[...] I think use one encoding on all platforms is good, however Windows uses UTF-16. All string from/to Windows needs to be converted, right? Is this not a penalty for Windows platform?

The Windows API is UTF-16, most text files and databases are not. You will always have some conversions. In most cases the conversion is hardly measurable, but there are cases where UTF-16 is better and cases where UTF-8 is better. And in some cases even UTF-32 is better. The LCL is graphical library, so string speed hardly matters.

What coding FPC's team chose to use on Windows in the next release of the compiler, UTF-16?

I'm not member of the FPC team, but afaik UTF-16 would be a big incompatibility, so the next release will still using system codepage string for Windows. Maybe eventually another flavor with UTF-16.

[...] Lazarus uses only 'string', not UTF8String, UnicodeString, RawByteString, etc. This will change?

Lazarus used only string (AnsiString), now it has to be more specific at some places to avoid unnecessary conversions by the compiler.

Mattias

Marcos Douglas wrote

I think use one encoding on all platforms is good, however Windows uses UTF-16. All string from/to Windows needs to be converted, right? Is this not a penalty for Windows platform?

I am just writing a file manager for Windows (hopefully can port it to Linux later) and I don't see any performance problems by using UTF8 in my program while the API is UTF16. Most (if not all) things that I do with files take much longer than the string conversion so it does not matter much.

IMO it makes very much sense to use UTF8 (which Linux uses anyway) throughout all programs. Having multiple string encodings within Lazarus/FPC makes it worse than necessary.

Maybe some day all programs and OS's use UTF8 so that we never need to think about encodings anymore (just dreaming...). At least, the more UTF8 is used the fewer conversions are needed.

I think that UTF8 is the English language of encoding ;-)

J?rgen Hestermann wrote

IMO it makes very much sense to use UTF8 (which Linux uses anyway)

Warning: Linux uses "array of byte" for filenames, not utf-8 AFAIK.

Martin

Yes, will always have conversions... the problem are conversions inside a only program (LCL RTL). ;-)

So, nothing will change for Lazarus users after the new FPC release.

Ok, thank you for all explanations.

Marcos Douglas

[...] So, nothing will change for Lazarus users after the new FPC release.

I do hope that something will change after a new FPC release. :)

Mattias

Ok. But how do you work, using SysToUTF8 / UTF8ToSys?

IMO it makes very much sense to use UTF8 (which Linux uses anyway) throughout all programs. Having multiple string encodings within Lazarus/FPC makes it worse than necessary.

It makes perfect sense to use only one encoding, whatever, in all code of a single program. ;-)

[...]

Marcos Douglas

Touch? :)

Marcos Douglas

I would like to understand: Why Java, .Net and others use UTF-16 as default encode and why Lazarus team chose UTF-8?

Thanks,
Marcos Douglas

Marcos Douglas wrote

I would like to understand: Why Java, .Net and others use UTF-16 as default encode and why Lazarus team chose UTF-8?

One reason is that Free Pascal did not support utf-16 at the time Unicode became urgent and later the FPC implementation of utf-16 strings was buggy. So there was no other choice than to use utf-8 in standard FPC 8 bit strings.

Martin

The impact of switching to UTF-8 is less when you care about backwards compatibility.

Michael.

Marcos Douglas wrote

I would like to understand: Why Java, .Net and others use UTF-16 as default encode and why Lazarus team chose UTF-8?

... and don't forget Windows API.
I believe the decision was made by people who didn't know the issue well enough, and some decision had to be pushed quickly. I am slowly learning the issues around Unicode. UTF-8 seems to be the best encoding for most purposes.
The only benefit of UTF-16 was supposed to be its fixed size character length, but finally it did not happen. All characters in the world did not fit into 16-bit space.
It means UTF-16 wastes space without any real benefits. Only UTF-32 would bring the fixed size character benefit.
What more, UTF-16 is confusing because it has variations. It all is well explained here:
http://www.utf8everywhere.org/

At my work we must switch to Unicode but the details of how to do it are still open. The code now works with both Delphi and FPC. There is a highly optimized DB engine where most data fits in a cache at run-time making it lightning fast. UTF-16 would almost double the space requirement and thus is out of question. The core parts must use UTF-8 anyway. One choice is to dump Delphi completely and use FPC+Lazarus for everything. Lets see...

Juha

Apart from that there's not much else you can do except contribute patches to help "unicode-ise" the FPC RTL...

The new AnsiStrings (with Encoding and automatic conversion) should be sufficient, Unicode is not required. In fact a move to a Unicode RTL would require that either Lazarus is converted, too, or that 2 RTL flavors (Ansi and Unicode) must be supported. Not a good idea, IMO.

DoDi

Now I understand. Thank you Martin and Michael .

Marcos Douglas

I will read, thanks.

Here too, more or less... I'm thinking to switch all own packages to UTF-8. But, in your codes, how do you works on Delphi -- or with Lazarus on Windows -- using your core parts? There are many calls from/to SysToUTF8 and/or UTF8ToSys from core to Windows?

Thanks,
Marcos Douglas

I do not quite agree. Moving from single byte encoding to utf-8 breaks many non-English applications which use character index. MSElang supports string8 (utf-8), string16 (utf-16) string32 (UCS4) and bytestring (any 8 bit encoding and/or binary data).

Martin

Read it with a grain of salt. ;-)
It does not consider all aspects of the matter.

Martin

Marcos Douglas wrote

Here too, more or less... I'm thinking to switch all own packages to UTF-8. But, in your codes, how do you works on Delphi -- or with Lazarus on Windows -- using your core parts? There are many calls from/to SysToUTF8 and/or UTF8ToSys from core to Windows?

If you need to call WinAPI, then you must convert obviously. In our case API calls are not needed by the core program. It is cross-platform code. However using a new Unicode-Delphi would cause many problems because all VCL functions and classes, including TStringList, expect UTF-16 string. When using UTF8String, the compiler converts between encodings all the time.
UTF-8 is needed in many places, thus we would need to duplicate much of VCL code for UTF-8. No good.
Using UTF-8 with FPC/Lazarus would simplify the task. LCL classes and functions work as expected etc.
I was even presented a possibility of doing a hybrid Ansi/UTF-8 system and a gradual data conversion plan.
If Lenght(s) = UTF8Lenght(s), then the string is an AnsiString, and so on...

If you call WinAPI a lot, then with UTF-8 you must convert encodings. But, if you are calling WinAPI a lot, then you are in trouble anyway.

As Michael Van Canneyt wrote, backwards compatibility with UTF-8 is good. For example all our lower-ascii data will work without conversions.
Also lots of code which is not designed for Unicode, will continue to work with UTF-8 but not with UTF-16. For example parsers for common markup languages (HTML, XML, BB-code) still magically work because all tags are in lower-ascii area.

Juha

Thanks for this tip. :)

Marcos Douglas

Martin Schreiber wrote

On Tuesday 17 December 2013 19:15:15 J?rgen Hestermann wrote: > IMO it makes very much sense to use UTF8 (which Linux uses anyway) Warning: Linux uses "array of byte" for filenames, not utf-8 AFAIK.

Of course, as long as it is only stored in the file system it is alway just an array of bytes. Encoding is irrelevant as long as you only identify files by the byte sequence. But when displaying such a file name on the screen then the display component mu?t have a clue about the encoding so that characters can be displayed. There must be a convention how to *interpret* that array of byte.

But it seems that actually such a convention does not realy exist when I read this correctly: http://unix.stackexchange.com/questions/2089/what-charset-encoding-is-used-for-filenames-and-paths-on-linux

I use the following:

---------------------------
var X,Path : UTF8String;
     FW     : Win32_Find_DataW;
H := FindFirstFileW(pwidechar(UTF8Decode(WinAPIPathName(Path))),FW);
...
X := UTF8Encode(UnicodeString(FW.cFileName));
---------------------------

where WinAPIPathName just prepends the "\\?\" string to the pathname to overcome the 255 char length limitation. Path is the UTF8 string for the file search and X holds the found file name(s) in UTF8 notation. When I later need an API-call I convert back:

---------------------------
... Windows.DeleteFileW(pwidechar(UTF8Decode(WinAPIPathName(AppendDir(Pfad,X))))) ---------------------------

Using UTF8String the compiler converts to UTF-16 automatically?

I thought that, e.g., override the RTL classes and functions:
type
  TStringList = class(Classes.TStringList)
    // using UTF-8
  end

If you call WinAPI a lot, then with UTF-8 you must convert encodings. But, if you are calling WinAPI a lot, then you are in trouble anyway.

I disagree if you code a program only to run on Windows. ;-)

I agree.

Marcos Douglas

Well, the same problem...
If there is no solution (for now), I prefer using SysToUTF8/ UTF8ToSys because is more simpler than use WideString API and conversion to UnicodeString, UTF8Decode, etc. Don't you think?

Marcos Douglas

Marcos Douglas wrote

Well, the same problem... If there is no solution (for now), I prefer using SysToUTF8/ UTF8ToSys because is more simpler than use WideString API and conversion to UnicodeString, UTF8Decode, etc. Don't you think?

It fails if any of the characters is outside the current codepage (e.g. Chinese or Taiwanese on my system). In that case the Wide API's do work.

Bart

Marcos Douglas wrote

Using UTF8String the compiler converts to UTF-16 automatically?

Yes, Delphi does that. Future FPC versions will do automatic conversion, too, but not only to UTF-16.

No, with FPC you don't need to do that. The existing StringList works ok.

With Delphi you would need to copy the whole class, name it TUtf8StringList, and replace "string" with "UTF8String". This new class must NOT inherit from Classes.TStringList.

Juha

Not _only_ to UTF-16? It will depend on the OS?

For now (2.6.2) works ok only for AnsiString... I'm talking about codify TStringList class to work with UTF-8 but no changes in string type arguments.

With Delphi you would need to copy the whole class, name it TUtf8StringList, and replace "string" with "UTF8String". This new class must NOT inherit from Classes.TStringList.

The same here... I think.

Marcos Douglas

Marcos Douglas wrote

Well, the same problem... If there is no solution (for now), I prefer using SysToUTF8/ UTF8ToSys because is more simpler than use WideString API and conversion to UnicodeString, UTF8Decode, etc. Don't you think?

As Bart already mentions, the ANSI (SYS) interface does *not* support Unicode. Also, you are not be able to access long paths (longer than 255 characters) when using ANSI API functions. Therefore the [W]ide (unicode) character API functions are a must.

Marcos Douglas wrote

Not _only_ to UTF-16? It will depend on the OS?

No, FPC string will know its encoding and the conversion is made to any encoding but only when needed.
Let's not go deeper into this subject here. The details of future FPC are still open and they are not yet documented.
When "Unicode" is mentioned, usually people start to argue about how it SHOULD be done.
You can search in fpc-pascal and fpc-dev histories for that.

For now (2.6.2) works ok only for AnsiString... I'm talking about codify TStringList class to work with UTF-8 but no changes in string type arguments.

Again no. TStringList in 2.6.2 works ok for UTF-8 encoded strings, too. The same is true for future FPC versions because they are not hard-coded for UTF-16 (as Delphi is).

No no no :)

Juha

Juha Manninen wrote

Yes, Delphi does that. Future FPC versions will do automatic conversion, too, but not only to UTF-16.

It's a long winding debate whether or not this is a good idea from a technical POW, but as Delphi does this, FPC seems to need to follow.

In fact there are decent positive aspects.

But it obviously is a negative aspect if TStringList and such functions are implemented using a fixed encoding scheme forcing conversions to and fro when e.g. using TStringList as an intermediate store. Here, a "generic" implementation (which Delphi does not provide) would be good. IMHO this is doable without loosing Delphi compatibility or performance.

-Michael

Juha Manninen wrote

When "Unicode" is mentioned, usually people start to argue about how it SHOULD be done.

:-) :-) :-)

Especially because the big boss "Delphi" does not provide a really good model to go for. Delphi String handling is done with UTF16 (using other encoding results in bad performance and other problems) in mind and with no respect to in spite of that there still are some soles that claim Unicode support is not a complicated thing :-(

-Michael

Juha Manninen wrote

What more, UTF-16 is confusing because it has variations. It all is well explained here: http://www.utf8everywhere.org/

my experience at: http://www.utf8bootcamp.org/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20131220/603f2f21/attachment.html>

So, these limitations exist in Lazarus too, right?

Marcos Douglas

The file functions with UTF8 in name use internally the W functions under Windows.

Mattias

Ok, you're right.

I didn't understand. If I have a TStringList instance, on Windows, I need to convert Text property to ANSI. But some components, e.g. TMemo, do these conversions automatically, but this is different.

But you talking about to make a new StringList... this is not the proposal. ;-)

Marcos Douglas

+1

That's I was talking about in previous mail, "using TStringList as an intermediate store".

Marcos Douglas

I didn't know, thanks.

Marcos Douglas

Marcos Douglas wrote

I didn't understand. If I have a TStringList instance, on Windows, I need to convert Text property to ANSI. But some components, e.g. TMemo, do these conversions automatically, but this is different.

TMemo is a GUI component. Then the string encoding matters and it must be converted to the native widgetset encoding. Still, the conversion is automatic. You don't need to care about it if you work with LCL components only and not with WinAPI directly.

TStringList is not a GUI component. It can be used for example in an embedded Linux program with no GUI.
It does not need to know the encoding (except for sorting maybe). With FPC no automatic conversions happen.

In Delphi things are different. The auto-conversion happens ALWAYS when assigning between eg. UTF-8 and UTF-16. It has nothing to do with WinAPI, or any other widgetset API.
Native string is UTF-16. If you have

  var MyUTF8Str: UTF8String;
  ...
  StringList.Add(MyUTF8Str);   <- triggers conversion
  MyUTF8Str := StringList[0];   <- triggers conversion again

The amazing thing is that such code works. Delphi does a good job in converting the strings.
It is also reasonably fast, but still not acceptable in a speed critical code. This was the problem in my employer's code base. We are thinking how to use UTF-8 for the core program without triggering many auto-conversions. One choice is to dump Delphi and use only FPC. Now the code still works with both.

Juha

P.S.
I am still wondering why you are so fond of WinAPI while you have a nice cross-platform API available.

I know, of course... :)

Then the string encoding matters and it must be converted to the native widgetset encoding. Still, the conversion is automatic. You don't need to care about it if you work with LCL components only and not with WinAPI directly.

Yes and so I wrote "TMemo, do these conversions automatically, but this is different".

TStringList is not a GUI component. It can be used for example in an embedded Linux program with no GUI.

Yes again. I use a lot TStrings as a transfer of information in many cases... no GUI envolved.

It does not need to know the encoding (except for sorting maybe). With FPC no automatic conversions happen.

And that is one of these problems because I need convert the Text property to the right encode.

That's it!
I think you talking about of new versions of Delphi, right? So I always read that "new Unicode implementation" in new versions of Delphi is wrong, broke things, etc. but you is writing other vision. These conversions, IMHO, could be automatic -- as Delphi does -- when I use the correct type of string, in that case UT8String. So, I can write my packages and opt to use only UTF8String or UTF16String in all arguments and the compiler convert for me. What is wrong in that approach?

If you do not want automatic conversions, use the RawByteString type. Delphi does not do conversions in that case, right? Thank you, I'm learning.

Marcos Douglas

P.S. I am still wondering why you are so fond of WinAPI while you have a nice cross-platform API available.

Fond? Of course not! I use WinAPI when I need or when I don't know another way to do the same using cross-plataform. I'm a "classic Delphi programmer". I still use Delphi (stoped in 7 version) today but all new projects I use Lazarus -- MSEgui a little. For example, I use a lot PostMessage, SendMessage, PeekMessage... Are these cross-plataform? If not, how can I do the same?

Marcos Douglas wrote

That's it! I think you talking about of new versions of Delphi, right? So I always read that "new Unicode implementation" in new versions of Delphi is wrong, broke things, etc. but you is writing other vision.

Yes, Delphi 2009+. Delphi 2009 is soon 5 years old, not really new any more. IMO it does a good job for such a fundamental change in string type. Only code that relies on "sizeof(char) = 1" does not work. It includes streaming strings, file I/O or I/O with some outside devices, using Length(Str) as a parameter for GetMem(), Move() etc. Most "clean" code works amazingly well, if you are ok with using UTF-16 everywhere.

Nothing wrong I guess. I hope it will be possible with FPC. Still, let's not speculate more, we already have such mail threads in fpc-dev list that continued for months.

If you do not want automatic conversions, use the RawByteString type. Delphi does not do conversions in that case, right? Thank you, I'm learning.

You can bypass the conversion sometimes by using RawByteString but it would be rather hackish. Remember, all VCL classes and string functions expect UTF-16. I don't want to try what happens if you pass them a UTF-8 encoded string using some hack.
The bottom line is: Use only UTF-16 with Delphi and it works very well.

For example, I use a lot PostMessage, SendMessage, PeekMessage... Are these cross-plataform? If not, how can I do the same?

LCL (and VCL) typically use events, like TNotifyEvent. They are basically just call-back functions.

Juha

I always said here -- FPC/Lazarus lists -- that FPC should never follow Delphi but you're making me change my mind about Unicode implementation.
Ok, no more speculations about how next FPC will work with Unicode.

Oh, not same. I use a lot Events -- no only Form or GUI components -- in my core codes but PostMessage is very different, eg., you call a PostMessage, show a Modal Form and the process will start after; the task code is not inside the instance of the Form and the Form knows nothing about the task.

Marcos Douglas

Juha Manninen wrote

The bottom line is: Use only UTF-16 with Delphi and it works very well.

I would not like Lazarus to do the same.
UTF16 is the worst of all possible unicode encodings.

anybody have a little time to help a 71 yr ole kid just startin' out?

Ok, true.
Some of the Windows message are ported to be cross-platform. I have used OnIdle handler and sometimes threads when I want the action to happen later.

Juha

J?rgen Hestermann wrote

> The bottom line is: Use only UTF-16 with Delphi and it works very well. I would not like Lazarus to do the same. UTF16 is the worst of all possible unicode encodings.

I believe LCL will continue to use UTF-8. Nobody knows yet how many changes are needed later with new FPC versions but no worries, that question is not acute now.

Juha

Bob Axtell wrote

> > anybody have a little time to help a 71 yr ole kid just startin' out?

What are you trying to interface to? If you don't need a graphical control I extended FPC's standard serial unit to work with Win-32, I think it's reliable although I usually use the Linux or Solaris variants. http://bugs.freepascal.org/view.php?id=18946

I am making a simple connection thru an RS232 LINK, and I have used Comport with Delphi6 very successfully in the past. The graphical control is needed to make the notebook app easily readable and usable by our installer, who has ASUS netbooks with WinXP installed.

Thanks for taking me under your wing.

--Bob

I use threads too, but I like make things as simple as possible and threads can be hard sometimes. Use PostMessage is very easy and simple.

Marcos Douglas

FPC 2.7.x can compile the windows unit in unicode (UTF16) mode. Most system and sysutils file related routines are already unicode (UTF16 with Rawbytestring overload).

Keeping UTF8 on Windows makes a majority platform seem only half supported. Not good either. Worse, it is Delphi incompatible.

So FPC 2.7.x can compile the windows unit in unicode (UTF16) mode. But how it will work with Lazarus that uses UTF-8? Lazarus will not to change to UTF-16 -- only for Windows -- then everything will stay the same to Windows programmers?

Thanks,
Marcos Douglas

Keeping UTF8 on Windows makes a majority platform seem only half supported. Not good either. Worse, it is Delphi incompatible.

You favor a special FPC and Lazarus for Windows, in addition to the UTF-8 version for all other platforms?

DoDi

IMHO the utf8 is not a done deal, and Delphi compatibility requires at least also UTF16 on other platforms.

QT is utf16, and so is Cocoa. Only GTK is utf8

So I would say UTF16, and maybe, if there is demand, some can get utf8 :-)

Not without conversions. UTF8 on Windows IMHO _NEVER_ was a good idea.

Lazarus will not to change to UTF-16 -- only for Windows -- then everything will stay the same to Windows programmers?

I think it is too early to say what will happen. One way or the other. Everybody is still searching, and the current 2.6.x based UTF8 support will need an overhaul anyway for 2.8.x.

I think 2.8.x will be a transition version anyway, and a definitive unicode solution will only in the major release after that.

However using a new Unicode-Delphi would cause many problems because all VCL functions and classes, including TStringList, expect UTF-16 string. When using UTF8String, the compiler converts between encodings all the time.

Then you can give your favorite string type a unique name, and set it to whatever is best in your favorite environment.

DoDi

Hans-Peter Diettrich wrote

Then you can give your favorite string type a unique name, and set it to whatever is best in your favorite environment.

The favorite string type in this case would be UTF8String. It already has a name. Please see what I was writing earlier.

Juha

Marco van de Voort wrote

So I would say UTF16, and maybe, if there is demand, some can get utf8 :-)

The question is:
Should FPC and LCL use a fixed encoding for all platforms or should the encoding be adapted for each WidgetSet/OS?

If it should be the same for all platforms then it should be UTF8 IMO. UTF16 is the most horrible decision (all bad things combined). UTF32 would at least have the advantage of fixed character size but pays this with *a lot* of memory consumption.
UTF8 has the lowest memory demand (in general) and a good backward compatibility.

On the other hand, adapting the string encoding for each Widgetset/OS would be a can of worms IMO.
A lot of additional knowledge about strings is put on the programmer because handling of strings has to be done differently depending on OS. That would be a hazadous decision and would only be of use if programs are exclusively written for one OS only.
But FPC/Lazarus is meant to be portable so this should not be done.

Ok, thanks for the explanation.

Marcos Douglas

Not necessarily. Supporting both on both platforms is a sane reason too.

One can't ditch utf16 because of Delphi compatibility. It will be hard to ditch utf8 because of old Lazarus compatibility.

But if I have to chose to kill one, it is utf8. It is the lesser used choice for unicode strings INSIDE APPLICATIONS. Yes, UTF8 is dominant in documents, but not in APIs.

If it should be the same for all platforms then it should be UTF8 IMO. UTF16 is the most horrible decision (all bad things combined).

For what? Most of the sentiments I hear are echoed discussions on the web that are mostly about document encodings, NOT application internal encodings.

However we

UTF32 would at least have the advantage of fixed character size but pays this with *a lot* of memory consumption.

(it is not fixed character, but fixed codepoint)

UTF8 has the lowest memory demand

Not according to 1 billion Chinese.

(in general) and a good backward compatibility.

Hardly. Only for western languages, and even there conversions often go wrong. That's why the whole BOM kludge became so important.

On the other hand, adapting the string encoding for each Widgetset/OS would be a can of worms IMO.

If you feel that way, I think Delphi compatibility should prevail. Old Lazarus code needs to be modified anyway.

Note that the language support for utf8 breaks down when you pass e.g. a "string" to rawbytestring on Windows. (because it is converted to the default 1-byte encoding, which is not utf8 in general).

As said, UTF8 on Windows is a crutch, and attempts to workaround that moves Lazarus in the direction of "portability to everything as long as it is unix" philosophies, a la Cygwin.

IMHO a bad direction. FPC has in general avoided having an outright preference and IMHO should continue to do so.

A lot of additional knowledge about strings is put on the programmer because handling of strings has to be done differently depending on OS.

It will anyway, even with utf8. Constructs that happen to work with Linux will fail on Windows. Because on Windows the default 1-byte encoding is not UTF8.

Moreover, I think people step over the Delphi compatibility card too easy. Way, way ,way to easy.

But FPC/Lazarus is meant to be portable so this should not be done.

FPC/Lazarus is supposed to be portable, not an emulated Unix on everything. Using other systems default encoding is emulation, and not portability.

In the meantime we have 2 Delphi compiler/RTL versions:
- Ansi (Win32)
- Unicode (UTF-16, multi target)
and 4 GUI versions
- VCL Win32
- CLX
- VCL.NET
- FireMonkey
summing up to 8 versions in theory, and 3 versions in practice.

So what does "Delphi compatible" mean *really*?

The FPC compiler supports multiple targets, and most probably can be managed to support both string types using the *same code base* (maintenance issue!). IMO this does *not* apply to the libraries (RTL and LCL) and existing applications, where Lazarus counts as the most important and prominent application. We can be happy to have one single LCL and IDE version, which is already incompatible due to the use of UTF-8 strings instead of Ansi. Multiple versions, for compatibility with the other Delphi combinations, are beyond *development capacities*.

This sheds a very different light on Delphi compatibility, meaning that a Unicode LCL and IDE can *not* be supported in parallel to the existing UTF-8 implementation. Dumping UTF-8 would discontinue support for the *entire* range of *existing* LCL applications, i.e. loose all the current Lazarus users :-(

So what should be the intended *audience* for a future Lazarus version?

IMO the biggest group are old fashioned Delphi (D7) users, which want their existing Ansi/VCL code base supported *without* complications and incompatibilities introduced by the newer Delphi versions. The subject of this thread clearly indicates that UTF-8 is *not* a solution for this group of users.

Another important user group is targeting mobiles, where time will tell whether FM will ever succeed, or shares the fate of Kylix or VCL.NET. IMO these should be happy already with fpGUI or mseGUI, no need to raise another competitor in this area.

But if I have to chose to kill one, it is utf8. It is the lesser used choice for unicode strings INSIDE APPLICATIONS. Yes, UTF8 is dominant in documents, but not in APIs.

That's my conclusion as well. But is that new audience worth to abandon the entire existing Lazarus audience?

DoDi

Marco van de Voort wrote

But if I have to chose to kill one, it is utf8. It is the lesser used choice for unicode strings INSIDE APPLICATIONS. Yes, UTF8 is dominant in documents, but not in APIs.

But in APIs it would not matter much to convert (in general the time for conversion is negligible compared to the time that is needed for the rest around the API call).

I have written a file manager for Windows that can log and store millions of files in memory. It uses the (UTF16) unicode API from Windows and converts the file names as UTF8 internally. There exists another file manager who uses UTF16 internally too which can also log millions of files. When logging the same source I can't see any difference in performance (even when logging multiple times so that everything is cached!) although I have to convert and the other one does not. But the memory footprints are very different.

> UTF16 is the most horrible decision (all bad things combined). For what? Most of the sentiments I hear are echoed discussions on the web that are mostly about document encodings, NOT application internal encodings.

IMO this decision is based on the assumption to choose one encoding for everything. So the same encoding is used *everywhere* as much as possible. Then UTF8 is the best solution.
Why use UTF16/32? They cannot be treated the same as ancient ANSI strings either. So what would be the reason behind it? Just wasting memory?

> UTF8 has the lowest memory demand Not according to 1 billion Chinese.

How many of the strings stored and processed on a chinese computer are in chinese language? A lot of the strings are still in english (HTML etc.). So for asian countries the real memory demand is a mix and is not so easy to determine. In most western countries UTF8 definitely uses less memory.

> On the other hand, adapting the string encoding for each > Widgetset/OS would be a can of worms IMO. If you feel that way, I think Delphi compatibility should prevail.

Why this?
Free Pascal/Lazarus should fledge and not repeat all the bad decissions of Borland/Embarcadero/..

Note that the language support for utf8 breaks down when you pass e.g. a "string" to rawbytestring on Windows. (because it is converted to the default 1-byte encoding, which is not utf8 in general).

I am not sure what you are talking about here.
For Windows I would use the unicode (UTF16) API interface exclusively and convert it to UTF8 internally. From then on, everything should be UTF8.

As said, UTF8 on Windows is a crutch, and attempts to workaround that moves Lazarus in the direction of "portability to everything as long as it is unix" philosophies, a la Cygwin.

For me the decision of what Unicode encoding should be used is primary OS independent. Just do the conversion once at the API interface level but then use internal what was decided to be "the best" (UTF8 IMO). Conversions seem to be unavoidable anyway. So it is just a decision where and when they take place. And the API level is a good place IMO.
And when other OS's use the same encoding it is even better but not the reason to chose one or the other.

> A lot of additional knowledge about strings is put on the programmer > because handling of strings has to be done differently depending on OS.

No!. That's just the aim: If *all* Free Pascal/Lazarus programmers can rely on having UTF8 in all cases then you only need to handle UTF8 strings. No IFDEFS to handle UTF16 on Windows and UTF8 on Linux. The same code just works on *all* platforms!

Constructs that happen to work with Linux will fail on Windows. Because on Windows the default 1-byte encoding is not UTF8.

The ANSI interface should not be used anymore. It is obsolete and only needed for ancient OS's like DOS. But programmers should not be encourraged to use it on modern platforms. Just use UTF8 *everywhere*. That should be the aim IMO.

The older delphi compilers are unsupported. We never supported anything but VCL 32/64, so this list seems artificially inflated to me.

So what does "Delphi compatible" mean *really*?

The same as it always has. VCL, and language level at a distance. The rest is irrelevant.

The FPC compiler supports multiple targets, and most probably can be managed to support both string types using the *same code base* (maintenance issue!).

Yes.

IMO this does *not* apply to the libraries (RTL and LCL)

RTL is less of a problem than one might think. The problem mostly only comes in at the classes level.

and existing applications, where Lazarus counts as the most important and prominent application.

Existing Lazarus applications are toast anyway, without changes.

We can be happy to have one single LCL and IDE version, which is already incompatible due to the use of UTF-8 strings instead of Ansi. Multiple versions, for compatibility with the other Delphi combinations, are beyond *development capacities*.

Then drop the old stuff, and simply go for full compatibility. Anything else will only cause the loss of all OSS Delphi projects (and even the commercial ones that support Lazarus).

And people like me that are torn between both systems.

This sheds a very different light on Delphi compatibility, meaning that a Unicode LCL and IDE can *not* be supported in parallel to the existing UTF-8 implementation.

There is no existing UTF8 implementation that can be continued as is anyway.

It was like that two years ago. But I see more and more people migrate to the unicode versions, and updating packages. The D7 base is eroding, and worse, many of its users are mostly hedging bets to keep their codebases running. Not to make new code. (and we need people that DO things)

It's like with turbo pascal in the (1.0.x) past. Yes, the numbers are huge, but all they say is they want something 100% compatible to effortless keep their codebases running. But when the times come to actually _invest_ in the code again, they pick something that is at least halfwhat modern. And all you are stuck with is oldtimers and l33t tinkerers.

That is the curse of supporting legacy targets, you can't do that forever without making yourself irrelevant.

Keep in mind that any Lazarus solution in production use based on 2.8.x is years away. The current activity levels in that group will be even less. Our decisions must be aimed not at the situation now, but good for at least 5 years.

Another important user group is targeting mobiles, where time will tell whether FM will ever succeed, or shares the fate of Kylix or VCL.NET.

Everywhere I see FM (Mobile plugin) buyers, I see existing Delphi users hoping for an easy conversion to mobile and a quick buck to tide them over the crisis. Not real go-getters that really go for mobile.

That makes me think this is not sustainable.

But Embarcadero is said to use it heavily internally, so they won't quickly kill it off, and I assume a certain kind of customers will adapt it.

But IMHO for us it is irrelevant

IMO these should be happy already with fpGUI or mseGUI, no need to raise another competitor in this area.

I don't really see any adaptation there. Those teams and offerings are again a magnitude smaller than Lazarus, and for most of those users switching from Embacadero to Lazarus is already the biggest step they are willing to make.

I myself hope for the two tracks way. It satisfies multiple demands, and the extra work is offset by less rewriting from current Delphi sources and less discussion.

But the prime point is that IMHO an utf8 Windows is insane, and it should be possible to port modern Delphi VCL apps at least to Windows. Preferably to all.

I started this thread. My problem isn't to use UTF-8 on Windows... my problem is use different encodings on the same code, ie, RTL LCL.

Use functions, always, to convert string between RTL and LCL and vice-versa IHMO is wrong because the final code is confusing. In a huge application you still need to think "here is UTF-8 or ANSI/UTF-16?"

Of course nobody will abandon the entire existing Lazarus audience. If the RTL will be UTF-16, UTF-32, whatever the Lazarus will continues -- I think -- working using UTF-8.

Marcos Douglas

Maybe. (less so for fine grained structures, e.g. think a virtual stringgrid with many nodes). But that is not the point, it makes that alien.

That's because the cost is hidden by slow harddisk movement. Directory scanning is about the slowest operation one can do on the same computer.

Well, that is a wrong (and arbitrary) assumption then. If you want to play make believe and reorganize the world top-down be my guess, but leave practical design matters in the current world out of it. We simply have to live in the world we live in, not the world that could have been

Document encodings are for end users, programming
interfaces are for programmers. Lazarus users are programmers, except for sourceencoding which IMHO can remain utf8 just happily (just like Delphi btw)

How many of the strings stored and processed on a chinese computer are in chinese language? A lot of the strings are still in english (HTML etc.).

Tags are. Text isn't. Depends on the webpage, but we are not talking about webpages. It was merely to point out that your size criterium is random and abitrary. It actually only matters for European languages.

Because most of users convert, and because having a clear agreed standard to work against is beneficial.

We have played this game many times before, and the only thing the Delphi nay sayers seem to agree upon is saying nay to Delphi compatibility. For what else there is going to be there are as many opinion as people. (and often more, e.g. with work and private hat on)

... running out of time, got a train to catch. Will continue later.

Sorry if I say something crazy, but what do you think to use UTF-16 on {mode delphi} and UTF-8 in {mode fpc}?

Marcos Douglas

<let's say irony>
Well (not directed at you, Marco), this thread has certainly exceeded my expectations. Thanks, very enjoyable!
Also nice that it's so similar to the same one last year (on the fpc list, I think).
</let's say irony>

<lame joke>
Anyway, I'll go massage a reindeer now...
</lame joke>

<serious>
Best wishes,
Reinier
</serious>

Marcos Douglas wrote

Use functions, always, to convert string between RTL and LCL and vice-versa IHMO is wrong because the final code is confusing. In a huge application you still need to think "here is UTF-8 or ANSI/UTF-16?"

That's true.
It's a pain to pay attention to this.
All units used should use the same string encoding IMO. But which?
I think that's the discussion in this thread.

If the RTL will be UTF-16, UTF-32, whatever the Lazarus will continues -- I think -- working using UTF-8.

But that would be a real pain.
In a program it should be possible to use strings
without the need to convert back and forth between encodings. So all strings from/to FPC and LCL routines should have the same encoding.

Someone agreed! :-)

All units used should use the same string encoding IMO. But which? I think that's the discussion in this thread.

Yes, this is the major problem... ;-)

> If the RTL will be UTF-16, UTF-32, whatever the Lazarus will continues -- > I think -- working using UTF-8. But that would be a real pain.

Would not.. IS a real pain today.

In a program it should be possible to use strings without the need to convert back and forth between encodings. So all strings from/to FPC and LCL routines should have the same encoding.

This will depend only on the FPC team...

When I created this thread I was looking for a way to only minimize this problem but...

Marcos Douglas

<curious>
So the answer for these problems is?
</curious>

Sorry but It is a real problem. No answer was presented so far and I have many system coded using Lazarus... this is important for me and, I think, for many Windows programmers.

Why these questions bother you?

Best wishes,
Marcos Douglas

Marco van de Voort wrote

> But in APIs it would not matter much to convert (in general the time for > conversion is negligible compared to the time that is needed for the

rest

> around the API call). Maybe. (less so for fine grained structures, e.g. think a virtual

stringgrid

with many nodes).

Well, I use VirtualStringGrid with millions of nodes in my program and it works like a charm (with UTF8) on Windows.

> I have written a file manager for Windows that can log and store

millions

When I log my whole hard disk multiple times the the logging comes totally from memory.
There is no hard disk there is no difference.

> IMO this decision is based on the assumption to choose one encoding

for everything.

Well, that is a wrong (and arbitrary) assumption then. If you want to

play

make believe and reorganize the world top-down be my guess, but leave practical design matters in the current world out of it. We simply

have to

live in the world we live in, not the world that could have been

It is the world of exact such thinking.
People are creating this.
It's not a law of nature.

Document encodings are for end users, programming interfaces are for programmers. Lazarus users are programmers, except for sourceencoding which IMHO can remain utf8 just happily (just like Delphi btw)

That's nonsense.
The *need* to convert between encodings should be restricted to just that: Only when it is needed.
We should never force people (neither programmers nor users nor anyone) to juggle encodings.
The aim should be to *reduce* conversions not to increase them.

Marcos Douglas wrote

So the answer for these problems is?

There is no answer. No, actually there are many answers but they all have problems. If you want your code to also compile with Delphi, then you have even more answers and problems. :)

Why these questions bother you?

The same questions and debate have continued for long but there are no good answers yet.
Unfortunately this is a transition period. Things will become smoother when time passes.

Right now, with FPC/Lazarus, you must use UTF-8 with the specialized UTF-8 functions, and convert when needed. Yes, I know this solution has issues.

Juha

"Marcos Douglas" wrote

Sorry if I say something crazy, but what do you think to use UTF-16 on {mode delphi} and UTF-8 in {mode fpc}?

That is already the case with mode delphiunicode. But the big problem are classes and their inheritance. Take TStringList for example. Let's assume it's declared with String=AnsiString and you override it in a unit with String=UnicodeString then you'll get problems with overloads/overrides, because UnicodeString AnsiString.

The mode concept is all good and well, but here it breaks down... :(

Regards,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20131224/45c72fe4/attachment-0001.html>

This mix would cause problems, of course.

Use functions, always, to convert string between RTL and LCL and vice-versa IHMO is wrong because the final code is confusing. In a huge application you still need to think "here is UTF-8 or ANSI/UTF-16?"

The simplest (feasable) solution IMO is the adaptation of (OS...) string types behind the scene, i.e. inside the RTL and widgetsets. Then you can have any common encoding in the application and library API, while encoding-dependent code is encapsulated in lower level functions receiving explicit (Unicode, UTF8String...) string types, so that the compiler can insert required conversions. Such explicit parameter types also were required for legacy code, where a specific encoding is assumed. I'm not sure how this conversion process can be automated or supported, perhaps removing/renaming the tradional UTF8... functions would help in spotting the procedures that require special attention.

The number of automatic conversions can be reduced in the next step, by e.g. adding overrides, or conditional code, for both string types one by one, as time permits.

DoDi

Whenever the encoding matters, most users and applications are best off with their regional Ansi encoding - all used characters are single bytes. UTF-16 extends the range of languages whose characters can be assumed to have a fixed size, i.e. all character sets in the BMP. Such fixed-size characters IMO are on the top of the wishlist of most users, so that none of them ever will be happy with UTF-8. Certainly UTF-8 was the best choice when Delphi (and FPC) did not have native UTF-16 strings, but when we have Unicode strings, now or soon, it should be dropped.

DoDi

<serious> Best wishes, Reinier </serious>

My best wishes to everybody, too :-)

DoDi

Hans-Peter Diettrich wrote

Whenever the encoding matters, most users and applications are best off with their regional Ansi encoding - all used characters are single bytes.

You forget that using ANSI API functions on Windows not only has the drawback that you cannot access all files (which have unicode characters in them) but also that there is the limit of 255 characters for the path length (while unicode API functions allow up to 32k characters). So you run into problems in 2 cases:

1.) if strings (file names) contain non-ANSI unicode characters 2.) if paths are longer than 255 characters

Do you realy advice people nowadays to restrict their programs so far by using ANSI API functions? I wouldn't. I was always wondering why so many programs fail with these 2 limitations on Windows after an alternative has been available for such a long time. Now you want to extent this time by yet another generation of programmers. That's not good. Hopefully not too many programmers follow this road...

UTF-16 extends the range of languages whose characters can be assumed to have a fixed size,

That's not true.
You still you cannot rely on having a number of bytes for characters in UTF16 either. Also, UTF8 would not have any BOM problem while UTF16 and UTF32 have. So UTF16 has all drawbacks of all encodings but no benefit (except that this awfull decision is used by Windows).

J?rgen Hestermann wrote

All units used should use the same string encoding IMO. But which?

UTF-8 of course! It's the newest Unicode encoding that overcomes all problems found in other encodings. It is also the only Unicode encoding that is backwards compatible with ASCII - hence the W3C and the rest of the Internet etc standardised on it. It is also future proof and can (again) be extended to full (4 byte range) or to using 5 or 6 byte code points [*1]. Performance wise, it is also NOT any slower than any of the other Unicode encodings.

Probably the only reason UTF-16 is still being used is because of Windows - which used to use UCS2, and moving to UTF-16 was easier at the time (and I don't think UTF-8 existed at that point).

[1] A couple years back they limited the range of UTF-8 so that it stays compatible for now with the limited range of UTF-16. But the UTF-8 encoding can actually go all the way to 6 bytes per code page, which is an absolute massive range.

Regards,
- Graeme -

Graeme Geldenhuys wrote

> But which? UTF-8 of course! It's the newest Unicode encoding that overcomes all problems found in other encodings.

This guy explains it very well.

https://www.youtube.com/watch?v=MijmeoH9LT4

Regards,
- Graeme -

J?rgen Hestermann wrote

So UTF16 has all drawbacks of all encodings but no benefit (except that this awful decision is used by Windows).

+1

Regards,
- Graeme -

For that purpose (file names) I vote for a dedicated string type, that matches the target platform requirements. Then the user has not to look at filenames on a per-character base.

Do you realy advice people nowadays to restrict their programs so far by using ANSI API functions?

How many users have to use API functions, which are bound to a single platform? And which of these do not understand how to handle strings of whatever encoding?

DoDi

J?rgen Hestermann wrote

So UTF16 has all drawbacks of all encodings but no benefit (except that this awfull decision is used by Windows).

This is not true. Everytime someone claims this nonsense I need to comment but I will not argue again. ;-)

Martin

As Juha said, there is no simple answer. If there had been a simple answer, it would have been implemented ages ago, of course.

Why these questions bother you?

This question as such does not bother me.

The endless repetitive discussions about U[1] with armchair theorists repeating the same arguments they used the previous discussion does.

Currently, people are working on getting U[1] support into FPC. I'd suggest helping them instead of blowing hot air in discussions. Even if you're one of those theorists that disapprove of whatever choices are made currently, the sooner it is clear that those choices will or will not work in practice, the better: if it turns out to be unworkable, an alternative solution can be tried.
Just talking about it endlessly does not help.

[1] Unicode, the standard, or any implementation of it, not necessarily the Delphi UTF16 flavour.

Hmm, you're right. Understood.

The mode concept is all good and well, but here it breaks down... :(

So, if the {mode} continue to be a way, I think it should be used on "platform level", not per "unit level".
Even if the programmer can to change this, he will change in all code. Thinking better, this is to be used on compiler level, not source level.

Regards,
Marcos Douglas

This is not possible. The precompiled files remain the same.

That was the original proposal from me. Add encoding to the target. (so i386-linux-utf8) and make a distro per target. Call them appropriately.

Encoding wise there are three options:

- ansi
- utf8
- utf16

but not all options are relevant for all targets. E.g. most *nix are utf8, so it wouldn't make sense to make an ansi port. Windows does not support utf8, so only ansi and utf16 would make sense.

Since that means typically two per target, it was suggested to combine this using dotted unit functionality.

Keeping it in the same distribution at least gives hope of keeping encoding agnostic units shared, but that required compiler extensions nobody started. (I personally don't see the benefit in this)

Even if the programmer can to change this, he will change in all code. Thinking better, this is to be used on compiler level, not source level.

Something like that, but not compiler, but unit directory on a per project basis. Or dotted unit prefix in the dotted alternative.

Yes. But the selection of UTF8, and the legacy concerns with that are for Lazarus, and lazarus alone.

Use functions, always, to convert string between RTL and LCL and vice-versa IHMO is wrong because the final code is confusing. In a huge application you still need to think "here is UTF-8 or ANSI/UTF-16?"

There are many scenarios up in the sky, and nothing is 100% certain, but it would at least be significantly better. It is already significantly better in trunk.

The only problem on Windows is that you must only pass a string with a very clear encoding to a RTL function.

so

 assignfile(f,s+s2+s3);

is dangerous if they are not all the same encoding. If there is any mismatch, it will be converted down to default encoding.

It is defined, but somewhat special.

There is no utf8 on Windows. One can try to mess with the defaultcodepage, but that will probably only force a different kind of problems.

On Windows there is only ansi or utf16, or keeping it manual.

The problems is as always, people talk about "own" things other than delphi, but that is only on a top down very high level. The practical implementation is something else, which leads to a stalemate.

Make sense.

Since that means typically two per target, it was suggested to combine this using dotted unit functionality.

I did not understand this... dotted unit functionality?

Keeping it in the same distribution at least gives hope of keeping encoding agnostic units shared, but that required compiler extensions nobody started. (I personally don't see the benefit in this)

... you mean 3 compiled units (ansi, utf8 and utf16) using dotted unit names functionality?

Maybe I did not understand:
Using only directories or per project we have the same problem that use {mode} directive, ie, TStringList could be compiled using utf-16 by default, the programmer inherit this class and compile your own directory or project using utf-8... something will break.

Marcos Douglas

When you say that is better in trunk is only on FPC context or there are improvements for Lazarus users too?

Yes but where is the difference between 2.6.2 and trunk, in that case?

You're right.
But if we imagine a perfect world that FPC and Lazarus use the same encode -- doesn't matter if is UTF-8 or UTF-16 -- everything would work. Do you agree?
So, if the encode chosen was UTF-8 for all, RTL only needs to decode strings -- on Windows -- before to call API functions. The same on Linux (whatever platforms that uses UTF-8) if the encode chosen was UTF-16.

My thinking is correct?

Marcos Douglas

Functions like mkdir/findfirst/assign etc are encoding safe. The only exception is the concatenation problem below.

1. UTF16 works fine
2. You can actually pass utf8 to functions, as long as you are careful with concatenations.

The point is (as shown by the above problem) is that the choice must align with what the OS offers. Because otherwise you are yet an island again.

E.g. the Windows unit (also in trunk) will only work in ansi or utf16.

So, if the encode chosen was UTF-8 for all, RTL only needs to decode strings -- on Windows -- before to call API functions.

The Windows unit is not wrapped, and the only unicode available on Windows is UTF16. And the windows target converts mixes of 1-byte strings (say ansi+utf8) to the default encoding (ansi).

One can attempt to fix that by messing with Windows encoding settings, but the effect of doing that for large applications is unknown.

Another possibility is using only
own unicode routines (linking in the tables into each binaries). But that again could lead to strange artefacts.

The same on Linux (whatever platforms that uses UTF-8) if the encode chosen was UTF-16.

Yes. I don't think that is a good default choice either. But at least it has some merits for modern Delphi compat.

My thinking is correct?

Oversimplified. The RTL will never abstract everything, and there is the issue of the default OS encoding.

In short, I don't think fighting the native encoding of an target is worth the shallow appeal of the "one encoding rules all" principle. That is mostly pushed by people that don't even use windows, and thus won't feel the pain.

"Marcos Douglas" wrote

> Since that means typically two per target, it was suggested to combine

this

> using dotted unit functionality. I did not understand this... dotted unit functionality?

Delphi XE 2 with the introduction of FireMonkey switched from normal unit names for the RTL to dotted ones (aka unit namespaces). E.g. SysUtils and Classes became System.SysUtils and System.Classes respectively, the Windows units moved into a Windows "namespace" (AFAIK) and Forms became VCL.Forms. Now the XE2 IDE and command line compiler also provide the possibility to specify multiple default namespaces (e.g. a VCL application would have System and VCL) to ensure backwards compatibility.

Now the idea is to have dotted units in FPC where String=UnicodeString and the legacy non-dotted ones where String=AnsiString. That only leaves out Delphi 2009 and XE compatibility (which uses non-dotted UnicodeString units), but that's a small price IMHO. Also there are a few further problems that need to be tackled with that approach.

Regards,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20131225/bce103f7/attachment.html>

Sorry Marco,

Marco van de Voort wrote

There is no utf8 on Windows. One can try to mess with the defaultcodepage, but that will probably only force a different kind of problems.

I cannot let you answer alone and make you appear as the only knowledgeable reference for this important subject,
it looks like defining default code page 65001 for Windows make it perfect fit to handle UTF-8
http://msdn.microsoft.com/en-us/library/dd317756%28VS.85%29.aspx

Jerome.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20131225/a53bc88e/attachment-0001.html>

"Marcos Douglas" wrote

> The only problem on Windows is that you must only pass a string with a

very

If in 2.6.2 your three strings contain text of different encodings then the resulting string might be garbage from the user's POV. In trunk the encoding is part of each string and if they differ then each strings will be converted to the default string encoding (defined by a global variable inside unit System) and thus the string might still be valid.

Regards,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20131225/14c161b0/attachment.html>

Jy V wrote

I cannot let you answer alone and make you appear as the only knowledgeable reference for this important subject, it looks like defining default code page 65001 for Windows make it perfect fit to handle UTF-8 http://msdn.microsoft.com/en-us/library/dd317756%28VS.85%29.aspx

Windows doesn't support using UTF-8 as the default system code page and never will. Michael Kaplan, from Microsoft, has talked about it a number of times in his blog. The original site is unfortunately offline, but it's available through the Internet Archive at https://web.archive.org/web/20120414160234/http://blogs.msdn.com/b/michkap/archive/2006/10/11/816996.aspx https://web.archive.org/web/20110108050100/http://blogs.msdn.com/b/michkap/archive/2006/07/04/656051.aspx

The short answer is that all of the selectable ANSI codepages have at most 2 bytes, and UTF-8 can have up to 4, which would require auditing/updating huge amounts of code.

"Jy V" wrote

Sorry Marco, On Wed, Dec 25, 2013 at 6:15 PM, Marco van de Voort

> > There is no utf8 on Windows. One can try to mess with the

defaultcodepage,

> but that will probably only force a different kind of problems. I cannot let you answer alone and make you appear as the only

knowledgeable reference for this important subject,

it looks like defining default code page 65001 for Windows make it

perfect fit to handle UTF-8

http://msdn.microsoft.com/en-us/library/dd317756%28VS.85%29.aspx

The Windows API *A functions use the system's (or more precisely user's) code page set through the selected locale to convert ANSI to Unicode. So even if you could set the system's codepage to UTF-8 (which you can not) you'd need the user to change his/her codepage to UTF-8 which is a definite no-go.

Regards,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20131225/99948616/attachment.html>

Simple explanation, I understood, thank you.

So the new Delphi "namespace" is virtual (eg: there is no VCL.Forms.pas file only a Forms.pas) or they have two options, two files?

If is "virtual" and could be changed in command line compiler, looks like an ideia that I had (posted on fpc-list) about "namespaces" to use two units with the same name in the same project. ;-)

I see.
Well, it seems that the way has already been decided and is in development.

Thanks again for the update, in a few words, about the implementations on the trunk.

Regards,
Marcos Douglas

Then the implementation is wrong. All string conversion is done via UTF (lossless), the result can be either UTF-8 (FPC) or UTF-16 (Delphi). The final conversion depends on the target, i.e. the declaration of AssignFile.

There is no utf8 on Windows.

Yep, that's why the Unicode (W) API should be used. No problem with UTF-8 strings there :-)

DoDi

If so, this flaw should be fixed immediately. Delphi uses lossless conversions, i.e. an up-cast to Unicode.

BTW the use of RawByteString variables or parameters *in Delphi* can result in stored strings of an encoding that doesn't match the declaration of the target variable. This in turn can confuse the compiler, when two such strings of the same declared (static) encoding, but of different actual (dynamic) encoding, are simply appended without further checks/conversions.

Such problems can be avoided by making RawByteString a compiler magic, that enforces a Unciode conversion whenever AnsiStrings of a different dynamic encoding have to be combined.

Furthermore the use of UTF-8 will allow for lossless conversions of AnsiStrings of any encoding, with the result still being an AnsiString. Here Delphi has the problem that a RawByteString result type requires a conversion of an intermediate Unicode string (UTF-16) into an AnsiString(CP_ACP), with possible losses. This is not required when FPC treats UTF-8 as a fully supported encoding, in addition to CP_ACP - it also were a strong argument for using UTF-8 for UnicodeString, *instead* of UTF-16. The related functions already exist in the FPC libraries, they only have to take precedence over CP_ACP (if different). Then additional UTF-8/16 conversions are required only on Windows, when calling external (API...) functions which expect/return WideStrings.

Conclusion:

FPC can treat RawByteString as *the one and only* string type of a variable dynamic encoding. Procedures accepting RawByteString arguments either retain the dynamic encoding of these strings, or convert parameters of different encoding into UTF-8. A conversion back to a different encoding may be required *only* when a RawByteString is assigned to a variable or parameter in another subroutine call.

There remains one problem with empty strings, whose declared encoding cannot be determined at runtime in the Delphi model, because empty strings are represented by Nil pointers. I can imagine two workarounds, to add an Encoding field to every string variable, or to make empty strings point to a string constant of their static encoding.

Alternatively typed AnsiStrings and RawByteString can be dropped, so that every AnsiString variable or parameter can have any dynamic encoding (equivalent to RawByteString), with the favorite encoding being UTF-8. This would allow to keep Lazarus and other existing code unmodified, all eventual string conversions can be inserted by the compiler, the obsolete UTF8... functions can be dropped.

DoDi

"Marcos Douglas" wrote

On Wed, Dec 25, 2013 at 7:41 PM, Sven Barth

> Am 25.12.2013 19:19 schrieb "Marcos Douglas" : >> > Since that means typically two per target, it was suggested to

combine

unit

> names for the RTL to dotted ones (aka unit namespaces). E.g. SysUtils

and

> Classes became System.SysUtils and System.Classes respectively, the

Windows

> units moved into a Windows "namespace" (AFAIK) and Forms became

VCL.Forms.

> Now the XE2 IDE and command line compiler also provide the possibility

to

Neither. They only have e.g. VCL.Forms.pas and using the option I mentioned (e.g. set to "VCL,System") the compiler will look for VCL.Forms.pas, System.Forms.pas and Forms.pas (dont know the implemented order though) if the using of a unit Forms is encountered.

Regards,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20131226/feda2993/attachment.html>

the resulting string might be garbage from the user's POV.

> In trunk the encoding is part of each string and if they differ then

each strings will be converted to the default string encoding (defined by a global variable inside unit System) and thus the string might still be valid.

If so, this flaw should be fixed immediately. Delphi uses lossless

conversions, i.e. an up-cast to Unicode.

No it does not. If the variables you concatenate are AnsiString and the variable or parameter you pass them to is AnsiString as well (AFAIK it even needs to be RawByteString) then the strings are converted to the system encoding before they are concatenated and passed. This is implemented Delphi compatible in FPC.

Such problems can be avoided by making RawByteString a compiler magic,

that enforces a Unciode conversion whenever AnsiStrings of a different dynamic encoding have to be combined.

RawByteString is already as magical as it gets and exactly is what's on the tin: a raw byte string. No automatic conversions ever. This is a type that is needed for implementing String handling in RTL so overloading it with another meaning will only result on problems.
If you want UTF-8 encoded strings then use UTF8String. Period.

Furthermore the use of UTF-8 will allow for lossless conversions of

AnsiStrings of any encoding, with the result still being an AnsiString. Here Delphi has the problem that a RawByteString result type requires a conversion of an intermediate Unicode string (UTF-16) into an AnsiString(CP_ACP), with possible losses. This is not required when FPC treats UTF-8 as a fully supported encoding, in addition to CP_ACP - it also were a strong argument for using UTF-8 for UnicodeString, *instead* of UTF-16. The related functions already exist in the FPC libraries, they only have to take precedence over CP_ACP (if different). Then additional UTF-8/16 conversions are required only on Windows, when calling external (API...) functions which expect/return WideStrings.

UnicodeString is *defined* as 2-Byte character reference counted string. There will be no change there.

Regards,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20131226/a403d439/attachment.html>

Marco van de Voort wrote

In short, I don't think fighting the native encoding of an target is worth the shallow appeal of the "one encoding rules all" principle. That is mostly pushed by people that don't even use windows, and thus won't feel the pain.

This is not true!
I am programming for Windows exlusively (currently) and still want UTF8 everywhere. UTF8 is the most useful encoding *and* Lazarus uses it *and* it is used in many other situations. And I don't want to think about encodings all the time. I want a single string type in my programs.

Therefore I now need to write my own Windows interface unit because FPC does not provide Unicode file API functions.
If the Windows unit of FPC migrates to unicode soon I still cannot use it because it would use the foolish UTF16 string type which I still need to convert to UTF8. What an incredible short-sighted decision.
The unique opportunity to establish a single Unicode string type encoding (UTF8) within the whole programming environment Free Pascal/Lazarus has been missed.

I don't like getting involved in this since I'm a minimal user of Unicode and even then I find it most convenient to explicitly translate to a uniform 16-bit internal representation.

So speaking as an armchair theorist and addressing the others, I suggest that looking at what the Perl6 implementors are trying to do might be worthwhile. One particular reason for this is that they're trying to get to grips with issues like how to categorise characters that (usually) appear in fixed combinations, for example the various styles of brackets.

Now obviously this is extremely remote, but at some point the compiler might have an interest in parsing non-ASCII text. Rather more pressing is making sure that highlighters etc. don't throw a fit as soon as they encounter something unexpected.

Please specify "AnsiString", of which encoding?

When I concat an AnsiString and an UTF8String and assign it to an OEMString

   o := a + u;

then I get these warnings in XE:

[DCC Warning] ConcTest.dpr(20): W1057 Implicit string cast from 'AnsiString' to 'string'
[DCC Warning] ConcTest.dpr(20): W1057 Implicit string cast from 'UTF8String' to 'string'
[DCC Warning] ConcTest.dpr(20): W1058 Implicit string cast with potential data loss from 'string' to 'OEMString'

I cannot see the system codepage used here.

What I want to point out are the string function overloads, where Delphi supplies only string (UTF-16) and RawByteString arguments, and AnsiString(CP_ACP) in unit AnsiStrings. FPC could add UTF8String overloads and use these when dealing with AnsiStrings of an encoding different from CP_ACP.

Please understand that the use of RawByteString in Delphi can lead to strings with wrong encoding. This type should not be available for declaring variables, only for parameters and function results. This restriction requires compiler magic.

UnicodeString is *defined* as 2-Byte character reference counted string. There will be no change there.

Sorry, I meant the generic string type.

DoDi

Jy V wrote

Sorry Marco,

No problem.

As the others already said, that is mostly a target for conversions. (you still need from/to utf8 conversion if you use utf8 documents, thus the conversion routines support utf8).

That is something else than Windows APIs (and stuff like MSXML, ADO etc) support utf8.

utf8 proponents often then say "then just set utf8 as default encoding", but some limited experiments from me created more problems than it solved. IOW it is always suggested as "the solution", but few to none people did anything substantial with it. I know the command shell can crash if you chcp 65001

It is mostly a suggestion to end debate and get their way.

I'm going to study the links posted in this thread (from the Microsoft guy) to get more info myself too.

It happened again. The word "Unicode" was mentioned and the result is an endless debate of how it should be done. Now > 100 messages and counting ...

I personally don't care much what the default encoding will be, but I wonder how easy it will be to use UTF-8 for my employer's code. The situation with FPC will be better than with Delphi because FPC does not convert automatically to default encoding ALWAYS. It only converts when the conversion is needed.
For example TStringList can be used for UTF8Strings and it does not trigger automatic conversion.
Isn't it so? Please correct me if I still got it wrong.

It means UTF-8 with FPC will be easier than UTF-8 with Delphi, even if UTF-16 was the default.

Juha

Hans-Peter Diettrich wrote

> is dangerous if they are not all the same encoding. If there is any > mismatch, it will be converted down to default encoding. Then the implementation is wrong.

Wrong according to you. Not wrong according to defined Microsoft applications.

This way of top-down thinking will turn FPC into a Java, where you are lugging along an own platform-within-an-platform everywhere.

IMHO this is not desirable.

> There is no utf8 on Windows. Yep, that's why the Unicode (W) API should be used. No problem with UTF-8 strings there :-)

If you totally drop Delphi compatibility you can do whatever you want. But IMHO that is more something for the Graeme's and Martin (MSEGUI's) of the world, not Lazarus.

OEMString

o := a + u; then I get these warnings in XE: [DCC Warning] ConcTest.dpr(20): W1057 Implicit string cast from

'AnsiString' to 'string'

[DCC Warning] ConcTest.dpr(20): W1057 Implicit string cast from

'UTF8String' to 'string'

[DCC Warning] ConcTest.dpr(20): W1058 Implicit string cast with potential

data loss from 'string' to 'OEMString'

I cannot see the system codepage used here.

Try to make o of type RawByteString. And maybe also use more than two strings.

What I want to point out are the string function overloads, where Delphi

supplies only string (UTF-16) and RawByteString arguments, and AnsiString(CP_ACP) in unit AnsiStrings. FPC could add UTF8String overloads and use these when dealing with AnsiStrings of an encoding different from CP_ACP.

That was already discussed some time ago between devs and was deemed not useable by Jonas. I'll try to find his mail with his explanation.

Regards,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20131226/7fd44e9a/attachment.html>

Am 26.12.2013 11:25 schrieb "Mark Morgan Lloyd" <

: Now obviously this is extremely remote, but at some point the compiler

might have an interest in parsing non-ASCII text. Rather more pressing is making sure that highlighters etc. don't throw a fit as soon as they encounter something unexpected.

The compiler can already parse UTF-8 files with strings and comments being real UTF-8 texts which can be seen as such inside an editor. AFAIK it can even handle UTF-16 files (though I'm not entirely sure about this). What it however does definitely not allow (and none of the current devs wants to work on this) is support for non-Unicode identifiers as newer Delphi versions support.

Regards,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20131226/8c07c327/attachment.html>

Ok, I didn't remember the situation correctly. When searching for Jonas' mail I mentioned below I also found this which I was referring to:

=== quote of Jonas begin ===

var
   mypath: utf8string;
   sr: tsearchrec;
begin
   { assign some utf-8 string to mypath }
   if findfirst(mypath+allfilesmask,faAnyFile,sr)=0 then
     begin
       ...
     end;
end

If the DefaultSystemCodePage is something different from UTF-8, the result of "mypath+allfilesmask" will be downgraded to DefaultSystemCodePage because the string constant "allfilesmask" is encoded using that code page. This is due to rule that "concatenating ansistrings with different encodings results in an ansistring with the encoding of the destination ansistring" is followed, and the destination ansistring is a rawbytestring here (the first argument of findfirst), in which case the ansi encoding is used.

=== quote of Jonas end ===

=== quote of Jonas begin ===

Adding explicitly named UTF-8 versions of routines with constant or value rawbytestring arguments (FindFirstUTF8 etc) with UTF8String arguments and that internally simply call through to the rawbytestring versions could perhaps be useful. Interestingly, Lazarus users probably won't suffer from this particular problem as they already use such routines from the LCL, and those routines can simply be adapted by simply removing all the UTF8ToSys calls (they will keep working in their current state though, they simply keep suffering from the same data loss issues they had before).

=== quote of Jonas end ===

Please note that Jonas states that different named overloads would be needed. Equally named UTF8String overloads won't necessarily work correctly.

Regards,
Sven

Ok... but if FPC, on Windows, will be UTF-16 and Lazarus continues using UTF-8 what is the difference?
This approach is not like Delphi. It has the RTL and VCL using the same encode... FPC RTL and LCL will continue "fighting"! :(

Marcos Douglas

As I already statet: RawByteString is not for application use!

Delphi has no problem with this code, because all strings are upgraded to UnicodeString.

If the DefaultSystemCodePage is something different from UTF-8, the result of "mypath+allfilesmask" will be downgraded to DefaultSystemCodePage because the string constant "allfilesmask" is encoded using that code page.

Delphi has no rule of "downgrading".

When mypath+allfilesmask is assigned to a variable, the result has the correct encoding, not necessarily CP_ACP.

Again: RawByteString is a mess, should be used with care.

The first argument of FindFirst (file mask) certainly *can not* be a RawByteString.

I see no argument for or against UTF-8 overloads here.

Please note that Jonas states that different named overloads would be needed. Equally named UTF8String overloads won't necessarily work correctly.

You see the need for making RawByteString a compiler magic? :-] It should be used only as the last resort, when no other string type matches a given string encoding.

As for FindFirst, a choice of the mask string exists only on Windows, depending on the use of the A or W API. Other targets have an dedicated encoding for filenames, that should be used in all file and directory functions. Even on Windows only the W API should be used nowadays; the A API (as used in older Delphi versions) was only for support of legacy Win9x systems, where not all W subroutine versions were available.

DoDi

It happened again. The word "Unicode" was mentioned and the result is an endless debate of how it should be done. Now > 100 messages and counting ...

Now that we are in pre-release of strings with Encoding, the debate enters a very new round.

That's the old state, where strings have no stored Encoding. As soon as AnsiStrings have an encoding, the default encoding becomes important for the reduction of automatic conversions. When the RTL is converted to UTF-16, you'll have to accept either this new default encoding, or any number of automatic conversions between Ansi and UnicodeStrings.

It means UTF-8 with FPC will be easier than UTF-8 with Delphi, even if UTF-16 was the default.

Delphi suffers from the use of CP_ACP, which was the only supported encoding before, and still is the only explicitly supported encoding when the AnsiString unit is used. In Lazarus we had the same "only one encoding" philosophy, except that here the default string type is UTF-8. With the encoded AnsiStrings the problem of other encodings and automatic conversion arises. Delphi solved most problems by changing "string" to UTF-16, so that only the forced used of AnsiString will ever result in automatic conversions due to different string encodings.

In FPC/Lazarus the situation is somewhat different, because now the default string type could be UTF-8, UTF-16 or even CP_ACP, with a number of users voting for each of them. Technically the simplest solution would be to keep the de-facto standard UTF-8, as assumed by Lazarus. But when "string" becomes UTF-16, as in recent Delphi versions, Lazarus and the LCL deserves heavy refactoring. That's the top discussion topic right now.

DoDi

Wrong, or better *broken*, with regards to expected results.

Not wrong according to defined Microsoft applications.

Where do you see Microsoft applications using Ansi strings, nowadays?

This way of top-down thinking will turn FPC into a Java, where you are lugging along an own platform-within-an-platform everywhere.

That's what FPC and Lazarus do already: they assume an UTF-8 environment, till now. That's okay for all targets execpt Windows, where a UTF-8/16 conversion is required on the app-WinAPI boundary.

DoDi

You certainly mean *non-ASCII* identifiers?

DoDi

Juha Manninen wrote

It happened again. The word "Unicode" was mentioned and the result is an endless debate of how it should be done. Now > 100 messages and counting ...

This is because still nothing definitive is chosen, after 4+ years of discussion (this started in April 2009, when the first D2009 details were leaked)

It was a situation I hoped to avoid by going for two encodings per target directly. Or at least try.

Well, currently, Lazarus has no other choice, since the unicode FPC is not ready. (only up to classes level).

This alone means that the current uncertainty will persist at least several years.

This approach is not like Delphi. It has the RTL and VCL using the same encode... FPC RTL and LCL will continue "fighting"! :(

I always considered the UTF8 choice of Lazarus a temporary solution till FPC caught up with Delphi.

The current situation really worries me, since at work I invested in FPC/Lazarus in the assumption that compatibility would increase, not decrease.

Hans-Peter Diettrich wrote

But when "string" becomes UTF-16, as in recent Delphi versions, Lazarus and the LCL deserves heavy refactoring. That's the top discussion topic right now.

Personally I think FPC and Lazarus should get rid of "string" altogether! It should be a user definable type that can be defined per project.

eg:
Projects could do the following

  type
    {$IFDEF WINDOWS}
    UnicodeString = UTF16String
    {$ENDIF}
    {$IFDEF UNIX}
    UnicodeString = UTF8String
    {$ENDIF}
   String = UnicodeString
   // or for backwards compatibility with old projects:
   //   String = AnsiString

or they could simply say they prefer to work with a specific encoding, so use UTF16String or UTF8String directly. Thus no alias type needed.

Also the very broken logic of UnicodeString = UTF16String should disappear. Unicode Standard UTF-16!!! That is just some sh*t Microsoft came up with and Delphi followed suite! It's just WRONG. The Unicode Standard exists of multiple encodings, not just one.

But that's just my 2c worth - and arguments like these have been raised before.

Regards,
- Graeme -

Please read 4 years of discussion backlog. It is not all language, there is something called "libraries", and they are generally installed precompiled.

Marco van de Voort wrote

The current situation really worries me, since at work I invested in FPC/Lazarus in the assumption that compatibility would increase, not decrease.

I think that's the root cause of the discussion:

Some want to make FPC/Lazarus into a (possibly exact) clone of Delphi (which means to follow every sh*t that is and will be put into this product) and others (like me) hope for a more Pascal like programming environment which at least avoids future (maybe even removes existing) obscurity crept into Pascal with Borland/Embarcadero.

The ease of use was the reason for success of Turbo-Pascal but meanwhile this goal has been put aside and it becomes a more C-like environment (with lots of ugly hacks..).

On the other hand I understand the demand for compatibility if there is a lot of (Delphi) code that needs to be reused and cannot be changed easily.

Marco van de Voort wrote

Please read 4 years of discussion backlog. It is not all language, there is something called "libraries", and they are generally installed precompiled.

I don't know of a single ISV that ships precompiled *.ppu files for FPC or Lazarus. They all include source code (yes even most Delphi ISV's do this now) that can be compiled with a project.

Regards,
- Graeme -

Yeah, and I wonder what is the plans for the Free Pascal project, now that Embarcadero is changing the compiler and language - specifically for mobile platforms. EMBT are introducing even more sh*t - as you put it. ;-) Enhancements for Desktop or Web development are getting very little attention these days by EMBT.

Regards,
- Graeme -

raudus, see e.g.
http://forum.lazarus.freepascal.org/index.php/topic,22059.0.html

being real UTF-8 texts which can be seen as such inside an editor. AFAIK it can even handle UTF-16 files (though I'm not entirely sure about this). What it however does definitely not allow (and none of the current devs wants to work on this) is support for non-Unicode identifiers as newer Delphi versions support.

You certainly mean *non-ASCII* identifiers?

Of course *blush* thank you for correction.

Regards,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20131227/0fc7ccf0/attachment.html>

What about "fpc" ? Binary downloads top source downloads 20-30 to 1.

I always think it is amusing when you talk about compatability, proponents always are described as brainless drones that can't think for them selves.

Please come back after you fixed 700 bugs.

:(((

I think Lazarus team did not think the same.

The current situation really worries me, since at work I invested in FPC/Lazarus in the assumption that compatibility would increase, not decrease.

If you are worried, imagine all of us! ;-)

Marcos Douglas

+1

Marcos Douglas

Ok, you're right... so compile these libraries again! This would not be the simplest way?

Marcos Douglas

+1
Your comments are smart.
I understand but IMHO FPC (and Lazarus) should make your own way. I know there are many Delphi codes cannot be easily rewritten. For this the FPC has the {mode delphi} but the {mode FPC}, IMO, should be different. Maybe a new way completly different than Delphi to work with strings, maybe the programmer could change the string type for all codes (rtl, packages, etc)...

If we continue to follow Delphi, means that we are always one step behind.

Marcos Douglas

Hmm...
Maybe Martin Schreiber saw a dark future and has already taken the initiative to keep the true living legacy. :-)

Marcos Douglas

Marcos Douglas wrote

If we continue to follow Delphi, means that we are always one step behind.

If we stop following delphi, we are multiple steps behind.

Most of the antis only agree on being anti, they only have simplistic topdown proposals and talk about an elusive "own way".

Fact is that the extensions of FPC are used much less than the Delphi compatibility aspect.

Marco van de Voort wrote

Fact is that the extensions of FPC are used much less than the Delphi compatibility aspect.

And would you mind sharing how you came to that conclusion? Can you share the data on that research?

From my personal experience of using FPC since 2004-2005, there is NO

need to have support for two compilers (Delphi & FPC) in a single project. Free Pascal is more than capable enough to stand on its own feet. Speaking as someone that has personally ported large Delphi and Kylix project not only to Free Pascal and Lazarus's LCL, but also to fpGUI - a completely VCL incompatible UI toolkit. I have done this for multiple projects, frameworks and GUI widgets. My conclusion after all this... A conversion is NOT THAT HARD, and it's a great time to review old code too. So it has a triple positive. Moving to a real cross-platform compiler, a real cross-platform toolkit (be that LCL or fpGUI) and being able to review and improve old code and designs (your second attempt and software is ALWAYS better that your first).

Many projects have moved away from Delphi in the last few years - mostly to other languages like C# or Java. That requires a total rewrite - which is infinitely more work than moving to Free Pascal.

Regards,
- Graeme -

Just look on e.g. the forums. All people are asking about Delphi packages.

>From my personal experience of using FPC since 2004-2005, there is NO need to have support for two compilers (Delphi & FPC) in a single project

Yes, but we all know you are a special case :-)

Point is if you make conversion harder PEOPLE WILL NOT EVEN TRY!

I think Lazarus team did not think the same.

Lazarus team has not thought it much. The question is not acute yet and we already have a working Unicode system in LCL. Thinking of it now when -------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20131228/9f5e55fb/attachment.html>

Marco van de Voort wrote

Just look on e.g. the forums. All people are asking about Delphi packages.

And once those Delphi packages are ported to Free Pascal, nobody needs Delphi any more. ;-)

Point is if you make conversion harder PEOPLE WILL NOT EVEN TRY!

Converting to Free Pascal and Lazarus will *always* be easier that rewriting everything in C# or Java - no matter how many incompatibilities Free Pascal might have with Delphi. The language still stays a lot more similar than the alternative. Yet, looking at the current employment market, it seems most companies opted to rewrite there Delphi projects in C# and Java - so they took the even harder route! Why? Probably due to more innovation happing in those other languages.

Regards,
- Graeme -

Lazarus team has not thought it much. The question is not acute yet and we already have a working Unicode system in LCL. Thinking of it now when FPC behavior is not decided would be waste of time.
My comment about UTF-8 was based on poor knowledge and maybe wishfull thinking.
Some other Lazarus developers have more knowledge of the issue but they are wise enough not to enter this discussion.

Let's wait till FPC developers get their work done and then let's discuss about LCL.
If you need UTF-8 solution right now, it is possible with LCL. It is also good to read the old mail threads because the same things get repeated again and again.

Juha

P.S.
Learning to type with iPad. Sent an unfinished text. -------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20131228/ee5ef8bd/attachment.html>

On Fri, Dec 27, 2013 at 06:39:38PM -0200, Marcos Douglas wrote: > If we continue to follow Delphi, means that we are always one step behind. If we stop following delphi, we are multiple steps behind.

FPC/Lazarus always was in front of Delphi. It had Unicode support (UTF-8) since long, supports multiple widgetsets, platforms and machines. Even the help system and is better and more user friendly, as well as are the editing helpers.

The compatibility problems are selfmade, IMO. Compatibility with all versions of a continuously moving target is near impossible, at least not feasable with the available manpower.

Now that Delphi introduced something really useful (encoded strings and automatic conversion), the new Unicode support should be integrated into FPC and Lazarus. When this works for the AnsiString version (UTF-8), somewhat compatible with D7, a UnicodeString (UTF-16) version can be considered, compatible with some newer Delphi version.

DoDi

Graeme Geldenhuys wrote

> Just look on e.g. the forums. All people are asking about Delphi packages. And once those Delphi packages are ported to Free Pascal, nobody needs Delphi any more. ;-)

Somehow there is still more Delphi use than Lazarus, so I'll bounce back the "statistics" request.

Not really, it is simply vendors pushing it, and bundling it with their products, giving the SDKs preferential treatment. Therefore I don't really think it is sane to compare FPC (or even Delphi) to C# and Java.

Worse, one of the motivators, webframeworks often need support serverside, and getting into the ISP's portfolios is hard, specially as native language.

But more importantly, however which way you turn it, there are still way more new users coming from Delphi than from other sources (and then I'm already generous, since those other sources also include other pascals).

And I see the numbers of _knowledgable_ users from "old" Delphi decreasing, and the more able people also working with /new/ Delphi (and e.g. testing Lazarus to see if they can get a subset running on some other target).

This trend will only increase, and follows the same pattern as the TP to Delphi migration years ago. Still a definitive switch is still some time off, since most OSS projects and vendors still support D7. (support before that is getting scarce).

But that is now. Decisions for FPC/Lazarus NG will only come to fruition in 2-3 years. I think the D7 installed base will erode, but only very slowly. But the /activities/ of that installed base, and their investments in new (D7 level) code will erode quicker. Again this prediction is based on the same pattern as with TP.

It did not. It half a solution, like the wellknown TNT components.

Even the help system and is better and more user friendly, as well as are the editing helpers.

There is some potential there, but the content is still subpar to Delphi, specially the Lazarus/LCL part. Way subpar.

The compatibility problems are selfmade, IMO. Compatibility with all versions of a continuously moving target is near impossible, at least not feasable with the available manpower.

I don't see , it is hard. Yes, it will be at a distance. But I don't see "impossible". Also major change is fairly rare.

But the unicode change has been a done deal for 6 versions. This is is not about the bleeding edge, this is about planning steps that Embarcadero brought to production nearly 5 years ago, and which affect many levels of the code (more so than later additions, with the dotted change being debatable)

A utf8 ansistring version will be per definition not compatible.

Touch?.

> Point is if you make conversion harder PEOPLE WILL NOT EVEN TRY! Converting to Free Pascal and Lazarus will *always* be easier that rewriting everything in C# or Java - no matter how many

I totally agree.

"...most companies opted to rewrite there Delphi projects in C# and Java..."

I agree. I see this here in Brazil too.
So, if the companies prefer to rewrite everything to another language, this is another prove that people do not want compatibility with Delphi (so much).

Regards,
Marcos Douglas

Marco van de Voort wrote

Somehow there is still more Delphi use than Lazarus, so I'll bounce back the "statistics" request.

Umm, I never quoted any stats.

Free Pascal and Lazarus projects are like Linux - there is NO reliable way of tracking usage. So at best, any usage claims or stats are just a best guess, nothing more.

Regards,
- Graeme -

I'am pretty sure this is not the case. It is a story of ?No one ever got fired for buying IBM.?

And you think they would switch instead to some marginal OSS language which is compatible to nothing and nobody knows? C# and Java are used because they provide a huge user base (user in the sense of programmers knowing it) and being developed by huge companies so people expect their code base has a future.

My impression is that most people asking Delphi stuff are up to either:
- old Delphi versions (D7);
- people really new to Delphi and programming in general.

I tend to agree with Graeme on this one.

-Fl?vio

I understand. But if the major companies prefer to use C# or Java instead Delphi well, they not care about Delphi compatibilities. If they care, why they would be leaving Delphi?

Regards,
Marcos Douglas

If they leave Delphi compatibility, they normally don't go for a marginal oss compiler.

The question is:
Why did they use Delphi before at all?

If the reason was that Delphi was a very common and widespread programming environment then it is a understandable behaviour to move to the next main stream environment as soon as budget and time allows.
Such people would never care about FPC/Lazarus (even when it was fully Delphi "compatible"). They would never think about using it.
So making FPC/Lazarus "compatible" would not hold any user of this group.

If the reason was that they like Pascal as an easy to learn and mantain language then they will invest into migration even if not all parts are the identical to Delphi.
Just the opposite:
They may like that not all misconcepts are repeated in FPC/Lazarus and they may like that it is open source.

So you're saying that FPC cannot survive without Delphi?

Marcos Douglas

+1

Marcos Douglas

The world is not only 1 and 0. FPC lives (and living means getting usefull code!) from being delphi compatible but filling the niches delphi leaves open. Everything else is "by-catch".

GPC proved your argumentation wrong. GPC took the "clean way" of extended pascal (you always complain about fpc's dyn. arrays. Just use GPC, it has the clean solution) Unfortuntaly GPC development stopped for years due to missing contributors. The people keeping FPC alive are those interested in Delphi compatibility.

Marcos Douglas wrote

> If they leave Delphi compatibility, they normally don't go for a > marginal oss compiler. So you're saying that FPC cannot survive without Delphi?

Define survive. But I'am saying indeed that FPC's usage would drop significantly if Delphi wouldn't be around anymore. A few years it might increase because people would use FPC to rescue old sources but after that FPC's usage would probably decay significantly.

Right.
I didn't understand one thing: If I'm a Delphi XE2 programmer (suppose), why I will need to keep FPC compatible with Delphi? If I'm a Delphi programmer I will use... Delphi.

Marcos Douglas

To remain alive or in existence.

But I'am saying indeed that FPC's usage would drop significantly if Delphi wouldn't be around anymore. A few years it might increase because people would use FPC to rescue old sources but after that FPC's usage would probably decay significantly.

Well, this is very frustrating... and even more because you, the FPC main developer, wrote. :-(

Regards,
Marcos Douglas

Because you (the XE2 programmer) might look at Delphi XE5's NewGen compiler (the LLVM based one) and think: "What the f*** are they doing with the language?! O.o"
Or you might want to be compatible to more platforms than provided by Delphi XE2 or you might want to keep using a VCL compatible GUI library on other platforms whereby in Delphi you need to use FireMonkey here.

Regards,
Sven

Florian Kl?mpfl wrote

The world is not only 1 and 0. FPC lives (and living means getting usefull code!) from being delphi compatible but filling the niches delphi leaves open.

What niches?
It seems you want a possibly exact clone so where are niches? And what is the reason behind this?
Just use Delphi then.

GPC proved your argumentation wrong. GPC took the "clean way" of extended pascal (you always complain about fpc's dyn. arrays. Just use GPC, it has the clean solution)

I don't know GPC so I can't comment on this.
But I never complained about dynamic arrays (I use them as often as possible) I complained about their syntax (automatic dereferencing on indexing).

Unfortuntaly GPC development stopped for years due to missing contributors.

And the reason is that is was not enough Delphi compatible?

The people keeping FPC alive are those interested in Delphi compatibility.

That seems to be the bottom line which makes any discussion useless. Don't write any opinions anymore they will be ignored anyway. All those who don't like the current developement should role their own. Let's split Pascal into hundreds of dialects instead of striving for an universal one. At least an honest statement.

If you can/want affort it; if it supports the target you need etc. ... niches ...

It seems you want a possibly exact clone so where are niches?

Between clone and good compatibility is a *slight* difference.

And what is the reason behind this?

The niches.

Just use Delphi then.

On linux? VCL/IDE on Mac OS X?

Have a look at extended pascal then.

What else? The main difference between GPC and FPC is/was that GPC aimed at iso/extended pascal while FPC had a delphi mode.

Yes.

Don't write any opinions anymore they will be ignored anyway.

Yes.

All those who don't like the current developement should role their own.

There is a slight but important different: discussions are useless, writing specifications and code not.

Let's split Pascal into hundreds of dialects instead of striving for an universal one.

Well, at least some people try to prevent this by taking delphi as the universal one.

My opinion as well, and its already hard at this moment as generics and string/utf8 are the first problems people will encounter while moving XE* code to FPC.

Oops, I misquoted.
Graeme actually replied to that opinion with this: "Converting to Free Pascal and Lazarus will *always* be easier that rewriting everything in C# or Java - no matter how many incompatibilities Free Pascal might have with Delphi. The language still stays a lot more similar than the alternative. Yet, looking at the current employment market, it seems most companies opted to rewrite there Delphi projects in C# and Java - so they took the even harder route! Why? Probably due to more innovation happing in those other languages."

IOW I think that for some years already, innovation would be a much better selling point for Free Pascal rather than Delphi compatibility.

-Fl?vio

Ok, understood and just for clarify I'm not a XE2 programmer... I said "suppose".

Regards,
Marcos Douglas

I also appreciate Florians always clear statements and made my decisions accordingly. :-)

Martin

Well, in this case twenty years teached me a lesson so I can make a clear statement. When I started FPC I had a lot of ideas how I could do things differently and how it could help my chess program initially written in TP. However, nobody shared those ideas, well, they had their own.

This repeated over years: generics: we tried a radically different approach, with a syntax being verbose and showing that they are something really special and not just another type, basically nobody liked the approach. So nobody worked on fixing the problems with them till delphi got them as well and people wanted a delphi compatible syntax. Now people complain that the compiler is slow because it is feature loaded, true. Having a delphi compatible only compiler for x86 only would result in a much faster compiler.

For strings, I proposed, the first time ~2000 I think, to introduce basically an opaque string type which can store any encoding, people complained, this will be too slow (true), incompatible with old code (true), incompatible with delphi (true). So it never happended. This list can be continued endlessly. Lesson learned.

"Marcos Douglas" wrote

On Sat, Dec 28, 2013 at 1:31 PM, Sven Barth

fully

for

compiler

> (the LLVM based one) and think: "What the f*** are they doing with the > language?! O.o" > Or you might want to be compatible to more platforms than provided by

Delphi

> XE2 or you might want to keep using a VCL compatible GUI library on

other

> platforms whereby in Delphi you need to use FireMonkey here. Ok, understood and just for clarify I'm not a XE2 programmer... I said "suppose".

Of course. I've just gone with your example where you supposed that you're one ;)

Regards,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20131229/5792cf28/attachment.html>

[...]

Perhaps the Delphi developers have a look at FPC discussions and innovations, and add useful features to Delphi?

DoDi

That would be a first.

There have been several occasions when we had a feature first. They have always added it differently.

Michael.

Well, would they have added it without such an inspiration?

And yes, they'll have to add it in a different way, to prevent patents etc. infringements. Like MS insisted in a remake of all yet known errors, during Windows evolution, just to proof that they are the inventors of something new ;-)

DoDi

Objective C support for example. Embarcadero simply stuffed it into their COM support while we already had the Objective Pascal implementation...

Regards,
Sven

Or you port stuff to Lazarus not in one big bang, but project by project.

Or you use a lot of external packages.

That already assumes that they really took a cue from FPC in the first place. More likely, like the FPC developers, they looked at C#,Java,C++.

Survive is a big word and very black and white.

Not sustainable on the current level with a serious hope of continued growth? Probably.

IMHO that effect has always been exaggerated. If it already happens, it doesn't really lead to growth in FPC/Lazarus, only people using the finished product in silence.