Hello,

disallowing "AnsiString" code for UTF-8 is a huge regression.

I use TStringList for UTF-8 strings. This is no longer possible, because automatic conversions cause question marks and data loss.

I also use a large amount of third-party libraries that use the AnsiString data type for UTF-8.

I really want to use FPC 3 due to other things, but this is a deal breaker. Why not add a simple switch or even a run-time Boolean global variable to turn off codepage conversions?

It behaves differently from Delphi too.

Cheers,
Tobias

Lazarus uses TStringList with UTF-8 all over the place.

Please post a complete example demonstrating the problem.

Mattias

Same answer as in my other mail. Set DefaultSystemCodePage to CP_UTF8.

This depends on the version of Delphi :)

Michael.

Sorry - this was only theoretical, because of the Backward compatibility section on the FPC Unicode Support page.

It says that a "defined way" to use strings is "you do not store data in an ansistring that has been encoded using something else than the system's default code page, and subsequently pass this string as-is to an FPC RTL routine".

That would mean I cannot use TStringList for UTF-8. The paragraph is misleading, really. Very theoretical. What you really need to tell people is something like this:

"Unicode aware Pascal code needs to set DefaultSystemCodePage to CP_UTF8".

I am sorry but I was really shocked this morning when I saw the question marks :)

Cheers,
Tobias

Very theoretical. What you really need to tell people is something like this:

That please update the wiki - it is user editable. Even a seasoned developers as myself still needs to get my head around all this FPC Unicode stuff. So any information and tips on the wiki would be greatly appreciated.

I haven't moved to FPC 3.0 yet, but when I do, I too will have lots of testing to do in my own code. I don't use LCL, but but do currently store UTF-8 text inside AnsiString's for years (on all platforms).

Regards,
- Graeme -

That please update the wiki - it is user editable.

Done:
http://wiki.freepascal.org/FPC_Unicode_support#Backward_compatibility

I hope this is correct.

Cheers,
Tobias

tobiasgiesen wrote on Mon, 04 Apr 2016:

It is incorrect in the sense that there is nothing utf8-specific about the way your code (ab)used ansistrings. I will fix it, since that page is more or less part of the official FPC documentation (since it's linked from the FPC 3.0 release notes).

Jonas

I use TStringList for UTF-8 strings. This is no longer possible, because automatic conversions cause question marks and data loss.

You are completely lost with this issue. The automatic conversion of encodings is a big step forward.
Just use the new UTF-8 mode provided by Lazarus and remove all explicit conversion functions.
http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus

Juha

Juha Manninen wrote

Just use the new UTF-8 mode provided by Lazarus and remove all explicit conversion functions.

This is the FPC mailing list. Not everybody here uses Lazarus or LCL, so making such a suggestion is wishful thinking. For example, your suggestion means nothing to me, I don't use LCL.

Regards,
- Graeme -

Graeme Geldenhuys wrote

This is the FPC mailing list. Not everybody here uses Lazarus or LCL, so making such a suggestion is wishful thinking. For example, your suggestion means nothing to me, I don't use LCL.

Yes, I should have mentioned that this feature does not require LCL. It only requires LazUtils package and LazUTF8 unit in your uses section.
It can be used in cmd line and server programs and I guess in fpGUI, too, although I have not tested.
But yes, it requires Lazarus IDE because LazUtils is a Lazarus package. At least you must create and compile the project using Lazarus IDE.

Anyway, this UTF-8 mode does more that sets the default String encoding. It also provides proper UTF-8 functions as backends for RTL's Ansi...() string functions.
It also uses cwstring although it pulls in clib.
Then typical users' code is amazingly Delphi compatible despite the different encoding, because code only seldom deals with individual codepoints beyond 7-bit ASCII.

Juha

First of all it's part of LazUtils. So you don't have to use the LCL for that. In fact you don't have to use LazUtils: some users simply copied the two units FPCAdds and LazUTF8. It's all open source.

Second I find it funny that the statement comes from you - a notorious promoter of software on forums/lists of competing projects.

And third setting the DefaultSystemCodePage is a good start, but not enough. Instead of explaining all the gory details, Juha promoted a more complete solution for UTF-8. This is useful for many users. They don't have to reinvent the wheel.

Mattias

[...] But yes, it requires Lazarus IDE because LazUtils is a Lazarus package. At least you must create and compile the project using Lazarus IDE.

Or simply copy the two units FPCAdds, LazUTF-8 or parts of them from here:
http://svn.freepascal.org/svn/lazarus/tags/lazarus_1_6/components/lazutils/

Mattias

"Unicode aware Pascal code needs to set DefaultSystemCodePage to CP_UTF8".

That can't be this ubiquitous. I do suppose that the default value is supposed to make sense in many cases.

OTOH, if - as you seem to suggest - there is any conversion at all when using TSTringList to store your UTF8 strings, (independent whether it "works" or not) this will introduce a decent performance problem. I don't know it that depends on the setting of DefaultSystemCodePage.

Please let us know what you find. (right now Lazarus does not seem to compile for me with 3.1.1, so I can't easily check myself.)

-Michael

Mattias Gaertner wrote

Or simply copy the two units FPCAdds, LazUTF-8 or parts of them from here:

Thank you Juha and Mattias - I'll take a look at those to see what they do.

Regards,
- Graeme -

(How) does the new UTF-8 mode in Lazarus change the way TStringList works (as this is what Tobias is concerned about) ?

-Michael

Mattias Gaertner wrote

for that. In fact you don't have to use LazUtils: some users simply copied the two units FPCAdds and LazUTF8. It's all open source.

This was not made clear until you explicitly mentioned it. Juha's initial comment was vague on the matter, and the original poster never mentioned they used Lazarus or LCL.

Second I find it funny that the statement comes from you

I simply wanted an answer or explanation that benefits anybody using FPC.

more complete solution for UTF-8. This is useful for many users. They don't have to reinvent the wheel.

Not having looked at the two units you mentioned... but if this is a general requirement for anybody using UTF-8 or similar with FPC 3.0, then wouldn't it be best to see if those units can be contributed to FPC's FCL? The ultimate "don't reinvent the wheel" location. ;-)

Regards,
- Graeme -

One would think so but:

1. Using UTF8 is a choice of lazarus. Other people may prefer UnicodeString. On Windows, UnicodeString is more 'natural' or 'native'.

2. The release cycle of FPC is rather long, so updates will be available not as fast as the lazarus team needs them.
And in view of 1. that may be a problem.

If memory serves well, there was initially an attempt to get some of the functionality into FPC by Felipe, but this was quickly abandoned due to above arguments...

Michael.

Michael Van Canneyt wrote

1. Using UTF8 is a choice of lazarus. Other people may prefer UnicodeString. On Windows, UnicodeString is more 'natural' or 'native'.

Based on Internet standards and most popular OSes (mobile devices included), UTF-8 is kind - so we all know Windows backed the wrong horse [encoding]. ;-)

[...Graeme runs and hides...]

2. The release cycle of FPC is rather long, so updates will be available not as fast as the lazarus team needs them.

That's a valid point.

Though it could probably be added as quick as in FPC 3.0.2. It's simply two new units that need to be explicitly used by somebody to have any affect, so it will not break existing code otherwise [if not used].

Regards,
- Graeme -

Well, in 2016, I still only use UTF-8, even on windows. It works without problems if you know what you're doing.

They should at least be renamed, to avoid confusion.

Other than that, I personally see no objections.

Michael.

Michael Van Canneyt wrote on Mon, 04 Apr 2016:

On Mon, 4 Apr 2016, Graeme Geldenhuys wrote:

[add LCL UTF-8 helper units to FPC]

I do: it's more units that we have to maintain, process bug reports and feature requests for, etc (or, in case they are supposed to remain copies of the Lazarus units, then it's extra work keeping them in sync and given the non-synchronised release cycles, they will almost never be in sync). We already have plenty of work with our own code.

Jonas

And that is why I wrote 'personally' :-)

Michael.

Am 04.04.2016 13:21 schrieb "Graeme Geldenhuys" <

: On 2016-04-04 12:06, Michael Van Canneyt wrote: > 1. Using UTF8 is a choice of lazarus. Other people may prefer

UnicodeString.

Qt uses UTF-16 as well...

(and our company's OS uses UTF-32)

Regards,
Sven

Sven Barth wrote

Qt uses UTF-16 as well...

I always thought that strange. After all, Qt was born as a Unix-type GUI toolkit. Unless I got my facts wrong. Then again, it's only in recent years that Unix-like systems moved to UTF-8. I think even FreeBSD didn't use UTF-8 out of the box (until the last one or two releases).

Java also uses UTF-16. What I like about those two are that they only have one string type. No confusion, but then they don't have as long a history as Pascal, so no legacy code to worry about.

Regards,
- Graeme -

Juha Manninen wrote

Just use the new UTF-8 mode provided by Lazarus and remove all explicit conversion functions. http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus

I just did some tests and it seems that TStringList (this is what Tobias is concerned about) (with the Lazarus installation straight from the svn) seemingly always uses UTF8.

Is this correct ?

-Michael

No. It uses the DefaultSystemcodepage. If the system codepage is UTF8, then
it will use UTF8.

Michael.

Michael Van Canneyt wrote

It uses the DefaultSystemcodepage. If the system codepage is UTF8, then it will use UTF8.

Thanks for the enlightenment.

Am I right assuming that the DefaultSystemcodepage is determined when compiling the RTL and/or the compiler) ? (As the default code page for the type Strings is not dynamic, but a built-in setting.)

So Lazarus (IDE/LCL) can't do anything about that, as it will not enforce re-building the compiler and/or the RTL.

-Michael

Michael Van Canneyt wrote

It uses the DefaultSystemcodepage. If the system codepage is UTF8, then it will use UTF8.

(Sorry for replying yet another answer to the same message of yours)

http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus says:

On the other hand a *String* is assumed at compile time to have /DefaultSystemCodePage/ (CP_ACP). /DefaultSystemCodePage/ is defined at run time, so the compiler conservatively assumes that /String/ and /UTF8String/ have different encodings.

So I was wrong assuming from your message that DefaultSystemcodepage is the default encoding for the type "String" (and with that for TStrings and with that for TSringList).

Now I still don't know whether/how the default encoding for the type "String (which I still assume is defined when building the compiler) is depending on the arch/OS the compiler is built for. (I only tested on Linux and here it rather obviously is UTF8. I assume on Windows it's UTF16 for Delphi compatibility).

-Michael

Your assumption is wrong.
The default encoding for the string type is determined at run-time, not at compile time.

Michael.

Michael Van Canneyt wrote

The default encoding for the string type is determined at run-time, not at compile time.

How can that work for string constants ? Will they in fact (virtually) change their encoding when DefaultSystemcodepage is different ?

For a test I did   result := StringCodePage('äü');

This results in a 0 which is CP_ACP, which is supposed to mean "in fact its the value of DefaultSystemcodepage", which in my test in Linux is 65001 which means UTF8.

Of course in this test it's correct. But if I start the executable in a system that imposes another value of DefaultSystemcodepage or if it is compiled in a way that results in the constant strings to be coded in another way than 65001, what will I see ? (Right now I don't have a Lazarus installation on Windows at hand).

-Michael

It depends.

For string constants there are slightly different rules. There the result depends on the {$codepage} directive of the source file.

See
http://wiki.freepascal.org/FPC_Unicode_support

Again, see
http://wiki.freepascal.org/FPC_Unicode_support

Michael.

Michael Van Canneyt wrote

For string constants there are slightly different rules. There the result depends on the {$codepage} directive of the source file.

Hmmm.

If not setting $codepage Ifor a constant string I get StringCodePage = 0,

If setting {$codepage UTF8} for a constant string I get StringCodePage = 1200 which is UTF16.

The program does work decently, anyway.

And ASM stepping shows that no conversion is done.

-Michael

Michael Schnell wrote

For a test I did result := StringCodePage('äü');

If you as a programmer knows the unit is saved in UTF-8 encoding, then add {$codepage utf8} to the top of the unit. That tells the compiler how to interpret string constants in that unit (without the need for any guessing).

Regards,
Graeme

Graeme Geldenhuys wrote

If you as a programmer knows the unit is saved in UTF-8 encoding, then add {$codepage utf8} to the top of the unit. That tells the compiler how to interpret string constants in that unit (without the need for any guessing).

I did some test with results that I can't get sorted.

Do you suggest that the codepage of the sourcecode is preserved by the compiler when creating the string constant in object code ? I did not expect this but it of course is a possible way to go.

But if that is true, StringCodePage of a constant never should return 0 which is CP_ACP, which is supposed to mean "in fact its the value of DefaultSystemcodepage" and hence is not predefined in the executable file.

Seemingly StringCodePage does not work correctly with string constants anyway: I found that setting {$codepage UTF8}, for a constant string I get StringCodePage = 1200 which is UTF16.

-Michael

That's correct. String literals in a codepage other than system are stored as UTF-16 in the binary and converted on assign. The conversion happens at runtime, so the string codepage is decided at runtime.

Mattias

[...] Do you suggest that the codepage of the sourcecode is preserved by the compiler when creating the string constant in object code ?

It depends.
There are two codepages. The real one and the one you tell the compiler.
If you tell the compiler that the string literal is 8-bit system codepage, it will copy it without conversion to the binary. Otherwise it converts it to UTF-16.

For instance using {$codepage utf8} tells the compiler to convert all your literals to UTF-16.

Without the {$codepage} the compiler preserves the real codepage.

Seemingly StringCodePage does not work correctly with string constants anyway: I found that setting {$codepage UTF8}, for a constant string I get StringCodePage = 1200 which is UTF16.

StringCodePage on a literal is pretty useless. You should use StringCodePage on variables.

Mattias

Mattias Gaertner wrote

That's correct. String literals in a codepage other than system are stored as UTF-16 in the binary and converted on assign. The conversion happens at runtime, so the string codepage is decided at runtime.

That's correct if the assignment is to a variable/parameter that has codepage CP_ACP. If it's an assignment to an ansistring(some_codepage) (except for rawbytestring/cp_none), then the string constant gets converted to that codepage at compile time.

Jonas

Mattias Gaertner wrote

That's correct. String literals in a codepage other than system are stored as UTF-16 in the binary

(Assuming with "other than system" you mean different from the DefaultSystemcodepage setting the compiler sees at it's runtime).

I see. And of course that will work.

Even though IMHO this is rather hard to understand, as UTF 16 seems to be a rather ineffective coding of constants (for software generally working with UTF-8) regarding as well storage as conversion effort.

Seemingly the compiler assumes that the executable likely at runtime will see DefaultSystemcodepage not equal the {$codepage setting in the source code and abstains form trying to prepare the possible optimization the user might have intended by using {$codepage .

and converted on assign.

By "on assign" you mean "when a constant string is assigned to a string that in it's type has a defined codepage (CP_ACP or any other 7 or 8 bit encoding, or CO_UTF16BE)", but not if it's CP_NONE or CP_UTF16).

The conversion happens at runtime, so the string codepage is decided at runtime.

I'll re-check if I see the convertor working...

-Michael

Mattias Gaertner wrote

For instance using {$codepage utf8} tells the compiler to convert all your literals to UTF-16. Without the {$codepage} the compiler preserves the real codepage.

I.e. (compiling in a UTF-8 based Linux)

- using {$codepage utf8} tells the compiler to convert all your literals to UTF-16
- not using {$codepage utf8} tells the compiler to convert all your literals to UTF-8

The compiler is making fun of me ... :-)

-Michael

Mattias Gaertner wrote

StringCodePage on a literal is pretty useless. You should use StringCodePage on variables.

Just exploring how the compiler works...

-Michael

That's a, wrongly described, implementation detail. When any {$codepage xxx} directive is specified, string constants in the source are represented in a way that makes lossless conversion to any other code page possible. This conversion to the target code page is performed at compile time where possible (when the target code page cannot change at run time), and otherwise at run time.

What this internal lossless representation is at compile time and/or run time is undefined.

- not using {$codepage utf8} tells the compiler to convert all your literals to UTF-8

No, it does not. Please tell me which sentence of
http://wiki.freepascal.org/FPC_Unicode_support#String_constants suggests that in any way.

Jonas

Jonas Maebe wrote

No, it does not. Please tell me which sentence of http://wiki.freepascal.org/FPC_Unicode_support#String_constants suggests that in any way.

I just was making fun of myself, naively supposing the contrary :-) ;-)

-Michael

Of course I do understand that.

But anyway, AFAIK, UTF8 already is a way of lossless coding, so I don't see a forcing necessity to convert that to UTF16 at compile time. And as far as I understand, if the user does not take some means, the executable will work with 8 bit coding and very likely with UTF8, so holding the constants as UTF16 increases as well memory as CPU resource usage.

(I am not trying to bash anybody ! It obviously does work nicely and supposedly close to always as expected, and the possible resource overhead will be extremely rare and minimal).

-Michael

BTW.:
http://www.freepascal.org/docs-html/rtl/system/defaultsystemcodepage.html says that DefaultSystemcodepage can be modified in the user suppose that will change the way strings with StringCodePage() = CP_ACP are handled.

I'll do some tests...

-Michael

Michael Schnell wrote on Tue, 19 Apr 2016:

The reasons are
a) the FPC compiler binary itself prior to 3.x did not contain any UTF-8 encoding support. All it could do was convert the source file code page to UTF-16.
b) FPC's widestring manager does not contain any interface to directly convert from one 8 bit encoding to another, only from 16 bit to 8 bit and vice versa (which also made it useless to convert to UTF-8 at compile time, since there was no way to convert it to another code page at run time except by making a round trip via UTF-16 anyway). The reason is that these helpers were already necessary to convert between widestring/unicodestring and other types when assigning such variables to each other
c) changing b) would require a lot of testing because not all code page conversion libraries/OS interfaces support converting from any arbitrary character set to any other arbitrary character set. While it is likely that they all support converting from arbitrary code pages to UTF-8 and back (like they do for UTF-16), this would still have to be tested and additionally such an interface would undoubtedly also starting to get used for other code pages by people unaware of this limitation. Adding an interface limited to converting from/to UTF-8 would be another option to address that though.

In the end it would be a lot of work, result in a lot of extra code that may not work everywhere (or in specialised routines), and it would be for a use case you can already address yourself if you absolutely want to be completely UTF-8-centric: you can declare your string variables as UTF8Stringm since then the conversion to UTF-8 for constant strings will happen at compile time. If your DefaultSystemCodePage is CP_UTF8, no extra conversion will happen when assigning/converting these variables to regular ansistrings. Furthermore, converting from UTF-8 to other code pages is probably slower than from UTF-16, since UTF-8 is a more complex encoding for most characters.

The fact that other things are less convenient if you use UTF8String is the price you will have to pay for such code specialisation (which probably won't make any noticeable difference in 99.999% of the cases anyway).

Jonas

Jonas Maebe wrote

The reasons are

Thanks a lot or the great explanation.

So there are good reasons to stay with the status-quo (unless doing a completely new versatile and straight forward String implementation that exceeds functionality and "mind" Delphi allows, but this again would be a whole lot of work with maybe limited benefit)

-Michael

Isn't some formality in these Unicode discussions called for? Use of everyday language to express things which can only be properly expressed and tested through source code is very confusing.

Consider these few sentences by Mattias

>>>

It depends.
There are two codepages. The real one and the one you tell the compiler.
If you tell the compiler that the string literal is 8-bit system codepage, it will copy it without conversion to the binary. Otherwise it converts it to UTF-16

>>>

How do you know what the "real" codepage is at runtime or compile time, as opposed to "the one you tell the compiler"?
Is there some macro that can print out which codepages is active at some stage in the compilation or at runtime?
Can codepages be switched around at various stages in the compilation?

Being an English speaker and writing software for English speakers only, these are issues I have never had to deal with and hopefully, never will, which is just wishful thinking, but I think that all these discussions must be accompanied by code and use cases whch explain them better. I have been looking in these Unicode discussion threads and I am none the wiser after all these years.

There must be some test suites in FPC and Lazarus which cover the issues being raised in these discussions. Wouldn't those be a better starting point?

Unicode discussions are leading me to appreciate Edsger Dijkstra more and more. ;)

vfclists . wrote

Isn't some formality in these Unicode discussions called for? Use of everyday language to express things which can only be properly expressed and tested through source code is very confusing.

The formal definitions can be found at
http://wiki.freepascal.org/FPC_Unicode_support

On the aforementioned page:
* "the one you tell the compiler": the source file code page if explicitly specified
* "the real code page": the value of DefaultSystemCodePage on startup of your program

Jonas