The RC1 and RC2 versions have been tested for a while now but we have not heard any complaints about the new improved Unicode support. Does that mean it works perfectly?

Windows is of biggest interest. There the new UTF-8 support breaks existing code when it depends on system codepage.
For that reason LCL can also be used without the new UTF-8 support by defining "DisableUTF8RTL".
The relevant wiki pages:
http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus http://wiki.freepascal.org/Lazarus_with_FPC3.0_without_UTF-8_mode

Feedback please.

This is from taazz, copied from the "New menu designer" thread:

LCL continues to work as before when you define "DisableUTF8RTL". Then AnsiString is coded with the system codepage by default. The UTF-8 conversion functions still work.
Supporting this backwards compatible mode became a high priority because you and some others expressed concerns about the default UTF-8 encoding.
Are you saying you will not even test the "DisableUTF8RTL" mode? Its known bugs were fixed already. More feedback would be nice.

If you still decide to use FPC 2.6.4, you can do it with Lazarus 1.6. It will be compatible for the life-cycle of 1.6.x. Then AnsiString is 100% compatible with Lazarus 1.4.4 (obviously). See, you can freely choose the best combination.

I don't know why you always find excuses to complain. It is not nice. Please stop it.

Juha

Juha Manninen wrote

LCL continues to work as before when you define "DisableUTF8RTL". Then AnsiString is coded with the system codepage by default. The UTF-8 conversion functions still work.

You may have a false impression of how Free Pascal/Lazarus is used by most people. At least I did not wait until the current version to use Unicode (and long paths) for file names and others. I think many did so.
Therefore we have build our own infrastructure/functions to work with the Windows unicode API functions. In this case using "DisableUTF8RTL" would be no option as it bombs us back to ANSI/short paths and still requires to do code changes. That's not good.

So we need to live with the changes and understand the logic behind it (which IMO is much more complicated than before). The amount of time we have to invest is unforseeable for us and therefore many get frustrated (especially, because the documentation is poor).

I have done the changes to (some) of my programs now and it works quite okay but it was a hard time until this was finished (and I still don't know whether bugs lurk here and there).

I don't know why you always find excuses to complain. It is not nice. Please stop it.

Please stop ignoring and gibing those who have problems with the new unicode types. I know how they feel and such statements are not helpful. If you don't want to help then ignore these mails. I know that those who have already worked on this for a long time cannot understand why not everybody instantly knows how the new codepage aware ansistrings work but be assured that for many this is completely new. So bare with our frustration and better ignore such mails than revile the writers.

I don't understand how DisableUTF8RTL bombs your system because it is the backwards compatible thing.
If your system worked with FPC 2.6.4, it should work with FPC 3.0 + DisableUTF8RTL.
If it does not work then there are bugs which should be fixed. Do you mean the A- / W- versions of WinAPI calls by the short/long path thing? Enabling or disabling the UTF8RTL does not affect WinAPI calls. It is a separate issue.

Excuse me! Are you really saying I have not helped with the Unicode issues? I am amazed the system works already as well as it does.

In my mail I explicitly asked for feedback so we can still improve this system. Yes, this is a very complicated issue. For example I am at my limits of understanding / not-understanding the details, but still I tried to improve things instead of complaining about how poorly the voluntary developers have done the job.

I have worked on the DisableUTF8RTL feature although I don't use it myself. Fortunately there were helpful people like "michl" (forum name) who fixed bugs.
I also wrote some wiki pages. They are not perfect but they are better than nothing.

Now we support 3 ways with Lazarus 1.6 :
1. The improved UTF-8 system with FPC 3.0. The default codepage of AnsiString = UTF-8.
2. Backwards compatible system with FPC 3.0 by defining DisableUTF8RTL. AnsiString uses system codepage.
3. Continue using FPC 2.6.4. All string types and encodings are 100% compatible with earlier versions.

The 4. way will be a Delphi compatible UTF-16 sometime in future.

How to improve the 3 supported ways? The issues I know are that FPC libs still use the old WinAPI calls in some places, and the TFormatSettings separators issue. Are there other bugs? How to solve them?

Juha

Juha Manninen wrote

I don't understand how DisableUTF8RTL bombs your system because it is the backwards compatible thing. If your system worked with FPC 2.6.4, it should work with FPC 3.0 + DisableUTF8RTL.

Yes, you are right.
DisableUTF8RTL resets *some* of the changes (those related to the API interface) and does not harm functions that already used the Win API directly.

But as one example, since FPC 3.0 you cannot use UTF8Delete on UTF8Strings anymore. This requires changes to existing programs that cannot be avoided by a compiler switch. Not everybody is prepared to invest this time just now.

This is no criticism of the good work that has been done to finally make the long overdue step to codepage-aware strings. Still everybody should also be aware that this is a radical change in the already complicated area of string handling and due to its complexity many FPC users experience these changes as a burden in the first place. On the long term it was unavoidable of course but just now it generates frustration.

Excuse me! Are you really saying I have not helped with the Unicode issues?

No, that's not what I said.
But if you got fed up with all these questions about unicode it would be better to ignore some of the mails that bother you instead of increasing the frustration of others by telling them they are just too lazy (or dumb) to read (and understand) some wikis.

J?rgen Hestermann wrote

But as one example, since FPC 3.0 you cannot use UTF8Delete on UTF8Strings anymore.

Already fixed in trunk (r50850 and r50910) and merged into 1.6RC2. Please stop complaing about things that were already fixed.

Bart.

J?rgen and taazz,
The DisableUTF8RTL system indeed is a fall-back for people who cannot use the new improved UTF-8 system.
The automatic encoding conversion may do tricks when you still must do explicit conversion to/from UTF-8 using the old clumsy functions. I don't know about all the issues involved. I don't plan to use it myself. It is not part of my interest. The wiki page explaining it should be improved and maintained by people who actually use the system which means J?rgen, taazz, michl and others. Now the wiki page has too few examples.

My interest has been the new UTF-8 system. It works much better than I could imagine.
Anybody who plans now to move their code to Lazarus 1.6, please try to isolate Windows codepage conversions into functions. UTF-8 then works automatically like magic outside of those functions.

Juha

Juha Manninen wrote

J?rgen and taazz,

...

The wiki page explaining it should be improved and maintained by people who actually use the system which means J?rgen, taazz, michl and others.

Now that is cynical!
Telling those who do not know anything about what
has been developed over the last years
to write the documentation theirselfs.

Not the developers who know what they changed
and what they had in mind when doing so should
document their work but those who just use it.

For whom is the documentation meant at all?
Why not drop it and everything is fine?
The code is there, just read it!

I think with this attitude the whole project is doomed to fail.

J?rgen Hestermann wrote

Now that is cynical! Telling those who do not know anything about what has been developed over the last years to write the documentation theirselfs.

No, my point is that I will not continue doing it. Somebody else must take charge. If nobody wants to do it, then bad luck for you. I have done my share for a feature I don't even use myself. I added the define + basic documentation for it and applied some patches from Michl to fix its bugs.
Are you saying that I should feel guilty now for not maintaining it further. This is how voluntary open source works. Basically people scratch their own itches. If somebody feels this DisableUTF8RTL feature is important then he improves it. If nobody feels so then it will not be improved. Simple as that.

Not the developers who know what they changed and what they had in mind when doing so should document their work but those who just use it.

Yes, it is already documented here:
http://wiki.freepascal.org/Lazarus_with_FPC3.0_without_UTF-8_mode However the future maintainer should add more examples and explain how to solve problems that come up.

For whom is the documentation meant at all? Why not drop it and everything is fine? The code is there, just read it!

Exactly! Why don't you do so?
You and me are in the same position. Neither of us gets paid for improving Lazarus. Both of us can improve things we feel important. Tell me, why don't you improve the feature you apparently consider important?

I think with this attitude the whole project is doomed to fail.

Right. What about your attitude?
Why do you attack people who actually created some substance for this project? Why don't you attack the people who did not create any substance? It would be more logical IMO.

Juha

J?rgen Hestermann wrote

Now that is cynical!

No it is not.
Lazarus has taken the direction of using the new UTF8 system. Most of us do not use the DisableUtf8 feature at all, since we don't need it. Hence we have no experience.

Users who want that feature should test it.
Failures will most likely be in cases we cannot think of (or else we would have fixed it already)

Your testing will give valuable info about bugs, workarounds, caveats etc. All of which you can add to the wiki, by you, since the wiki belongs to the community, and you have the proper information.

For whom is the documentation meant at all? Why not drop it and everything is fine? The code is there, just read it!

Please stop this kind of ranting.
The only thing you accomplish is that you alienate yourself from us.

Report bugs.
We will try to fix them, which may involve you having to do more tests (as it has always been).
Provide patches if you can.
Stop complaining about things that are already fixed. We will work with you, as long as you don't work against us.

Bart

2016.01.17. 12:34 keltez?ssel, Juha Manninen ?rta:

Feedback please.

Hi,

A complicated example:

Label1.Caption:=WideChar($25BD);

With fixes_1_4(and FPC 3.0.0) I see a question mark on the screen, with fixes_1_6 I see the wanted "WHITE DOWN-POINTING TRIANGLE". I am happy with fixes_1_6! :-)

Gabor

With or without -dDisableUTF8RTL ?

Bart

2016.01.18. 23:20 keltez?ssel, Bart ?rta:

With or without -dDisableUTF8RTL ?

Checkout, make bigide. I did not change anything.

Gabor