Hi,

Not sure if this URL will work for all, but it is the announcement of Beyond Compare 4 beta - in the Google+ Delphi Developers page.

https://plus.google.com/u/0/115543678481226143222/posts/TcfAzzewBkr

Craig Peterson mentioned that the OS X version is created with Lazarus 1.2. :)

"Beyond Compare 4.0 beta is now available, on Windows, Linux, and OS X. In addition to the OS X support, it adds an enhanced UI, folder merges, Subversion, WebDAV, and Dropbox support, inline editing in the hex compare, and more.
http://www.scootersoftware.com/beta? "

...and...

"For anyone wondering, the OS X version was done using Lazarus 1.2, the Windows version uses Delphi XE, and the Linux version still uses Kylix.?"

Regards,
- Graeme -

Graeme Geldenhuys wrote

"For anyone wondering, the OS X version was done using Lazarus 1.2, the Windows version uses Delphi XE, and the Linux version still uses Kylix.?"

Noticed he posted that the reason the Linux build still uses Kylix was to limit the amount of big changes... so perhaps a Linux Lazarus version is in the future ;)

"As for Windows, I don't really want to use Lazarus there. The Lazarus and Free Pascal teams have done excellent work, but there are downsides: Unicode support in Free Pascal on Windows is still incomplete"

Which hopefully makes clear that Delphi compat unicode is needed by some :-)

Maybe he could forget Delphi and make everything using FPC and Lazarus when the Unicode support on Windows get ready. :-)

Marcos Douglas

Marcos Douglas wrote

> Which hopefully makes clear that Delphi compat unicode is needed by some :-) Maybe he could forget Delphi and make everything using FPC and Lazarus when the Unicode support on Windows get ready. :-)

Sure, right after Microsoft adapts Lazarus :-)

If you dream unrealistically, set your targets high, since it won't matter anyhow :-)

Come on, was just a joke. ;-)

Marcos Douglas

"As for Windows, I don't really want to use Lazarus there. The Lazarus and Free Pascal teams have done excellent work, but there are downsides: Unicode support in Free Pascal on Windows is still incomplete"

I wonder what exact "support" he means.

Which hopefully makes clear that Delphi compat unicode is needed by some :-)

UTF-8 supports the same range of characters as UTF-16. When functions are missing, they can be added to either encoding.

Above quote only suggests to me that people with such a detailed opinion about "Unicode support" don't know what they really need.

DoDi

Hi All,

Yep, Beyond Compare 4 for OS X uses Lazarus with the Carbon backend. Thank you to all of the hard working Lazarus and Free Pascal developers that helped make it possible.

While I did mention some things in that post that I didn't like about Lazarus/Free Pascal, I think you've done excellent work. I was really impressed with how much worked out of the box and how compatible everything was, both in the LCL and the compiler. The Objective Pascal extensions are really well done. Not having the IDE go brain dead while debugging was a welcome change too. :)

We actually got the core product up and running fairly quickly; the last two years have been spent making it more at home on OS X. We've submitted most of our changes back to the team, and will get the remaining ones out soon. Unfortunately a number of the enhancements, like per-window sheet support, rely on a lot of our application level behavior, so they can't be incorporated into the LCL easily, but I am happy to answer any questions or share the relevant code snippets for anyone who wants them.

We will definitely be looking into LCL/Qt for our Linux version once we get v4 released. Seeing the Qt5 bindings has made me very happy.

Marco van de Voort wrote

On Tue, Dec 24, 2013 at 04:44:37PM +0000, Graeme Geldenhuys wrote: Which hopefully makes clear that Delphi compat unicode is needed by some :-)

As long as "string" supports Unicode, and AnsiString/UTF8String casts do appropriate conversions, I don't really care whether it's UTF-8 or UTF-16. One of the things that made code sharing easier is that OS X and modern Linux both use UTF-8 for C strings, so we were able to use string/TStringList/etc on all three platforms, with all three compilers, and know that it was always Unicode. AnsiString was used in places on Windows where we needed it, but that was about it. There were cases where we had to use UTF-16 on OS X to minimize conversions during drawing, and there were cases on Windows where we had to use UTF-8 to reduce memory usage.

Delphi compatibility in general has been very important, though, and I appreciate all the efforts in that direction. BC's source code needed it, of course, but we also rely on some large third party libraries, like Indy and SecureBlackBox, and have had to port several others ourselves (ZipForge, Abbrevia, chsdet, some bits of the JCL). I'm quite confident that if there had been a significantly higher barrier to bringing our code over we would have switched to a non-Pascal environment instead.

In any case, thanks again. We really couldn't have done it without this community.

Hans-Peter Diettrich wrote

I wonder what exact "support" he means.

Wonder no more:
1) "string" uses a Unicode encoding
2) AnsiString/UTF8String casts perform appropriate conversions
3) File APIs support Unicode filenames
4) Windows headers use W version by default.
5) LCL and RTL agree on string encoding
6) In a stable Lazarus and FPC release

As long as the language and libraries agree on an encoding I don't really care whether it's UTF-8 or UTF-16. I've also been working with Unicode since 2001 though, and have had to deal with plenty of different encodings, file formats, and API conversions in that time, so I don't think our usage is indicative of the average Delphi programmer, and I wouldn't want to speak for anyone else.

Hi Craig,

Thanks for giving some interesting details of my favorite tool!

I use BC on Linux (Debian 7 LXDE is the latest) and on Windows. I need to get a Mac so I'm glad to learn about things going there.

I just added a Lazarus port (which I hope works on the Mac) to this page: http://www.ctrlterm.com/ports.htm

And also, thanks for your help with Abbrevia!

Best regards,
Paul Breneman
www.TurboControl.com

"As long as the language and libraries agree on an encoding I don't really care whether it's UTF-8 or UTF-16"

+1

Marcos Douglas

Actually this is the MSElang approach. MSElang supports "string8" (utf-8), "string16" (utf-16), "string32" (UCS4) and "bytestring" which can hold any 8-bit encoding or binary data. There is no "string" type defined in MSElang and there are no implicit type conversions in MSElang. "
floatvariable:= floatvalue + integervalue;
string8variable:= string8value + string16value;
"
will not work and must be written as
"
floatvariable:= floatvalue + float(integervalue);
string8variable:= string8value + string8(string16value); "
The compiler then will do the conversions.
The MSElang RTL is the bare minimum, most of the functionality (lists, file API...) is implemented in an optimized manner in the framework, in MSEgui for example, which already does not use much of the FPC RTL or - in a hypothetical case - in LCL. ;-)
Every framework can use the stringtypes it likes, common RTL parts could be made in collaboration between the frameworks. Because MSElang compiler supports all without preference and the framework development does non need to wait on implementation decisions on complier or base-RTL level the development can be accelerated.
It is apparent that such an approach is very unfeasible for a commercial compiler manufacturer because the loose of control. I assume that the Delphi architecture has been carfully chosen in order to maximize the control over the ecosystem.
But for an opensource undertaking the MSElang principle could work. :-)

Martin

"Craig Peterson" wrote

On Dec 25, 2013, at 5:11 PM, Hans-Peter Diettrich

> I wonder what exact "support" he means. Wonder no more:

Most of these already apply to 2.7.1, you just need to wait for 2.8.0 to be released somewhen (hopefully) next year :)

1) "string" uses a Unicode encoding

For best compatibility you'll currently need to set the ANSI codepage to UTF8 as String=UnicodeString provides problems with inheritance etc.

2) AnsiString/UTF8String casts perform appropriate conversions

Should work already.

3) File APIs support Unicode filenames

Should work on the major targets as well.

4) Windows headers use W version by default.

Done in trunk since a few months.

5) LCL and RTL agree on string encoding

That's a usually hottly debated topic...

6) In a stable Lazarus and FPC release

Still takes a while...

Regards,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20131226/252e4c7e/attachment.html>

Thanks for this list :-)

Everything except (2) should have been available already, that's why I was wondering. Automatic conversions (2) were hard to implement without stored string Encoding, should come soon.

DoDi

(short status update)
Most of 1-4 is done on trunk, except (4) is not default and (3) might not be enitrely complete. Anything OO is not done (like tprocess).

You mean won't compile or raise an exception at runtime?

But what is the string type of MSElang RTL? If it will use string16 and my application uses string8 I will need to use conversion all the time, like SysToUTF8/UTF8ToSys functions... no?

Automatic conversions is good.
IMHO should exist a global variable that have the default codepage -- MSE is string16 -- and the user can change.
The compiler needs this variable to use the default encode when the user concatenate two or more strings with different encode. So, RTL can be string16, apps string8, Windows API string16, whatever... but the compiler will convert all calls automatically.

Marcos Douglas

Will not compile.

There will be not many functions with strings in MSElang RTL, most of them will be moved to the framework. BTW somebody can write a Delphi compatible framework-RTL if desired.
The remaining string routines (mostly string comparison and the like) will have overloaded versions for "string8", "string16", "string32" and maybe "bytestring". Bytestring will assume 8-bit encoding of the current locale.
RTTI will use "bytestring" with pure ASCII. If tcomponent (or its equivalence) is part of MSElang RTL (not yet decided), component names and the like also will use "bytestring" with pure ASCII.
File API is not part of MSElang RTL.

I don't think so. One easily can see how problematic automatic type conversion is. It actually stalls Free Pascal development since several years. It is seldom used, to type the conversion functions in code is not a big deal and helps to identify problematic designs.

IMHO should exist a global variable that have the default codepage --

It is not necessary. Because MSElang does not define and does not know the type "string" one can define "string" oneself or it is defined in the framework.
Lazarus probably will write
"
type
string = string8;
"
Although I would recommend to replace all GUI "string" types by "lazstring" and to define
"
type
lazstring = string8;
"

MSE is string16 -- and the user can change.

MSEgui will not use "string". What is currently "string" in MSEgui will be changed to "bytestring". GUI strings will remain "msestring". Filenames will remain "filenamety".

MSElang does not allow to concatenate different string types, they must be explicitely converted.
"
string8var:= string8value + string8(string16value) + string8(bytestringvalue1) + //assumes encoding of the current locale

       cp1251toutf8(bytestringvalue2); //uses conversion function,
                                       //not part of MSElang RTL

"

Martin

IMO your thinking is correct: simplify things, codify the minimum necessary, encapsulate others things. But you are constructing a "new" compiler, for your framework... you don't care about Delphi, LCL, etc -- I think FPC+LCL should not follow Delphi too but this is not the real world today -- and these things simplify the implementation.

You said File API is not part of MSElang RTL... so I will need to codify my own? This is not wrong but I think there should be a default implementation for these things... anyway.

IMHO the major problem is follow Delphi...

Resume:
MSElang do no allow to cancatenate different string types (must be explicitely converted) and the RTL does not will use string8. So if I want to use Lazarus (LCL) with MSElang I will continue to have the need to convert all call from/to MSElang RTL because they have different encode. Is that right?

Regards,
Marcos Douglas

Marcos Douglas wrote

Resume: MSElang do no allow to cancatenate different string types (must be explicitely converted)

Correct.

and the RTL does not will use string8.

Wrong. MSElang RTL suports string8 (utf-8), string16, (utf-16), string32 (UCS4) without preference. Additionally there is bytestring for any 8-bit encoding or binary data. Please remember, MSElang RTL is the bare minimum, file API for example must be implemented in the frameworks.

So if I want to use Lazarus (LCL) with MSElang I will continue to have the need to convert all call from/to MSElang RTL because they have different encode. Is that right?

No. You will use LCL as before. LCL implements the needed functions (file API for example) optimized for Lazarus purpose. As people who like to use MSEgui will use MSEgui functions which are optimized for use in MSEgui framework and fpGUI users probably can use a subset of LCL because fpGUI and Lazarus both use utf-8 and try to be mainstream compatible. It is possible that the framework developers work together in order to build commonly used base units and classes. It makes compiler and RTL-framework development somewhat independent.
The close coupling of compiler and framework is necessary for a commercial software producer in order to optimize profit, opensource development does not need it. It is not necessary that the compiler and its environment dictate how the frameworks must be implemented.

Martin

How do you will code this? Overload for each string type?

file API for example must be implemented in the frameworks.

I think this is an option even in FPC and Lazarus, ie, I see people, on the future, creating your own "RTL" because FPC and Lazarus do not agree about Unicode. :-)

Very interesting.

The close coupling of compiler and framework is necessary for a commercial software producer in order to optimize profit, opensource development does not need it. It is not necessary that the compiler and its environment dictate how the frameworks must be implemented.

+1
I agree.

Regards,
Marcos Douglas

PS: This thread is very interesting. Maybe you should open this topico in mse-list too.

Marcos Douglas wrote

> The close coupling of compiler and framework is necessary for a commercial > software producer in order to optimize profit, opensource development does > not need it. It is not necessary that the compiler and its environment > dictate how the frameworks must be implemented.

Profit has nothing to do with it. Usability has.
A compiler without some minimal support libiraries is dead-born.

Pascal has extensive file support. (Readln/Writeln) it makes for a large part of the system unit. if you take that out, little 'needed' is left, but at the same time you reduce the usability of the compiler.

That said:
Nothing stops anyone from implementing a totally separate RTL for FPC.

The number of internal functions that must be implemented for the compiler to be able to function is rather limited.

If you check the archives, you'll notice that I have often recommended people to do so.

Michael.

Marcos Douglas wrote

> file API for example must be implemented in the frameworks. I think this is an option even in FPC and Lazarus, ie, I see people, on the future, creating your own "RTL" because FPC and Lazarus do not agree about Unicode. :-)

Yes, I already started building my own file management routines mainly because I was never sure about the string encoding/type when using the RTL functions.

The current mismanagement regarding strings is a real pain. Even when looking into documentation there is no information about which string encoding the routines use/need.
For example, not a single word about string encoding here: http://www.freepascal.org/docs-html/rtl/sysutils/findfirst.html How can this documentation be used for programming? The most effective way to find out seems by trial-and-error. Not very inviting for programmers.

Yes.

In MSEgui it has been done so from start for many RTL parts. Currently most of needed functionality is implemented in MSEgui units.

Martin

There is no mismanagement. The released version is very clear.

The future version (trunk) is not yet clear, but if you are using that, you should be aware of the risks you decide to take.

We could remove all ansistring support tomorrow, if we felt like it.

Even when looking into documentation there is no information about which string encoding the routines use/need. For example, not a single word about string encoding here: http://www.freepascal.org/docs-html/rtl/sysutils/findfirst.html

For a simple reason: currently, in the released version of FPC, there is no need to document encoding.

The language guide clearly states that in Delphi mode, String = Ansistring. In ObjFPC mode it is shortstring, unless set to ansistring as well using {$H+}.

So, in both cases 'string' = a single byte string, system codepage.

There is - in the released FPC - no overloaded version with widestrings.

Hence there is no need to document encoding: it is implicit.

As soon as overloaded versions of these functions are released, the codepage/encoding issues will be documented.

Michael.

??.
So I can use any string type I like for the path of FindFirst? Strange.

The language guide clearly states that in Delphi mode, String = Ansistring.

What language guide?
If I look at the web documentation of FindFirst there is absolutely *no* information about the string type of the path. Why not?
It should be the reference.
It does not even tell me that it is a string:

   const Path: ;

What should the user think here?
That it can be an untyped variable like for fillchar?

In ObjFPC mode it is shortstring, unless set to ansistring as well using {$H+}. So, in both cases 'string' = a single byte string, system codepage. There is - in the released FPC - no overloaded version with widestrings. Hence there is no need to document encoding: it is implicit.

I don't understand what you are trying to say.
When I (as a programmer) want to use i.e. FindFirst I have to hand over a path. I now want to declare a string type for this.
Which one should I use?
Could it be whatever I want?
I can't find out with the documentation.

No. Ansistring.

> The language guide clearly states that in Delphi mode, String = Ansistring. What language guide?

The Object Pascal Language guide.

http://www.freepascal.org/docs-html/ref/ref.html

Specifically

http://www.freepascal.org/docs-html/ref/refsu10.html#x33-360003.2.4

Aha. That was a bug in fpdoc when releasing 2.6.2. Normally, it should read

const Path: string;

My apologies, I thought this had been fixed since the release :(

What should the user think here? That it can be an untyped variable like for fillchar?

No.

String.

Which means either AnsiString or ShortString.
So, in both cases a single byte string, system codepage.

Could it be whatever I want? I can't find out with the documentation.

I admit that the bug in fpdoc has not exactly helped. I have since fixed that, and in version 2.6.4, this problem should be fixed.

But, assuming that it says 'string', that gives you enough information.

Michael.

Michael Van Canneyt wrote

I admit that the bug in fpdoc has not exactly helped. I have since fixed that,

You mean it will be fixed here too some day?

http://www.freepascal.org/docs-html/rtl/sysutils/findfirst.html

Maybe a open source project would be a good idea. ;-)

The current mismanagement regarding strings is a real pain.

YES.

The FPC 2.6.2 do not has Unicode support yet. I think this is the motive.

Not very inviting for programmers.

I agree.

Marcos Douglas

Maybe is a good idea you separate some stuff:
1- lang
2- RTL
3- GUI framework

Marcos Douglas

So you do that, ie, make your own RTL? The major problem is work on Windows and I don't know if you work it.

Marcos Douglas

Yes.

As it happens, I am preparing the documentation for 2.6.4, and I checked: the bug is fixed. So the bug should be fixed on the website one of the next days.

Now, I realize that the strings issue is rather messy, Delphi engineers really messed it up.

Following your message, I looked up Delphi's docus:

http://docwiki.embarcadero.com/Libraries/XE5/en/System.SysUtils.FindFirst

No mention of encoding either.

I am looking for a way to make sure that when the unicode version is made, the documentation will make clear what 'string' means if it appears somewhere in a declaration.

You can help by making suggestions. Anything that does not require me going over all functions that require string arguments is good :)

Michael.

Why would I ? I don't need it.

Frankly, I don't understand what all the fuss is about. Everything is perfectly clear to me, I know how to deal with issues, if they pop up.

Yes, 2.8 or 3.0 will be a big change. They will include a unicode RTL somehow.

If you work with 2.6.X, there is no problem to understand the RTL: everything is ansistring: Single byte string, system codepage.

All the rest is either
- People who don't grasp the consequences of the above simple sentence.
- People spreading FUD.

And yes, I realize that if you really, really need unicode with 2.6.x, you have a problem with the current RTL. No-one ever denied this. At least, the Lazarus team tried to give a workaround with their UTF-8 functions.

The FPC team is working on a more fundamental solution. So either be patient, or contact me for instructions on how you can help.

But please, Please, PLEASE stop these pointless discussions.

Michael.

So PLEASE, tell us how to make a code on Windows safe, readable and strong using FPC 2.6.x (AnsiString) and Lazarus (UTF-8) because I and most people still not know.

Yes, 2.8 or 3.0 will be a big change. They will include a unicode RTL somehow.

Yes. So, how I can protect my code TODAY to the uncertain future?

If you work with 2.6.X, there is no problem to understand the RTL: everything is ansistring: Single byte string, system codepage.

I do not have problems to understand that.

Yes, yes. I'm not here to complain. I'm here to understand how I can solve my problems and, at the same time, help other people. I'm not doing FUD.

My questions are simple:
1. How can I make programs on Windows using FPC and Lazarus without problems with Unicode stuff?
2. What I can do now to protect my code for the new FPC implementations on the future?

The FPC team is working on a more fundamental solution. So either be patient, or contact me for instructions on how you can help.

I will be patient... but tell me how I can help anyway.

But please, Please, PLEASE stop these pointless discussions.

Sorry man, but other people have the same problems. That is real.

Thanks, anyway. Peace.

Regards,
Marcos Douglas

http://docwiki.embarcadero.com/Libraries/XE5/en/System.SysUtils.FindFirst No mention of encoding either.

Wake up! In XE5 String is UnicodeString, so what's unclear?

DoDi

Only use the lazarus UTF-8 functions.

You can not.

Who said that this was possible ?

No, but you do not seem to accept the consequences of this statement.

Namely: it is NOT possible to use unicode TODAY with the current RTL, and be future proof at the same time.

I didn't say you were.

My questions are simple: 1. How can I make programs on Windows using FPC and Lazarus without problems with Unicode stuff?

Today, you cannot.

2. What I can do now to protect my code for the new FPC implementations on the future?

You cannot.

Did I ever deny that there are problems ?

I quote myself:
"And yes, I realize that if you really, really need unicode with 2.6.x, you have a problem with the current RTL. No-one ever denied this."

When 2.8 arrives, you'll have work to do.
We'll try to do as much as possible to reduce this by offering you a choice. But you will have work to do, nothing can protect you from this.

I made the switch from Delphi 7 to Delphi 2010/XE once. This was a huge task.

You will just have to try and compile your code with the Unicode RTL, and fix problems as they pop up. Or stay in AnsiString and workarounds for UTF-8.

Michael.

Hi Marcos,

I do agree a Unicode enabled RTL is required, but just curious...

Marcos Douglas wrote

1. How can I make programs on Windows using FPC and Lazarus without problems with Unicode stuff?

What problems would those be?

I've implemented UTF-8 methods in fpGUI that wrap some RTL methods - similar I believe to what LCL does. Granted my Unicode needs are not that great, but I have had users of 8 different languages test Unicode support. It all worked well, or at least it was easy for me to fix any issues that cropped up in the tests.

Regards,
- Graeme -

Marcos Douglas wrote

So PLEASE, tell us how to make a code on Windows safe, readable and strong using FPC 2.6.x (AnsiString) and Lazarus (UTF-8) because I and most people still not know.

That is not true. Most people using LCL already know how to use UTF-8 with it, using AnsiString and the UTF8 functions.

Yes, yes. I'm not here to complain. I'm here to understand how I can solve my problems and, at the same time, help other people. I'm not doing FUD.

No, instead you behave like a child who repeats the same demand until his parent get tired and give it to him.
Now a clean Unicode implementation cannot be given to you because it does not exist.
You could have read the same things from FPC mailing list archive. They have been repeated again and again and again in many many threads, some of which went on for months, during past ~ 4 years.

Besides you already hijacked other threads for the same discussion. This one was originally about "Beyond Compare 4".

Let's give peace for FPC developers so they can implement the stuff instead of answering the same questions again and again. I personally am interested how easy it will be to work with UTF-8. However I know there are no answers yet so I will ask about it. I know this concern was expressed by many people and it has been acknowledged somehow.

Juha

However I know there are no answers yet so I will ask about it.

However I know there are no answers yet so I will NOT ask about it.

Uhhh...

> 1. How can I make programs on Windows using FPC and Lazarus without problems with Unicode stuff? What problems would those be?

The first problem is already to get an overview about all possible ways to do it. There are *a lot* of different functions available from *a lot* of different units which all have their own features and drawbacks.

How do you do UTF8 string sorting in Windows with correct locale?

Of course.

Would be if you know how will be the new Unicode implementation, which encoding will be used by default, etc.

Right.

Ok, I understand. Thanks.

Regards,
Marcos Douglas

Graeme Geldenhuys wrote

Hi Marcos,

Hi Graeme,

I talked about in thread "SysToUTF8/UTF8ToSys".
It's a pain to code a huge system, with many packages on Windows, because I always have to think about what encoding to use depending on which part of the code (eg: RTL is AnsiString, LCL is UTF-8, Synapse is AnsiString, some own packages are AnsiString too, two are UTF-8, etc.)

Using fpGUI do you need to thinking about the encode or do you use UTF-8 at all? You need to use functions, at all over code, like SysToUTF8?

^^
I'm thinking get out this thread -- and others about "how the FPC team will implement the Unicode" -- because this is not worth. If I can help FPC or Lazarus team, I will. If not, well, I will be make my own "RTL" coding to not suffer (much) in the future.

Regards,
Marcos Douglas

I never said not work. I said "how to make a code on Windows _safe_, readable and _strong_".
[[ IMHO ]], use functions at all over the code to convert strings is not safe -- according to my criteria -- and, if is not safe, isn't strong. But I can be wrong.

If you get my first mail, I only ask: "Is there some trick..."

Now a clean Unicode implementation cannot be given to you because it does not exist.

Ok, thank you.

You could have read the same things from FPC mailing list archive. They have been repeated again and again and again in many many threads, some of which went on for months, during past ~ 4 years.

Ah, Ok! I'm the wrong person now?? I'm the ONLY person talking about "Unicode, the forbidden topic"?
Why don't you talk like that for ALL people that said something about it?

Besides you already hijacked other threads for the same discussion. This one was originally about "Beyond Compare 4".

Me? See the history again.

I learned something. Maybe others learned something too. This worth for something.
But Okay, let's give peace for all.

Regards,
Marcos Douglas

Marcos Douglas wrote

I learned something. Maybe others learned something too. This worth for something. But Okay, let's give peace for all.

Sorry for my harsh words.
I realize I could skip reading all the Unicode threads but somehow they are addictive, maybe because I must work with Unicode conversion myself in future.

Juha

That is exactly the point I was trying to make:
it is not necessary too mention encoding issues at every function.

Michael.

One receipe is to use function wrappers for all relevant ones (without knowing fpGUI much I think this may be what Graeme did):

1. Write your own string functions wrappers.
2. Use only your wrapper functions in your code.
3. If something changes in the future adapt or rewrite your wrappers, not the complete source of your programs.

It's the same thing I personally do for floating point types. For testing I need to have the ability to switch between single and double or other types. So I define my own general basic float type and switch it:

type
  //basefloat = single;
  //basefloat = double;
  basefloat = extended;

String are a little more work, but if your wrapper functions are inlined there will be no problem, even if they do anyhow, Marc

Marcos Douglas wrote

because I always have to think about what encoding to use depending on which part of the code (eg: RTL is AnsiString, LCL is UTF-8, Synapse is AnsiString, some own packages are AnsiString too, two are UTF-8,

OK, I see what you mean - when interacting with 3rd party packages.

Using fpGUI do you need to thinking about the encode or do you use UTF-8 at all? You need to use functions, at all over code, like SysToUTF8?

fpGUI only uses UTF-8 internally, so developers using fpGUI simply use the functions fpGUI supplies - instead of the FPC RTL ones. eg: use fpgFindFirst instead of FindFirst. All such functions are the RTL names, but with a 'fpg' prefix, so makes it easy to remember. They are all in a single unit too, the fpg_utils.pas unit.

fpGUI also tries to hide the fact of what Unicode encoding is being used used, that way it has methods like fpgFromOSEncoding() or fpgToOSEncoding() instead of SysToUTF8() etc. I think the fpGUI ones are easier to remember. NOTE: these two methods will only be used if interacting with 3rd party packages... fpGUI handles the RTL for the developer already. And if some RTL method isn't covered, I'll add it quickly - as it is very simple to do.

Regards,
- Graeme -

Creating your own set of routines and classes is very easy, but now comes the real question: From what I understand the new Unicode FPC will be a one way ticket to the Delphi solution, which means that TStrings, TComponent, and all other classes will only work with the new string type. So what will you do then? Your current solution just like the current LCL solution will not work. There are only two solutions:

1> Stop using the RTL and write your own
2> Migrate to the new system

Which way will fpGUI choose?

This is not correct. We'll provide 2 RTLs:

one where string = unicodestring
one where string = ansistring.

This means existing code will function as it used to function if you choose the second one.

Michael.

Michael Van Canneyt wrote

This is not correct. We'll provide 2 RTLs: one where string = unicodestring one where string = ansistring.

wow ... it will probably be a lot of work and maintenance work to keep all packages working with both RTLs.

Only the cases where you do low-level PChar access or Move() data into a string, need to be checked. These cases need to be checked anyway if we would move to a 2-byte char.

All the rest should be more or less transparant, actually.

Michael.

Only the cases where you do low-level PChar access or Move() data into a string, need to be checked. These cases need to be checked anyway if we would move to a 2-byte char.

What about binary data in strings (buffers)? Will FPC supply a dedicated encoding, that will exclude such strings from automatic conversions, or must the Delphi way be used (TBytes)?

DoDi

As far as I know, TRawByteString must be used for such cases.

Abusing ansistring for arbitratry data was a bad idea in the past. Using TBytes would have been a good idea in any case.

Michael.

Not so much if you compared to when you add up all these discussions :-)

I think that will go the Delphi way in the end. It should have been done directly anyway.

Felipe Monteiro de Carvalho wrote

1> Stop using the RTL and write your own 2> Migrate to the new system Which way will fpGUI choose?

I was always under the impression the new FPC will have a Unicode and AnsiString RTL.

Either way, nothing stops me from making my methods (eg: fpgFindFirst) to simply be an alias to a Unicode RTL method. I can't see why that wouldn't work. Plus that would mean all existing fpGUI project will continue to work, and the fpg* prefixed ones could be marked deprecated in fpGUI.

Once a Unicode FPC goes into beta, I'll start evaluating my options for fpGUI.

Regards,
- Graeme -

Michael Van Canneyt wrote

These cases need to be checked anyway if we would move to a 2-byte char.

Does that now mean the FPC's Unicode RTL actually means UTF-16?

ps:
1) So what will the size of Char be in bytes?

2) What will the size in bytes be of ThousandSeparator and DecimalSeparator be? eg: fpGUI had to introduce its own versions of these, with a 4-byte max size (as per UTF-8 spec) because some locales require a 3-byte UTF-8 value.

Regards,
- Graeme -

It means string = unicodestring, so yes.

BUT

All file open routines already exist as UTF-16 or single byte variants today in trunk. That will not change.

So where Delphi changed the system unit so it has only the equivalent of

Procedure Assign(Var F : Text; AFileName : UnicodeString);

FPC will have (in fact, it already has in trunk) overloaded versions:

Procedure Assign(Var F : Text; AFileName : UnicodeString);
Procedure Assign(Var F : Text; AFileName : AnsiString);
Procedure Assign(Var F : Text; AFileName : RawByteString);

So depending on the string type you use, the 'correct' one will be chosen.

ps: 1) So what will the size of Char be in bytes?

in the unicode RTL: 2

In the unicode RTL they will be UTF-16 characters, so 2 bytes.

Michael.

2 bytes - are you sure? What about code points that don't fit in 2 bytes? Won't they be supported?

First find a ThousandSeparator and DecimalSeparator that would actually need this. Then we'll see.

Idle theorizing is fine for mathematics and theoritical physics (huge fan of those), but not here :)

Michael.

Well, I'm not a fan of paying lip service to specs while ignoring them, because those things can come back and bite you, but I don't particularly care as it seems any unicode [1] support will take years to materialize anyway ;)

[1] the standard, not the Delphi imple... etc

Oh, and ROFLMAO. Thanks :)

Michael Van Canneyt wrote

Idle theorizing is fine for mathematics and theoritical physics (huge fan of those), but not here :)

That sounds like someone in the eighties saying:
Do you realy think that this software will be used beyond year 2000? Two digits for the year is enough.. ;-)

I always stumble over encodings and the documentation is of no help. Another example I found recently:

I wanted to display error messages and
found that SysErrorMessage should give me what I want. It is said to be "cross platform", so I thought the encoding is the same (UTF8) as for all other "cross platform" functions (otherwise why is it called "cross platform"?). But only by accident I found out that the encoding cannot be UTF8 because it did not display all characters correctly (which only becomes apparent when german umlauts appear so it could go wrong for a long time until it this bug is discovered because errors should not happen very often and even if they appear not all contain umlauts).
So I took a look into the documentation to find out about the encoding:

http://lazarus-ccr.sourceforge.net/docs/rtl/sysutils/syserrormessage.html

There it says:

---------------------
function SysErrorMessage(
ErrorCode: Integer
):;
SysErrorMessage returns a string that describes the operating system error code ErrorCode. ---------------------

Yet another documentation bug?
What kind of string is given back?
But even if it would say "string" it would be of no use for me. Which encoding?
How should a user find out what he has to do in his program to display a correct string in a ShowMessage() or whatever? This is a real mess.
Actually I found out by trial and error that

       ShowMessage(Trim(SysToUTF8(SysErrorMessage(GetLastError))));

is required to get a correct string (Trim is needed because trailing spaces and tabs may appear).

I would expect that for all these functions where stings are involved at least a link to the encoding is added that tells the user that he will get back an AnsiString with System-ANSI-encoding (or whatever he will get back on other OS's). How else should a user find out about this?

J?rgen Hestermann wrote

This is a real mess.

Yes, but there is no need to repeat it again and again. As I replied to Marcos Douglas, this same discussion and complaining has continued in FPC mailing lists for ~ 4 years. Everybody SURELY knows there are problems with Unicode.
There is no easy solution so it takes time, but the developers already have a valid solution baking.
Let's be more constructive.

ShowMessage(Trim(SysToUTF8(SysErrorMessage(GetLastError))));

LazUtils has SysErrorMessageUTF8() which is identical to your wrapper except for Trim() call.
BTW, trailing spaces and tabs are not part of the character encoding problem. Lazarus offers a working UTF-8 solution.
If you need a solution now, please use it, otherwise wait for the coming FPC solution.

How else should a user find out about this?

By learning once that a "string" currently means AnsiString with System-ANSI-encoding.
Michael Van Canneyt also tried to explain it to you. Once you know it, the documentation does not need to repeat the same thing always when "string" is used.
It also does not explain "integer" every time an "integer" is used.

Juha

All appearances of string were 'annihilated' in the 2.6.2 release due to a bug in fpdoc.

What kind of string is given back? But even if it would say "string" it would be of no use for me. Which encoding?

Ansistring = The system encoding.

Simple logic would have told you this in about 30 seconds.

Simple: an object pascal programmer should know that

String = Ansistring

Or he must stop programming Object , from this, BY DEFINITION, it follows that the encoding is system encoding.

The FPC 2.6.x RTL uses ansistring everywhere and therefor supports system encoding only.

It's really this simple rule:

String = Ansistring => System encoding.

Michael.

Thank you. I was beginning to become very desperate :-)

Michael.

Michael Van Canneyt wrote

First find a ThousandSeparator and DecimalSeparator that would actually need this. Then we'll see.

Several locales on OS X use U+00A0 no-break space as a thousands separator, which is $C2$A0 in UTF-8, so it doesn't fit in a single AnsiChar. We had to add a lossy conversion to $20 to preserve the appearance, if not the semantics, of it. There was a thread on the Mac-pascal mailing list a few months back about it.

"Craig Peterson" wrote

On Dec 31, 2013, at 3:19 AM, Michael Van Canneyt

> First find a ThousandSeparator and DecimalSeparator that would actually

need this.

> Then we'll see. Several locales on OS X use U+00A0 no-break space as a thousands

separator, which is $C2$A0 in UTF-8, so it doesn't fit in a single AnsiChar. We had to add a lossy conversion to $20 to preserve the appearance, if not the semantics, of it. There was a thread on the Mac-pascal mailing list a few months back about it.

As Michael wrote the locale specific constants will become UnicodeChar (2 Byte UTF-16) thus the U+00A0 will fit nicely there. The only problems would arise if such a constant would need to be composed of multiple UnicodeChars (codepoints?).

Regards,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20131231/d81c6389/attachment.html>

Juha Manninen wrote

As I replied to Marcos Douglas, this same discussion and complaining has continued in FPC mailing lists for ~ 4 years.

It seems there is a reason for this.

> ShowMessage(Trim(SysToUTF8(SysErrorMessage(GetLastError)))); LazUtils has SysErrorMessageUTF8() which is identical to your wrapper

How should I know?
It is not mentioned in the documentation for SysErrorMessage. (I know these are different projects but this doesn't make it less annoying). When searching for "Error Message" I only find SysErrorMessage but not SysErrorMessageUTF8. Otherwise I would have used it.

I cannot (and do not want to) ask in this forum for each and every function whether alternatives exist.
I was hoping for a valid documentation about them with a "see also" rubric. There are so many libraries around each repeating similar functionality (though with slight differences) that it becomes hard to get an overview. I as a former Turbo Pascal/Delphi user tend to forget that everything is split into multiple independent projects which all use their own standards.

BTW, trailing spaces and tabs are not part of the character encoding problem.

True. I never said this.

Lazarus offers a working UTF-8 solution. If you need a solution now, please use it, otherwise wait for the coming FPC solution.

It is much easier for me to use the WIN-API directly and write my own wrappers. Searching for existing solutions seems to require more time and still often does not fit my needs.

> How else should a user find out about this? By learning once that a "string" currently means AnsiString with System-ANSI-encoding.

Yes, I now remember why I abandoned the use of FPC internal functions that use STRING years ago:

ANSI = WIN-ANSI-API = max path length 255 and non-Unicode.

So completely useless today.
I have to remind myself to avoid functions that use STRING (if I recognize them).

Michael Van Canneyt also tried to explain it to you. Once you know it, the documentation does not need to repeat the same thing always when "string" is used.

Well, meanwhile I am so accustomed to use UTF8 that I find it hard to remember that STRING still means this old ANSI encoding type. It is also not easy to find out that functions use STRING because this word has been wiped out in the documenation.

It also does not explain "integer" every time an "integer" is used.

Bad enough.
A link to an explanation would help so many people. But it seems the documention is only for those people who already know everything about FPC/Lazarus.

Sven Barth wrote

As Michael wrote the locale specific constants will become UnicodeChar (2 Byte UTF-16) thus the U+00A0 will fit nicely there. The only problems would arise if such a constant would need to be composed of multiple UnicodeChars (codepoints?).

No. A codepoint is a 32-bit number. UTF8 and 16 encode these in shorter space only.

A character is one more codepoints. (usually a main codepoint, and codepoints for accents)

Still it might not be a bad idea to make such things a string instead of a character in time.

Thanks for clarifying the terms. I hope I remember them till the next Unicode discussion :P

Still it might not be a bad idea to make such things a string instead of a character in time.

Might need adjustment inside the corresponding routines though as they might expect to only work with one Ansi-/UnicodeChar.

Regards,
Sven

Marco van de Voort wrote

Still it might not be a bad idea to make such things a string instead of a character in time.

Hence in fpGUI I have a TfpgChar defined as a 4-byte string, and can easily be extended if needed.

Regards,
- Graeme -

Why that? Why not simply a variable length string?

What is needed here is a chart explaining what the options are and the use cases where they apply. This is an area most developers don't understand their code breaks in some way. Without use cases simply quoting standards and how this or that library or API does this or that won't be of any help.

A chart explaining where they have gone wrong and what options should apply will be more useful than repeated arguments about what is wrong or what right or wrong paths FPC, Lazarus, Delphi or Microsoft have taken.

The experts in this area will have to switch to educator mode rather from developer mode if these arguments are going to end.

Marco van de Voort wrote

Why that? Why not simply a variable length string?

Because I was under the impression a ShortString (eg: String[4] declaration) is faster than AnsiString, and lighter on resources.

Regards,
- Graeme -

This might bite you with 2.7.1 as ShortString is considered as being encoded in CP_ACP. Thus on systems were the system code page is not UTF-8, but the values are UTF-8 characters this might lead to unexpected conversions if you assign to an UTF8String or UnicodeString, because the compiler will insert a conversion from CP_ACP to
UTF8String/UnicodeString which will convert your UTF8 character to something completely different.

Regards,
Sven

Sven Barth wrote

This might bite you with 2.7.1 as ShortString is considered as being encoded in CP_ACP. Thus on systems were the system code page is not

Thanks for pointing that out, and I've made a note to test this when the time comes. As previously mentioned, fpGUI only uses UTF-8 internally, and has fpg* specific functions for working with RTL and custom string functions. So when there is a String[4] or AnsiString type used somewhere, it is what some here consider a "AnsiString hack".. ie: an array of bytes representing a UTF-8 string, but stored in an AnsiString container. When TfpgChar or TfpgString is used, the framework knows it contains UTF-8 encoded text.

All this will be revisited and tested once the FPC 2.8.0 (or 3.0) release gets closer to realisation.

Regards,
- Graeme -

Is there a way to ignore all these and make everything to work with UTF-8? Like setting some global variable that makes all strings (ansistrings) "UTF-8 codepage" or something that also causes any system-specific calls (Assign, Writeln, etc) to do automatic conversion from/to whatever the underlying OS uses?

Kostas Michalopoulos wrote

Is there a way to ignore all these and make everything to work with UTF-8? Like setting some global variable that makes all strings (ansistrings) "UTF-8 codepage" or something

We will have to wait for FPC 2.8.0 (or 3.0) which should have much better built-in Unicode support. String encoding conversion should then be taken care of automatically. Unfortunately it seems that the FPC RTL (there will be two of them) will be AnsiString or UTF-16 only. The RTL encoding is not configurable!

So under all Unix-like systems (Linux, MacOSX, FreeBSD - basically every platform except Microsoft ones) there will be lots of string conversions from/to the OS or any libraries (which are normally UTF-8) to the FPC RTL which is going to be UTF-16. The constant conversion will also kick in when you do streaming to/from file or any TCP/IP communications - which both normally use UTF-8.

I would have thought the Free Pascal team would improve their design over Delphi. eg: Seeing that automatic encoding conversion is seamless, I thought it shouldn't be hard to have native encodings on each platform, and the RTL can then be a dynamic Unicode implementation (it shouldn't care what encoding is used, as long as it is one of the Unicode encodings). By that I mean UTF-8 is used under Unix like systems, and UTF-16 under Windows. The UnicodeString type should have lived up to its name, and not be an alias for UTF16String. But alas, this is not going to happen.

So we as developers have to use UTF-16 everywhere, or define our own dynamic types (which really should have been done at RTL level). For example:

  {$IFDEF Unix}
   RealUnicodeString = UTF8String;
  {$ENDIF}
  {$IFDEF Windows}
   RealUnicodeString = UTF16String;
  {$ENDIF}

Then use the RealUnicodeString type in your applications and frameworks to minimise encoding conversions. But like I said, when you do this under Unix like systems, you are still going to get conversions when talking to the UTF-16 only RTL. Sad, but that is the way the Free Pascal team is going.

Once that FPC release is made, then we will start seeing what performance impact it will have on all systems. Now is too early to tell.

Regards,
- Graeme -

+1

You'll be able to set a variable that will control the default codepage. This variable can be set to utf-8. That should do what you want.

Michael.

+1

You always said this, ie, UnicodeString should be UTF-8 on Unix plataform and UTF-16 on Windows, and I always agreed with you. This make sense, this would be a true UnicodeString type. Delphi is the only "trouble" to do this happens?

Should be so:

=== BEGIN ===
{$IFDEF Unix}
    UnicodeString = UTF8String;
{$ENDIF}
{$IFDEF Windows}
    UnicodeString = UTF16String;
{$ENDIF}
// the alias
string = UnicodeString;
// the automatic conversions
function UnicodeToUTF8(const S: UnicodeString): UTF8String;
begin
  {$IFDEF Unix}
      Result := S;
  {$ENDIF}
  {$IFDEF Windows}
      Result := UTF16ToUTF8(S);
  {$ENDIF}
end;
function UnicodeToUTF16(const S: UnicodeString): UTF16String;
begin
  {$IFDEF Unix}
      Result := UTF8ToUTF16(s);
  {$ENDIF}
  {$IFDEF Windows}
      Result := S;
  {$ENDIF}
end;
=== END ===

Maybe we are not seeing something, many details...

Regards,
Marcos Douglas

That is great, i *really* don't want to think about different encodings between OSes and such and if i can make a "UTF-8 everywhere" flag (where everywhere means LCL, FCL, RTL, Windows, Linux, OS X, etc :-P) it would be perfect for me :-)

Marcos Douglas wrote

Maybe we are not seeing something, many details...

Agreed.

Regards,
- Graeme -

So that's why you use the ansistring RTL there, unless you value Delphi compatibility. On those targets (except a few embedded Linux and OpenBSD ones), the default encoding is utf8 and any ansistring is utf8, case closed.

The constant conversion will also kick in when you do streaming to/from file or any TCP/IP communications - which both normally use UTF-8.

Please stop with your baseless fearmongering.

That's irrelevant, for sources that don't have a compiletime known codepage (if only CP_ACP aka "default"). Those are bytebuffers with an encoding described by headers, and thus fall outside the scope of automated conversions.

I would have thought the Free Pascal team would improve their design over Delphi.

We have consistently chose Delphi compatibility for years, so why you are thinking this is fairly strange.

You are overfixating on the word "unicode" in unicodestring.

The real important one is string.

And avoid calling any -A functions on Windows.