Hello,

Some things weren't clear from the previous discussion, so I would like to clarify them.

For instance, the GetTempFileName routine:

http://www.freepascal.org/docs-html/rtl/sysutils/gettempfilename.html

The routine is currently ANSI, but we need a unicode version of it. How would that unicode version look like? We currently have 3 unicode string types planned AFAIK: UTF8String, WideString and the new variable encoding type (how is that called?).

I assume that the new variable encoding type would be used for all unicode routines, am I right?

Or would versions for all 3 types be added? (for example, if someone donates utf8 routines).

thanks,

No.

I assume that the new variable encoding type would be used for all unicode routines, am I right?

No, it will be RTLString which type depends on the OS.

Florian Klaempfl wrote

No, it will be RTLString which type depends on the OS.

Ok, so code would be something like this:

var
  OSString: RTLString;
  MyString: UTF8String;
begin
  OSString := SomeRTLRoutine;
  MyString := OSString;

?

It will be funny to use a string type about which nothing is known. I wonder if people will abuse this and start operating system dependent code.

Why would you do this and not
MyString := SomeRTLRoutine;
?

Florian Klaempfl wrote

Why would you do this and not MyString := SomeRTLRoutine;

You are right, that should do it. I was thinking about var parameters.

If I understand that right, this may cause some overhead, that in some(few) cases is not needed.

If I write an application using stringtype "X" (WideString for example), then in the above "MyString" would be WideString.

The in/ouput for SomeRTLRoutine are RtlString, they are OS depended. If I compile for a OS using "UTF8" then that means for each and every call, it needs a string conversation.

Of course I understand, *if* some RTLFunction calls the OS, then the string must be converted. But if I simply want to extract the drive letter, or trim the path, and get the file name, without actually accessing the file or OS? Should it be possible to skip converting?

Best Regards
Martin

In our previous episode, Martin Friebe said:

Correct.

If I write an application using stringtype "X" (WideString for example), then in the above "MyString" would be WideString.

Correct.

The in/ouput for SomeRTLRoutine are RtlString, they are OS depended. If I compile for a OS using "UTF8" then that means for each and every call, it needs a string conversation.

Correct.

Of course I understand, *if* some RTLFunction calls the OS, then the string must be converted. But if I simply want to extract the drive letter, or trim the path, and get the file name, without actually accessing the file or OS? Should it be possible to skip converting?

Use rtlstring. Do the conversion to widestring after.

IOW, you should do it the other way around. Use the OS dependant stringtype for mostly encoding independant operations, and only the few things where you need specific encodings force a certain encoding (using utf8string or widestring)

.

I agree, using RTlString will probably help fpc to optimize your exe for each OS.

But, using RTLString means you do not know, if you have UTF8 or not. Because UTF8 behaves slightly different from other Strings, many operations can not be performed on RTLString

foo[1], copy, pos ... simply because you do not know, if the result is a char, a codepoint or a subcodepoint (single utf8 byte)

RTLString is or will be great, if you simply need to store an OS depended string in order to later give it back to the OS. (eg open file, remember file name, but do not process it (displaying it would be vi OS), and save file back to the same name.)

For this you could also use ByteString: if there is such a thing, and if it behaves as "not converting, if assigned to any string"

Best Regards
Martin

---
Disclaimer: Just to keep this discussion where it was:
- I do understand why the above is as it is (string index not being utf8 chart access).
- I do not believe that this is correct too (and any discussion should be a new thread)

In our previous episode, Martin Friebe said:

Correct.

You don't know that about UTF-16 either. Even though that is no problem in most cases, it is slowly time to abandon too simplistic thinking about strings. The best solution is to minimize editing, and localize them in certain parts of the code, keeping most of the code encoding agnostic.

And yes, if you lazy, you lose performance due to automatic conversions. It has always been that way (also when mixing short and ansistring)

This is not just a good thing for OS interfacing code, but a good thing in general.

True, good point

most cases, it is slowly time to abandon too simplistic thinking about strings. The best solution is to minimize editing, and localize them in certain parts of the code, keeping most of the code encoding agnostic.

True, too. But we are talking Pascal, not some other language. string[index], copy, pos, length have always been part of Pascal.

Of course they are still there, to be used in the few parts of your code, where you specialize on whatever string type you deal with. But otherwise, using RTLString IMHO will abandon this part of pascal syntax. A function of which the result can not be used, as it can change at compile time => such a function can not be used. (or we will have buffer overflows, code injection and more ...)

I admit that the Problem started (and that has been discussed more than enough) starts with UTF8string (yes even with utf16 string). But in this case those functions became a new, but predictable meaning. I can do utf8string[1], and I can use the result. Only I have to be aware what it means.

I can *not* do rtlString[1], as at the time of code writing I can not be aware what it means. It is only decided, at compilation time. IFDEFs won't help neither, because they can only cope with the set of stringtypes know at the time the code is written. This breaks each time FPC will be extended.

and localize them in certain parts of the code, keeping most of the code encoding agnostic.

Sorry I can't help taking that into another direction, (which also has been discussed before). The above quote sounds like a sentence from a introduction into "object orientation". Sure it is the right thing.. It is right for OO. So it should be right for strings as well. Just again, it simply will be a new language, which a string-object, instead of pascal.

And yes, if you lazy, you lose performance due to automatic conversions. It has always been that way (also when mixing short and ansistring)

In other words, write pascal code, just do not use some of the (imho) most common elements of pascal syntax?
I acknowledge a language is a living thing, and needs to be adjusted to the new things, that come up over time. I only ask, if this is the best way?

We're open to proposals, make one.

In our previous episode, Martin Friebe said:

So keep using ansistring? It doesn't change.

Of course they are still there, to be used in the few parts of your code, where you specialize on whatever string type you deal with. But otherwise, using RTLString IMHO will abandon this part of pascal syntax.

It removes ASCII legacy. I don't see you complaining about the fact that char is not 8 bit anymore, and that that abandons that part of the pascal syntax.

A function of which the result can not be used, as it can change at compile time => such a function can not be used. (or we will have buffer overflows, code injection and more ...)

Hence my suggestion to minimize this functionality.

Yes. As widestring[1] also requires interpretation. That's unicode.

I can *not* do rtlString[1], as at the time of code writing I can not be aware what it means.

You don't have to. You carry it around as long as you can, and when you don't can, you assign it to your type of choice and bite the penalty.

Delaying that as long as possible avoids excessive penalities, which IMHO are as much part of the Pascal language. Doing that would hurt the general purpose nature by turning into basic. (and then I mean the real Basics, not the C-with-basic-syntax that is FreeBasic), or worse: Excel.

It is only decided, at compilation time. IFDEFs won't help neither, because they can only cope with the set of stringtypes know at the time the code is written. This breaks each time FPC will be extended.

Any such big transition as ASCII -> Unicode will break. However we have had these discussions before, but avoiding all pitfalls is simply too costly, and that breaks other Pascal traditions.

It is an introduction to abstraction maybe. I don't see the OO in there.

It is right for OO. So it should be right for strings as well. Just again, it simply will be a new language, which a string-object, instead of pascal.

This is all gibberish for me. I never said OO, and never will.

There is no "just". Strings simply get more complicated if you go unicode, and that can't be hidden. Either you stay with safe ASCII strings, or you use Unicode. If you do the latter, you will have to adapt anyway.

And top-heavy emulation layers are not Pascallike either.

I acknowledge a language is a living thing, and needs to be adjusted to the new things, that come up over time. I only ask, if this is the best way?

IMHO there is not even a choice, since there simply no is a viable alternative.

Well, RTLString will have it's place, same as something like RawByteString would. It sounded to me, that this will be the only way of communicating with the RTL.

My opinion is that it should be the programmers choice. I a programmer wants or needs a simpler way (keeping all the strings in is application in one format, which will be known to him) then he/she should have that choice. And then on this type the person could perform any index or index-like operation.

That would mean that in order to avoid conversation, some functions of the RTL would be needed in overloaded versions for each string type. IMHO this applies only to those, which do not (or not always) make calls to the OS. Any other function does the conversation anyway. (It will be a case by case base)

It is nothing urgent, as it can always be added to the RTL. In the end this is an optimization, not a feature.

All the code
  Widestring := RtlFunction;
  Utf8string := RtlFunction;
will run, it may just perform badly.

And that is my point. Not everyone using pascal, is an expert. Pascal is also an attractive language to beginners. And not every bit of code will be written for a bg application; some may go into a small tool, for a very specific may be valid reasons to not keep every string in your code as RTLString.
I believe FPC should do a good job for this kind of code too?

Anyway. I know "Patches are welcome" .... Unfortunately at the moment I will have to keep it to contribute ideas.

Also it would be nice (so I do not know how) not to have to duplicate code, in order to archive this. Something like generics, maybe.

Best Regards
Martin

That would mean that in order to avoid conversation, some functions of the RTL would be needed in overloaded versions for each string type.

- people complain already now about bloat
- you can only overload by parameter type not by result
- you cannot overload types in records, classes etc.

It does not abandon the syntax. It only adds to it's meaning (*adds*, any existing meaning is unaltered.).

I can still do: foo[1] for *any* type of string. (well yes even RTLstring, but see below)
- If string happens to be an old ascii string, that still works as it always has
- If string happens to be any unicode => that is still the same syntax, but with a new meaning.
The new meaning doe snot break anything, because it only applies to new types.
It is usable too, because I know, I am dealing with codepoints, or sub code points. And I know how they look, and how to identify them

The introduction of RTLString is fine. I do say it is a good thing. RTLString does not interfere with the above. In fact even for RTLstring the syntax foo[1] does exist. Just it is not useful. If I tread it as utf8 sub code point, I can be wrong. If I tread it as ascii, I can be wrong. If I tread it as UTF16 I can be wrong

My argument was not against RTLString. However it was my understanding that RTL functions will "enforce" RTLString. That they will only exist for RTLString, and they will *not* exist for other string types. That I would call enforcing RTLString, because of penalties on using other string types.

I acknowledge, that if the end result of calling the RTL function, is an OS call, the conversation/penalty is always there. But not every RTL function ends up in an OS call.

See above: Yes it requires interpretation. But it allows me to do so

I can not see how I can interpret RtlString[1]. If the result is bigger than 128, then I must know what type it is. If it is ANSI, it is a single byte char. If it is utf8, it is a sub-codepoint which will be part of a codepoint.
If it is widestring, well yes, here breaks my assumption that RtlString[1] returns a byte.... ouch

As I said in another mail. Every programmer starts as a beginner. And for many of those this is the last thing to think about.

Best Regards
Martin

Not true if fpc will follow Delphi. The new AnsiString type will be also automatically converted in Delphi 2009. See the Marco Cantu doc about Unicode (linked some threads ago).

Luiz

I see this as a theoretic consideration. Please give a real world (!) code example when this causes a problem.

If you assign the result of an rtl function to an rtlstring, this means you don't care about the type of rtlstring[1] or the knowledge, that it's type is rtlchar is enough for you. If you assign it to an ansistring/widestring whatever, you know what you get.

To use safely RTLString, at first look, would be be sufficient to use overloaded functions from the Characters unit (introduced in Delphi 2009). See http://www.jacobthurman.com/?p=30 how you can use them to get Copy, Pos behavior.

Next week, i'll implement those functions for UTF16 and UTF8 and do some tests.

Luiz

And yes, if you lazy, you lose performance due to automatic conversions. It has always been that way (also when mixing short and ansistring)

Of course you are very right here !

If you are lazy and write your code like you are used to, you will not get optimum performance with a new compiler that now allows for Unicode. But the code still needs to be working as expected (as with a compiler version that does not allow for Unicode, but simply uses ANSI or whatever OS and locale depending 8-Bit code).

In most programs that will not be a problem at all as doing extensive string calculations in user-code is not necessary.

Of course, if you want to take real advantage of Unicode (using characters outside your current locale) or if you want to optimize (for speed or for memory size) you need to be aware of the Unicode stuff and write your code appropriately.

-Michael

From: "Michael Schnell"

It is planned to allow users to build ANSI version of RTL which will be fully compatible with existing user code.
But if you choose to use unicode RTL, you must keep in mind all unicode specific things...

Yury.

So keep using ansistring? It doesn't change.

Only if the bytes in the ANSIString in fact are ANSI (which the compiler in the moment) does not take care for if doing myANSIString := myUTF8String etc.

I feel that with Widestring the pos() etc paradigms stay usable in more cases than with ANSIString.

-Michael

I don't see you complaining about the fact that char is not 8 bit anymore, and that that abandons that part of the pascal syntax.

When doing the most common string stuff like
case s[i] of
'1', 'a', 'ä':
...

This does not really hurt.

even
n := ord(s[i]) - ord('0'); works with 16 bit/char strings.

-Michael

It is planned to allow users to build ANSI version of RTL which will be fully compatible with existing user code. But if you choose to use unicode RTL, you must keep in mind all unicode specific things...

This will be very helpful for the time being. Let's hope that the LCL will follow the Path of allowing the user to choose if he actually wants to use Unicode in the user code without explicitly calling Unicode functions.

-Michael

It is not helpful because on an utf-8 system ansistring contains utf-8. Ansistring just means: use the system locale 8 bit encoding.

Martin Friebe wrote

All the code Widestring := RtlFunction; Utf8string := RtlFunction; will run, it may just perform badly.

Yes and no.

Let's assume the platforms windows and unix having UnicodeString (UTF-16) and UTF8String as native types respectively. You choose to use UnicodeString type in your app.

Using the rtlstring approach you get:

Under windows: the native string type of platform is the same as you are using no conversion is taken. Good.
Under unix: the native string type of platform is NOT the same as you are using ONE conversion is taken. Bad.

Now let's assume that fpc team decided to use a fixed unicode encoding for the RTL. Let's say a UnicodeString RTL.
You choose to use UnicodeString type in your app.

Under windows no conversions. Everything is UTF16. Good. Under unix the RTL must internally convert from the native type (UTF8) to UTF16. Bad.

The same result as above.

But someone else wants/needs to use UTF8 strings in your project.

Under windows you will get one conversion: UTF16 -> UTF8. Bad. Under unix you will get TWO conversions: UTF8 -> UTF16 -> UTF8. Very Bad.

Luiz

Can you give a real world example where a different RTLString for each platform solves a problem?

If you assign the result of an rtl function to an rtlstring, this means you don't care about the type of rtlstring[1] or the knowledge, that it's type is rtlchar is enough for you. If you assign it to an ansistring/widestring whatever, you know what you get.

What string type will be TStrings.Items and the many other strings in the classes.pp?

Mattias

It solves for example the problem that there are platforms where no unicode support is available or desired and it avoids unneeded conversions. I'd be fine using utf-16 on all platforms :)

Not yet decided though I'd make them RTLString as well.

In our previous episode, Luiz Americo Pereira Camara said:

As far as I know, the default is still "ascii in the default system ascii encoding".

See the Marco Cantu doc about Unicode (linked some threads ago).

I got it from Alan Bauers blog in may (before Tiburon was out), but while ansistring changes, afaik the widestring to ansistring-without-qualifier stays the same?

About: keeping all the strings in is application in one format, which will be known to him

Only small programs can do that. All others use third party packages. If you want choice, then all used third packages must support all possible choices. Unlikely.

Sorry, I can't follow here.
Please enlighten me, why an overloaded function with an internal conversion is better than an implicit conversion?

[...] Also it would be nice (so I do not know how) not to have to duplicate code, in order to archive this. Something like generics, maybe.

The goal of RTLString is to avoid duplicate code in the RTL.

Mattias

In our previous episode, Mattias Gaertner said:

It avoids pingpong repeated conversions between OS encoding and whatever encoding is default.

IMHO rtlstring.

Florian Klaempfl wrote

> I assume that the new variable encoding type would be used for all > unicode routines, am I right? No, it will be RTLString which type depends on the OS.

The more I think about it the more I like this solution. I think it's better then the previous idea of a string with encode information inside it.

In our previous episode, Mattias Gaertner said:

This is not possible, since you don't control OS + headers. Most stuff will come from the outside in the system encoding.

This way you can do the whole app in the system encoding, and only face conversions when outputing to the GUI, which is (relatively) infinitely slow anyway.

You did nail a big problem though, and a weakness in Delphi's design. What to do with classes that are used both straight and in the GUI?

Only small programs can do that. All others use third party packages. If you want choice, then all used third packages must support all possible choices. Unlikely.

If you want to be the lowest common denomitor and ask nothing from the 3rd party packages, it is the best to stay with Ascii.

Marco van de Voort wrote

You did nail a big problem though, and a weakness in Delphi's design. What to do with classes that are used both straight and in the GUI?

You mean like TStrings?

I think we will eventually roll our own TUTF8Strings

We could add a unit in FPC for all kinds of UTF-8 versions of routines.

Florian Klaempfl wrote

> What string type will be TStrings.Items and the many other strings in > the classes.pp? Not yet decided though I'd make them RTLString as well.

I think you can't change TStrings because that would break all code using it (huges amount of code).

I would recommend adding a similar class with a different name.

:)

and it avoids unneeded conversions.

I understand it 'avoids unneeded conversions' *inside* the RTL, by adding implicit conversions to the code accessing the RTL.

I'd be fine using utf-16 on all platforms :)

Me2. At least for the file functions.
I have some doubt about the classes.pp.

:(
TStrings is dog slow and the only reason, why it was still reasonable was assigning strings was only reference counting. If TStrings uses a platform dependent string, this is a big performance problem.

Mattias

A real world example please.

Mattias

In our previous episode, Felipe Monteiro de Carvalho said:

Doesn't work per se. Tstringlist is also used in libraries, to save GUI parts etc.

A better solution would be to simply not try to fix this and give lazarus their own copy of said classes, so that they can keep the encoding of that in sync with whatever they decide for their own encoding.

That would actually require less fixups (a few conversions procedures for the rare points where tlclstrings are passed to e.g. registry units. Lazarus already has their own XML units).

In that case, I would recommend giving it the "string with attached encoding" style type so you don't need 5 tstrings variants.

Regarding how to deal with file system representations, conversions etc, it may also be interesting to look at Apple's NSString class (http://developer.apple.com/documentation/Cocoa/Reference/Foundation/Classes/NSString_Class/Reference/NSString.html ) or, if you prefer a procedural approach, CFStrings (http://developer.apple.com/documentation/CoreFoundation/Reference/CFStringRef/index.html )

I'm not suggesting to mimik that exact API, but only to see what kind of APIs they support (and are deprecating). NSString/CFString (one is just an OOP version of the other) are also a "universal string container" type, with embedded encoding.

For example, there are routines such as
* CFStringGetCharacterAtIndex() (and more optimised approaches as documented there, such as CFStringGetRangeOfComposedCharactersAtIndex())
* CFStringGetFileSystemRepresentation() (basically the "rtlstring" version of the string)
* CFStringConvertWindowsCodepageToEncoding() ("Returns the Core Foundation encoding constant that is the closest mapping to a given Windows codepage identifier.")
* ...

The advantage when using such a type is that you also only need to convert it (internally, hidden from the user) on demand or when some helper routine requires it (such as e.g. case-insensitive comparisons). Otherwise, no conversion whatsoever is necessary.

Jonas

This must be added to the classes.pp and TStrings must know it, so that Assign et al works.

We could add a unit in FPC for all kinds of UTF-8 versions of routines.

Yes, that is a good idea anyway - independent of RTLString and the current topic. Perhaps this should be discussed in a separate thread.

Mattias

In our previous episode, Felipe Monteiro de Carvalho said:

Depends. The few last msgs kept me thinking, and if what I saw on the web about Tiburon is correct, they simply control the type of ansistring in tstringlist, and default let it be the system encoding. (default like in old delphi). For unicodecontrols they set some new property or so to change it to UTF8, and take the conversion penalties for granted.

This allows them to do

  MyStringList.SaveToFile('SomeFilename.txt', TEncoding.Unicode);

It has to be something like that, since if mystringlist always was ansistring in whatever ISO encoding, that would be a pretty pointless unicode control.

In our previous episode, Jonas Maebe said:

(nsstring)

The advantage when using such a type is that you also only need to convert it (internally, hidden from the user) on demand or when some helper routine requires it (such as e.g. case-insensitive comparisons). Otherwise, no conversion whatsoever is necessary.

Do they have some way to indicate that a procedure/method only supports a certain encoding? Or do you have to manually force the encoding in that way?

I prefer a declarative way to solve this.

In our previous episode, Mattias Gaertner said:

It allows the user to stay conversion free, and have some control over how many conversions are being done.

It is way better than making this decision for him, and forcing him to an encoding he normally wouldn't use in the first place.

> I'd be fine using utf-16 on all platforms :) Me2. At least for the file functions.

I would too. If all platforms had chosen it. But they didn't.

Because exactly why? See also my previous msg.

Only at places where we had the choice. The LCL uses the FCL xml units.

Compatibility was always the bigger goal for lazarus. IMHO a TLCLStrings breaks more than it would solve.

Mattias

In our previous episode, Mattias Gaertner said:

I don't have to. If you guaranteedly have two encodings into place and combine code out multiple sources this will happen for sure.

Keeping as much as possible in the system encoding, will limit this only to legacy packages that haven't been properly ported yet, and at least a hope staying away from converison hell.

No.

Or do you have to manually force the encoding in that way?

Yes.

I prefer a declarative way to solve this.

In the Pascal case, you could simply declare your parameter as UTF8String (or whatever) and the compiler would insert a conversion from this universal string type into a utf8string.

Jonas

Mattias Gaertner wrote

Compatibility was always the bigger goal for lazarus. IMHO a TLCLStrings breaks more than it would solve.

I don't fully understand how the Tiburon TStrings works, but consider that we are used to mixing TStrings with LCL code, and then we migrate to the proposed UTF8String.

Now every assignment of a string to TStrings will have a implicit conversion.

Unless there are many overloaded methods in TStrings, one for each encoding, and it is able to internally use our desired encoding so that no useless convertions are done.

If this requirements aren't met, we need a new class.

In our previous episode, Mattias Gaertner said:

As said it can be fixed. Maybe even easier than I thought.

Compatibility was always the bigger goal for lazarus. IMHO a TLCLStrings breaks more than it would solve.

A lot will change. Even with Delphi not everything automatically is unicode, and they only have one platform to regard.

I usually am sb who is pretty serious about Delphi compatibility, except for some of the more bizar post D7 experiments. This however is different. While I really like Tiburon as Delphi user, I loathe it as FPC user.

It will never be totally transparent, whatever you do. Just like something things never were transparent when porting to Linux.

We are a multi OS compiler, not a version of Wine oriented towards Pascal development.

This is exactly what I was thinking. The conversion is simply passed on to a different piece of code. So the end result is the same - you still have conversion.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Mattias Gaertner wrote

Compatibility was always the bigger goal for lazarus. IMHO a TLCLStrings breaks more than it would solve.

You mean compatibility with Delphi?

With Tiburon I think this will become very hard, if can, however, keep compatible with previous Delphi versions.

In our previous episode, Graeme Geldenhuys said:

Not necesarily, since you might not use a different type. Or only use them in a few rare cases where you must do character level access.

Or you might convert to UTF-32, because the char level access is particularly difficult, or you want to be correct.

IOW, you give the programmer the choice about the type, instead of forcing him an arbitrary one, based on your favorite platform.

Marco van de Voort wrote

In our previous episode, Graeme Geldenhuys said: IOW, you give the programmer the choice about the type, instead of forcing him an arbitrary one, based on your favorite platform.

This is the part I like about this approach. The most likely fixed encoding to be adopted would be UTF-16, and something not very nice would happen to Lazarus users in UNIXes:

LCL (UTF-8) ------> RTL (UTF-16) ---> Operating System (UTF-8)

2 useless conversions.

In our previous episode, Jonas Maebe said:

Clear. I just wondered how they solved it.

> I prefer a declarative way to solve this. In the Pascal case, you could simply declare your parameter as UTF8String (or whatever) and the compiler would insert a conversion from this universal string type into a utf8string.

I know. With that modification I thought that was the best too until Tiburon details emerged.

Actually I still think that our original is the best, not considering compatibility issues, but I don't think the difference is worth losing at least base level Tiburon compatibility over. Specially because their solution has some advantages too (can more gradually change ansistring code)

In our previous episode, Felipe Monteiro de Carvalho said:

Btw will the LCL remain forcedly UTF-8 ? I thought the current Lazarus unicode support was temporary and all options were still open, depending on the outcome of FPC unicode support options?

Marco van de Voort wrote

Btw will the LCL remain forcedly UTF-8 ? I thought the current Lazarus unicode support was temporary and all options were still open, depending on the outcome of FPC unicode support options?

It is certainly not temporary, also considering people won't be very happy to see us make a big incompatible change right after telling them to convert their source code to UTF-8. I think we have a responsability to stay coherent here.

I think we may consider migrating to UTF8String when it's implemented and if it proves a viable solution.

Not to mention: What would the alternative be?

The LCL is a visual component library. It's string speed is slow anyway. Except maybe for TMemo.Lines and running through a big TreeView.

Same is true for file functions. The OS overhead checking for permissions and the other file system issues makes even a triple encoding/decoding a non issue. For example under Mac OS X the lazarus IDE uses the CFString functions to compare a filename. This normalizes the string each time. CompareFilenames is used easily hundred thousands of time and no one said yet, that the IDE runs slowly under OS X.

It's a different thing for TStrings. Many algorithms need the O(1) time accessing a Items[i].

Mattias

In our previous episode, Felipe Monteiro de Carvalho said:

Well, euh, you will need a change anyway from manual to automated ?

I think we may consider migrating to UTF8String when it's implemented and if it proves a viable solution. Not to mention: What would the alternative be?

Well, the logical one of course:

RTLString, IOW encoding platform dependant. Except maybe selected widgets like synedit. (Borland stores source in utf-8 too on windows)

En/na Felipe Monteiro de Carvalho ha escrit:

LCL (UTF-8) ------> RTL (UTF-16) ---> Operating System (UTF-8)

Is the last step always true? Doesn't qt support utf-16?

Bye

Luca Olivetti wrote

> LCL (UTF-8) ------> RTL (UTF-16) ---> Operating System (UTF-8) Is the last step always true? Doesn't qt support utf-16?

This is for operating system calls, not graphical library calls.

Marco van de Voort wrote

RTLString, IOW encoding platform dependant. Except maybe selected widgets like synedit. (Borland stores source in utf-8 too on windows)

A string whose encoding is unknown is very inconvenient for developers. The idea just saves itself in the RTL because of the eventual need to do some extremely high performance applications. For Lazarus it would be simply a useless inconvenience.

So how did people work for years with ansistring?

In our previous episode, Felipe Monteiro de Carvalho said:

I don't see that so strongly as most.

The idea just saves itself in the RTL because of the eventual need to do some extremely high performance applications. For Lazarus it would be simply a useless inconvenience.

Same as above. That is an opinion, not fact, and I don't agree.

It is btw not just about performance, but also about predictability. Less encodings in use, means better preditability.

If RTL+LCL are in the system encoding (with LCL mostly hiding odd ball libs as QT if that is the widgetset), you have a fair chance not to have a multiencoding app, without thick layers of emu. Or at least keep the encoding dependant part localised to a fairly small part of the program.

To be honest, I think a case for LCL follows widget set encoding could also be made.

In our previous episode, Florian Klaempfl said:

Depends on country I guess. Here one simply skips accents, except for a few apps/fields where e.g. "own names" (of persons, cities etc) are used. At least until recent years.

Florian Klaempfl wrote

So how did people work for years with ansistring?

A ansistring used in the way proposed by FPC is extremely inconvenient for any GUI application which will be run in different parts of the globe. You develop a application in a russian machine, sends it to a english machine and it shows rubbish instead of text. Even if you actually could read that russian GUI.

It introduces dependency of what will be shown at runtime with the operating system you are running it. It's exactly the mess Unicode was invented to end with.

People worked for years with ansistring suffering from it's short comings.

I never suggested the RTL to be in a fixed encoding. I fully agree that this would be far worse.

I suggested to have a rtl, that has overloaded functions for each string type.
of course that sounds easier than in fact it will be.

Florian pointed out a few issues, like overloading by result is not possible (yet?). And code duplication would be a maintenance hell. But those limits can be overcome. Maybe not in full for the first Unicode fpc release.

I can see that in order to get at least something (and in a way forward compatible) to all the waiting users of fpc, the RTLString solution is a good solution (or compromise: full function, limited optimization).

The functions that can be overloaded with what fpc already has, could be written for the various types. Maybe a template system for plain functions (like generics for objects) could be found? So code would not be duplicated.
Maybe fpc could be extended to allow overloading by result? (sure that has other uses too?)
It's just a suggestion.

Best Regards
Martin

Martin Friebe wrote

I suggested to have a rtl, that has overloaded functions for each string type. of course that sounds easier than in fact it will be.

This is about the same as having all string routines in 3 flavours: RTLString, utf-8 and utf-16

the utf-8 and utf-16 could be done by assigning rtlstring to the adequate type.

I think this is probably what we will end up with, because users of a particular encoding will build convenience routines for their favorite RTL routines.

I meant more that a lot of people simply ignored in their code that ansistrings could be also multibyte even not considering UTF-8.

No, it is fundamentally impossible. First of all, because a Pascal compiler does not look at how the result of an expression is used while parsing the expression (so while evaluating a function call, the compiler has no idea what the "expected" result type is). That's simply how Pascal works.

And how could the compiler decide what to call here:

function f: longint;
begin
   result:=1;
end;
function f: string;
begin
   result:='s';
end;
begin
   writeln(f);
end.

There are an immense amount of similar examples possible (and if you add operator overloading in the mix, it increases exponentially), and you'd get a huge amount of "can't decide which overloaded function to call" errors even if the compiler were hacked to take the use of the function results into account.

Jonas

Let the programmer define a scoring function (to be called by the compiler when it has multiple options) that takes two functions and decides which combination is most efficient or whatever ;-).

Micha

I made a difference between rtl function which will call OS functions (the one that would need to internally convert the format) and those that do not make OS calls (or do not always make them)

Yes I agree, and said so before: If a rtl function is going to pass on the data to the OS, and conversation is always needed, then no overloading is needed. Use RTLString.

If an function does not use the OS (e.g extract file-path or name) then no internal conversion is needed. Therefore overloaded functions would give benefit to some programmers. (to name one group: most beginners, who have more than enough to worry about; and will be glad if strings are kept simple to use ( a fixed known type, not a type that looks somehow abstract, because you do not know its implementation at the time you write your code)

Yes I acknowledged that this would be a problem. And also RTLString has the benefit, of allowing unicode fpc to be available far earlier than it would otherwise.

The question remains, could it then be extended/optimized? Maybe a generic like template (for functions, instead of objects)? Which needs to be written only once, and the will be specialized for each string type?

Best Regards
Martin

That I agree with. A string with encode information in it (I read that as a type-data field at runtime ) is not good (IMHO)

Best Regards
Martin.

I never needed an argument for the introduction of RTLString. It's a great idea. It allows a programmer at hi/her choice to make applications much more portable.
Even so, in fact it is not new. You could always just have introduced your own type, and set it OS depended with a few IFDEFs

I argue against a RTL that *only* provides a RTLString interface. Instead of various overloaded methods (see other mails, for which set of functions/methods this actually makes sense).

I have however so far got good arguments for starting with RTLString solution.
- a solution with many overloads would defer release
- overloads (currently) lead to code duplication (so that needs to be solved first)
- overloads are not possible based on function results

But on the long term (fpc 2.6 or maybe 2.8) surely those issues can be overcome (if wanted)?

The doc i was referring is: http://dn.codegear.com/article/38980

Some quotes:
"The change in the definition of the Char type is important because it is tied to the change in the
definition of the string type. Unlike characters, though, string is mapped to a brand new data
type that didn't exist before, called UnicodeString. As we'll see, its internal representation is
also quite different from that of the classic AnsiString type (I'm using the specific terms classic
AnsiString type, to refer to the string type as it used to work from Delphi 2 until Delphi 2007; the
AnsiString type is still part of Delphi 2009, but it has a modified behavior, so when referring its
past structure I'll use the term classic AnsiString)."

"AnsiString is a single-byte-per-character string type based on the current code page of the
operating system, closely matching the classic AnsiString of past versions of Delphi;"

"RawByteString is an array of characters with no code page set, on which no character
conversion is accomplished by the system (thus partially resembling the classic AnsiString, when
used as a pure character array)."

"
THE NEW ANSISTRING TYPE
Differently from the past, the new AnsiType string carries one further piece of information, the
code page of the characters in the string. The DefaultSystemCodePage variable defaults
to CP_ACP, the current Windows code page, but it could be modified by calling the special
procedure, SetMultiByteConversionCodePage. You can do this to force an entire
program to work (by default) with characters in a given code page (that the operating system
installation must support, of course).
In general, instead, you'd either stick to the current code page or change it for individual
strings, calling the SetCodePage procedure (introduced earlier while talking about characters
and code pages). This procedure can be called in two different ways. In the first case, you
change the code page of a string (maybe loaded by a separate file or socket) because you know
its format. In the second case, you can call it to convert a given string (something that happens
automatically when assigning a string to one of a different code page, as discussed later).
"
Luiz

Marco van de Voort wrote

> A string whose encoding is unknown is very inconvenient for > developers. I don't see that so strongly as most.

I know I spoke only generally, but it's hard to speak about and foresee the effects of something which currently doesn't exist (a cross-platform library using a string type whose encoding is unknown). Everyone out there has choosen an encoding. wxWidgets went with utf-16, Gtk+ went with utf-8. Who is choosing a string with unknown encoding?

I see only some cases where the string is a class, but those are very different from what is proposed.

It is btw not just about performance, but also about predictability. Less encodings in use, means better preditability.

I think it's the other way around. If you know what encoding is expected you have a more predictable result. You know where a conversion will take place. For example:

MyUTF8String := MyRTLString;

So we get an error that characters are being lost, but only in Windows ... ummm, it turns out there was a problem with the conversion routine in the RTL.

To be honest, I think a case for LCL follows widget set encoding could also be made.

It was already investigated that the conversion time is negletible compared to the paint time.

Using RTLString increases the problems for developers, because they need to identify when they need to do something which require knowing the encoding and increases the size of the code to add the conversion. Remember that we are expecting to build software in a RAD way.

So in one hand you have no substantial gain and in the other some annoyance.

If you have a function overloaded in utf6 and utf16, then You will not need to have it in RTLString. RTLString will be equal to either of the 2?

Martin Friebe wrote

If you have a function overloaded in utf6 and utf16, then You will not need to have it in RTLString. RTLString will be equal to either of the 2?

But it's easier to implement like I proposed, and it seams some people want to have RTLString for higher performance.

I don't see, how a TLCLStrings will *not* break Delphi and Lazarus compatibility. Maybe you can give some more details, how it should work.

With Tiburon I think this will become very hard, if possible at all. We can, however, keep compatible with previous Delphi versions.

Before duplicating base classes I would rather like to find a good compromise for the base classes.

Mattias

Point taken.

And point also taken that it's not a solution to to a "half job", meaning overload only methods based on parameters, but not results. This would be a false pretence, ending people n a believe they do the right thing where they do not.

Another idea:
The programmer should be able to set himself what RTLString means, (with the default being the OS depended). It isn't really a nice solution though. But I throw it into the discussion anyway.

Otherwise I''ll have to see, If I can come up with something else.

It's variable: no conversion, one conversion, two conversions. Depends of the system and the user string type. Say the RTL is changed to UTF16.

If the user chooses to use UTF8String in unix you will have 02 conversions: UTF-8 -> UTF-16 -> UTF-8

Luiz

Mattias Gaertner wrote

I don't see, how a TLCLStrings will *not* break Delphi and Lazarus compatibility. Maybe you can give some more details, how it should work.

It was just a initial idea. I now see that TStrings could be improved.

I asked this once and no one did an affirmative. I hope this gets clear soon, to avoid problems with code of third part LCL developers (like me).

Luiz

True.

Yes.
I wonder, how many are there that really need this? Can you estimate?

Mattias

On Mon, Dec 1, 2008 at 8:34 PM, Luiz Americo Pereira Camara

I asked this once and no one did an affirmative. I hope this gets clear soon, to avoid problems with code of third part LCL developers (like me).

I'll start a discussion about this in lazarus-dev.

You mean how many rttl functions would benefit. I do not know. But I do think it's the wrong question.
In order to do it you would want something like function-templates (generics equivalent for functions).

I do not believe such a template system would be developped for the sole purpose of doing the above rtl work.

If however such a template system would be implemented anyway (because it would benefit everyone in the same way generics do (actually I never checked: can generics do simple functions or only classes?)), so if it was there, then why not use it for the rtl too? In such a scenario does it matter if you optimize 5 or 5000 functions?

Will not get lost. UTF8 holds all unicode range. It will be converted from UTF-16 to UTF 8

Not necessarily, the compiler would select the adequate conversion or not convert. Says you have a function that expects a UTF8 encoding and you set that parameter as UTF8String. Just do MyUTF8 := MyRTLString. Automatically in windows you get a conversion, under unix no.

Remember that we are expecting to build software in a RAD way. So in one hand you have no substantial gain and in the other some annoyance

Initially i'm not in favor of using RTLString, IMO LCL should choose an encoding UTF16 or UTF8 and then let the compiler convert only the system calls.

Here's the conversions we would got:

LCL as UTF8

OS-WidgetSet Conversion LCL <-> WidgetSet Conversion LCL <-> System calls
----------------- --------------------------------- -----------------------------------
Windows-win32: YES
YES
Windows-gtk2:
NO YES Windows-Qt:
YES YES Linux-Gtk2:
NO NO Linux-Qt:
YES NO

LCL as UTF16

OS-WidgetSet Conversion LCL <-> WidgetSet Conversion LCL <-> System calls
----------------- --------------------------------- -----------------------------------
Windows-win32: NO
NO
Windows-gtk2: YES
NO
Windows-Qt: NO
NO
Linux-Gtk2: YES
YES
Linux-Qt: NO
YES

LCL as RtlString

OS-WidgetSet Conversion LCL <-> WidgetSet Conversion LCL <-> System calls
----------------- --------------------------------- -----------------------------------
Windows-win32: NO
NO
Windows-gtk2: YES
NO
Windows-Qt: NO
NO
Linux-Gtk2: NO
NO
Linux-Qt: YES
NO

Luiz

Exactly.
If RTLString is several encodings this will happen for sure. And worse: It will happen more often depending on library and platform.

Keeping as much as possible in the system encoding, will limit this only to legacy packages that haven't been properly ported yet, and at least a hope staying away from converison hell.

This depends on the apps.
You can optimize for one encoding or optimize for one per platform. I know how to optimize for widestrings, for ansistring and for UTF-8 strings, but I have no experience in optimizing for multiple encodings. Maybe someone who did that can give some advice. The real world problem is (as Florian wrote): Some platforms have no choice, because they only support 'ansi'.

I welcome the RTLString type. And I don't care what encoding is used for slow RTL functions and I trust that you have more experience than me to find a good solution here.
But for TStrings I think multiple fixed encodings will make more trouble than Jonas' (dynamic encoding) and Lazarus' (one encoding for all platforms) ideas.

Mattias

Luiz Americo Pereira Camara wrote

Initially i'm not in favor of using RTLString,, IMO LCL should choose an encoding UTF16 or UTF8 and then let the compiler convert only the system calls.

The complete phrase: "Initially i'm not in favor of using RTLString, IMO LCL should choose an encoding UTF16 or UTF8 and then let the compiler convert only the system calls. But there's seems some advantage of RtlString regarding necessary conversions."

Luiz

The real world problem is (as Florian wrote): Some platforms have no choice, because they only support 'ansi'.

This one is trivial to solve. Lazarus supports utf-8 everywhere, including Windows 98 which doesn't support unicode. All you need is a best effort approach.

Op Mon, 1 Dec 2008, schreef Felipe Monteiro de Carvalho:

This is possible if you implement a Unicode support implementation in Pascal (i.e. uppercase, comparison, conversion). But there is more in the world than Lazarus. Forget about Win98. Do you really want to fit the huge tables required for this into for example, a Game Boy Advance? Doesn't fit, this machine has 32+96+256KB RAM.

Daniël

In our previous episode, Felipe Monteiro de Carvalho said:

Yes, for the core routines. It is nuts to make stuff like scandatetime in two different encodings.

In our previous episode, Micha Nelissen said:

Yes, that is how script languages solve it. However in anything designed for larger programs, this is less usefull, since only one error can result in endless debugging.

I never suggested the RTL to be in a fixed encoding. I fully agree that this would be far worse.

I suppose there are (quite decently workable) solutions for this. Either the RTL (and LCL, FWIW) comes in multiple versions that are used as appropriate (user selectable and/or automatically selected), or a string type is used that knows about it's internal coding and conversions are dynamically done when appropriate.

-Michael

This is about the same as having all string routines in 3 flavours: RTLString, utf-8 and utf-16

What about (real) ANSIString (OS/locale based coded) ? This needs to be allowed as the program might need to read such files.

-Michael

Btw will the LCL remain forcedly UTF-8 ? I thought the current Lazarus unicode support was temporary and all options were still open, depending on the outcome of FPC unicode support options?

I understand they could not do it differently (other than just providing no Unicode support at all), maybe as there is no automatic type conversion support with FPC (e.g. ANSIString<->UTF8String).

As the current version is not very satisfying I do hope for a change, but I also do understand that the Lazarus team will wait for what the next FPC version offers on that behalf.

-Michael

Not to mention: What would the alternative be?

Even if I am not satisfied with the current state of Lazarus on that behalf, I would not dare to suggest that Lazarus should do any change here before the next version of FPC offers a new string handling with either a string type that knows it's internal coding and with that any conversions can be done automatically, or with multiple string types defined of which the compiler knows how to convert them if necessary, or with whatever solution the FPC team comes up with.

-Michael

I meant more that a lot of people simply ignored in their code that ansistrings could be also multibyte even not considering UTF-8.

Ignoring that ANSI Characters > $7F are locale depending makes a program work perfectly in a single country and mostly decently in many others.

Ignoring that ut8-code-points can be coded in two code-elements in an "ANSIString" makes a program work only in countries that use just ASCII, Thus not in at all in Europe.

-Michael

You can optimize for one encoding or optimize for one per platform. I know how to optimize for widestrings, for ansistring and for UTF-8 strings, but I have no experience in optimizing for multiple encodings.

Don't forget that the ansistring type is actually multiple encodings and even multi byte (even not considering UTF-8). The point is: nobody took care of it.

So it works in far east with its multi byte ansi encodings?

Maybe we should make such classes simply generics: using wrappers as the map and list class already do, the size impact shouldn't be that big.

Don't forget that the ansistring type is actually multiple encodings and even multi byte (even not considering UTF-8). The point is: nobody took care of it.

IMHO a major confusion is generated by calling a string that is supposed to hold UTF8 data "ANSIString". This never should have happened ! If the Unicode support requires that there are strings that hold ANSI code and those that hold UTF8 code they should be denoted correctly as ANSIString and UTF8String. Storing ÚTF8 in an ANSIString is a sin :). IMHO, not providing automatic conversion between these type is a major shortcoming of the compiler/RTL and if it not does so, it should not provide the types. (Which does not mean that providing the (best possible) automatic conversion between these type solves all problems !)

-Michael

Nobody talks in this case about UTF-8. Even *ANSIstrings* in there native meaning can contain multi byte chars, there are *multi byte* ansi char sets. However, everybody codes 1 char=1 byte when using ansistrings which is plainly wrong. Guess why Delphi has functions like CharToByteIndex or NextCharIndex.

It is not helpful because on an utf-8 system ansistring contains utf-8. Ansistring just means: use the system locale 8 bit encoding.

I meant it would be helpful to (optionally) have a complete ANSI based system (as was the previous version of Lazarus and FPC).

I don't understand what you mean by "an utf-8 system ansistring contains utf-8". The LCL/RTL in the ANSI based system would provide the appropriate conversions for the user code. UTF8 coded System Strings (hopefully type "UTF8String" and not "ANSSrting" :) ) would only appear in user code if the user directly does low level system interface calls. Here of course (s)he needs to know what (s)he is doing anyway.

-Michael

The more I think about it the more I like this solution. I think it's better then the previous idea of a string with encode information inside it.

Would Lazarus be able to follow ?

Do you think it's possible to have the compiler take care of any necessary conversions automatically ?

-Michael

These of course are not part of the "many others". (BTW handling Russian with ANSI is not easy at all, we already have been there :) ).

But those that do software for "far east" are not likely to ignore multi-byte characters in utf8 coded strings (as the OP suggested).

-Michael

The point is: if everybody takes care of the fact that ansistrings can be multibyte, having utf-8 in ansistrings (if it's the locale encoding), is no big deal at all.

Nobody talks in this case about UTF-8. Even *ANSIstrings* in there native meaning can contain multi byte chars, there are *multi byte* ansi char sets.

If there is a widely used multi-byte ANSI encoding, why so we need Unicode ?

IMHO the introduction of Unicode has been necessary as (like you suggested) multi-byte ANSI encoding was commonly ignored nearly completely and there never has been _compiler_ support for them. Thus IMHO it's quite appropriate to only call "ANSI" only the 1-Byte ANSI code versions (to be able to tell them technically from Unicode, the compiler support of which is discussed right here).

-Michael

So, really? What is not supported?

Thus IMHO it's quite appropriate to only call "ANSI" only the 1-Byte ANSI code versions

... tell this 1+ Billion (Billion=10^9 in this case) people in China.

(to be able to tell them technically from Unicode, the compiler support of which is discussed right here).

The point is: if everybody takes care of the fact that ansistrings can be multibyte, having utf-8 in ansistrings (if it's the locale encoding), is no big deal at all.

I do understand. But (in a real world) do you know anybody who does.

If it would be "appropriate" for ANSI code handling to take care of Multi-byte encoding we would not need locale-based code tables and en effect Unicode would not have been invented.

-Michael

Multibyte ansi chars are still not unique and require the code page for proper interpretation, this is why Unicode has been invented. By using properly functions like ChartoByteIndex or NextCharIndex, it makes very likely no difference for string processing code if the strings are multi byte ansi or utf-8 unicode.

Op Tue, 2 Dec 2008, schreef Michael Schnell:

What compiler support should be necessary to handle i.e. EUC-JP? You want a variable of type char to contain the JIS-0213 coordinates?

Unicode, and in particular UTF-8, has not taken off either because languages got support for it. In fact, the most common language, C, has no string reason Unicode has taken off because of document exchange, which in the internet age got very common. Another reason is the growing importance of the Far East, developers want therefore better support for the Far East languages, but note this Unicode motivation exists mainly for Western software developers.

Daniël

So, really? What is not supported?

If just ignoring the fact is enough support, OK, it's supported :).

... tell this 1+ Billion (Billion=10^9 in this case) people in China.

I did not know (or suppose) that code used for Chinese characters is called "ANSI" ("American National Standards Institute").

I supposed one of the main intentions for the move to Unicode was the ability to support Chinese above all. So they did not seem to have been content with what was done before.

Is anybody from China here to offer the footage ?

-Michael

> So, really? What is not supported? > If just ignoring the fact is enough support, OK, it's supported :).

What FUD is this? Pleaes give an example where the FPC compiler doesn't handle multi byte ansistrings properly.

Or do you just want to troll around? This problem can be solved ...

Sorry for bad language :( ! I did not mean to be aggressive. (Did you see the smile indicator ?)

I did not suggest it handles this wrong in any way, but I just don't see in what way there might be any explicit compiler support for multi-byte ANSI. (You did mention the RTL function provided.)

I understand that there never has been a discussion on if there should be any explicit compiler support for multi-byte ANSI, but this thread _is_ a discussion on explicit compiler support for Unicode.

-Michael

Op Tue, 2 Dec 2008, schreef Michael Schnell:

It is not the Chinese that are pushing for Unicode. GB2312 suits them well. Likewise, JIS 0213 suits the Japanese well. Those encodings also have the characters to support Western languages, or Greek, or Russian. Unicode is used as a technical solution to handle Chinese is for an important part a Western development. Eastern developers already support Eastern languages.

Daniël

It simply needs no explicit support except what it has already. Mainly the rtl and the user program has to take care of it and we did this already in the rtl but the compiler required no fix in this regard so far.

For me, these attempts to make compiler do everything automatically sound like getting yet another typing saver. The situation is already dangerously close to "write once, debug forever". Recently (after Lazarus 0.9.26 release)
I had encountered some cases when a trivial function call resulted in a couple
of conversions inserted silently and resulting outcome could be explained only
by tracing it or looking at the assembler code.
Another example is issue #11327. Initially I perceived it as a code generation issue, but after digging in it was clear that it's caused by first choosing an incorrect overloaded function, then inlining it, then attempting to optimize. There are already at least 17 overloaded Pos() functions, and the compiler simply gets lost between them. That issue is likely to be fixed, but the fix will be for the consequences of the problem, not for its origin.

Making the conversions automatic does not make the language clean, instead it hides the potential errors and author's intentions. Moreover, it forces anyone to (implicitly) use these conversions, even those who don't need it.

A notable fact is also that while all these endless speech about "lack of Unicode support in compiler", nearly all well-known Unicode processing software is written in languages that have no built-in support not only for Unicode, but for the strings itself.

Regards,
Sergei

Zitat von Florian Klaempfl :

Yes, they did. They ran their programs only on systems with ansi encoded strings or simply passed the strings unchanged.
That's why the lazarus solution even work with broken UTF-8 strings. But now a lot of implicit conversions will be added so all strings must have valid encodings. You can no longer pass unknown encoded strings through the functions.

Mattias

It simply needs no explicit support except what it has already. Mainly the rtl and the user program has to take care of it and we did this already in the rtl but the compiler required no fix in this regard so far.

I do see your point !

But my point is that with the introduction of Unicode, compiler support for handling of these things is introduced (and the RTL and the LCL). I think this should result in making user-code largely unnecessary and not in requiring those programmer, that did not need multi-byte support for serving the users they want to deploy their software to, to finally start to introduce multi-byte handling in their user-program code.

IMHO, if ever possible, a new version of a program should make life for the majority of the actual users easier and avoid making life more complicated for those that not willingly decide that they need the complexity.

-Michael

Thanks for pointing this out.

GB2312 suits them well. Likewise, JIS 0213 suits the Japanese well.

Are these called "ANSI" ?

-Michael

Zitat von Michael Schnell :

UTF-8 is unicode and it is the system encoding on linux, OS X, some BSDs and Solaris. So ansistrings are UTF-8 there.

Mattias

Thanks for pointing this out. > > GB2312 suits them well. Likewise, JIS 0213 suits the Japanese well. Are these called "ANSI" ?

Every well educated windows programmer knows that the ansi functions/strings whatever are not limited to the so-called "ansi" code pages (which aren't ansi either afaik) or is CP850/CP1252 as on my german windows an ansi code page ;)?

First, there will be a bytestring type being not converted. Secondly, I'am rather sure we find ways to cut these conversions as much as possible down.

For me, these attempts to make compiler do everything automatically sound like getting yet another typing saver.

Maybe I am just being lazy, but it's not a typing saver but regarding the previous not-Unicode aware versions it's more a "preventer of a typing enhancer" :) .

OTOH it's not just the typing but to work with commonly used things that "just work" in other programming systems (including previous versions of FPC/Lazarus) - like doing a "case" of a character type - the user programmer needs to learn about the internal encoding of Unicode text. I think this should be avoided. Pascal has been a great language for programming newcomers up till now. Simple things - like characters and strings - should just work (unless you explicitly need extended handling).

I don't suggest that there is a simple solution for this (other than not doing Unicode at all) but it's worth discussing.

-Michael

Op Tue, 2 Dec 2008, schreef Michael Schnell:

Thanks for pointing this out. > > GB2312 suits them well. Likewise, JIS 0213 suits the Japanese well. Are these called "ANSI" ?

Yes, code page 936 and code page 932 are valid ANSI code pages.

These standards by themselves of course not, because they are a Chinese respective Japanese industrial standard.

Daniël

UTF-8 is unicode and it is the system encoding on linux, OS X, some BSDs and Solaris. So ansistrings are UTF-8 there.

I still don't understand what ANSI has to do with "System".

AFAIK, The term "ANSI Code" stands for a (codepage depending) definition for a character encoding and Unicode is another one. Both are independent of Operating systems.

-Michael

I now understand that GB2312 and JIS 0213 in fact are the ANSI code pages 936 and 932.

-Michael

Michael Schnell wrote

I still don't understand what ANSI has to do with "System".

Ignore the name "ansi". Take it as a string type with the system encoding. I think it will solve the confusion.

Zitat von Michael Schnell :

... and some things that "just don't work" like i18n.

the user programmer needs to learn about the internal encoding of Unicode text. I think this should be avoided.

Tell the unicode consortium. My guess: they know already.

IMHO it has been already discussed too often.

Mattias

Felipe Monteiro de Carvalho wrote

Ignore the name "ansi". Take it as a string type with the system encoding. I think it will solve the confusion.

Of course if you ignore "ANSI" and just use the type named "String" there is no confusion as it's clear that the coding is not predefined.

That is exactly what I wanted to say: If you don't use it for ANSI coded information don't name the type "ANSIString". As FPC provides the type "ANSIString" out of the box it should be used appropriately and this any new user will suppose that there is support for conversion between this type and other string types that explicitly are called differently according to their suggested internal coding (such as UTF8String.

If it does not work that way this calls for major confusion.

IMHO it has been already discussed too often.

I did not start it and only 1% of the contributions are mine - and yours -, so quite obviously there is a decent common wish for a solution of what is percept as a problem.

-Michael

Really? Pascal is a strongly typed language which avoids automatic conversion between types as much as possible.

that there is support for conversion between this type and other string types that explicitly are called differently according to their suggested internal coding (such as UTF8String.

avoids automatic conversion between types as much as possible.

I feel that it's a goody of a strongly typed language that automatic type conversions can be done by creating the appropriate code statically instead of having this embedded in the objects as with variants.

If doing a simple assignment "a := b;" types are either converted
appropriately or a compiler error is generated.

All integer and real types are converted automatically. If you try to do "myInteger := myString;" you get a compiler error. But if you do "myANSIString := myUTF8String;" the compiler generates an assignment without a conversion, even though the types are provided by the system (not by the user) and named according to the possible internal coding. (We don't need to discuss why this is like that, the discussion is about if it should stay this way.)

-Michael

The type is called ansistring simply for backwards compatibility.

You could start arguing that everything should be intuitive. Take C for example. What does the && operator tell you about what it does? Shouldn't it have a intuitive form? But in the end this is how the language is and this is a useless discussion.

I don't think that all C programmers will rewrite their code anymore then pascal programmers will rewrite theirs so that you can find a better name for ansistring.