Hello.

Yesterday there was a commit:

http://svn.freepascal.org/cgi-bin/viewvc.cgi?view=rev&root=lazarus&revision=32374

"don't use UTF8String type in the code. It will cause problems when cpstrenew branch is merged"

What problems could it cause, is it because Lazarus currently uses AnsiString as UTF8? I thought Lazarus is going to continue using UTF8 and then it shouldn't be a problem using UTF8String in it or in its applications. I would rather think that all instances of AnsiString would need to be removed because the compiler would do Ansi<->UTF8 conversions where it previously didn't.

We are using UTF8String in our application just for the sake of denoting that it is UTF8 encoding not Ansi. If I would like to start testing FPC trunk with the merged cpstrnew branch, should I replace all UTF8String with String?

cobines wrote

"don't use UTF8String type in the code. It will cause problems when cpstrenew branch is merged"

Lazarus must use either UTF8String everywhere or nowhere.

We are using UTF8String in our application just for the sake of denoting that it is UTF8 encoding not Ansi. If I would like to start testing FPC trunk with the merged cpstrnew branch, should I replace all UTF8String with String?

I don't know how you use it.

Best regards,
Paul Ishenin.

2011/9/19 Paul Ishenin :

Lazarus must use either UTF8String everywhere or nowhere.

I thought that Lazarus will continue to use UTF-8 on all platforms and String will mean String<65001> and it will be interchangeable with UTF8String. Is that not so?

Why change UTF8String to AnsiString and not String, like almost every other string parameter?

Why not apply the same to AnsiString and change all to String since Lazarus does not work with Ansi code pages anyway?

For example, if UTF8ToUTF16 was left to accept UTF8String I would think it would force the parameter to have UTF-8 code page, which would be more correct. And this is what I don't understand, how will it break when UTF8String is left.

I don't know how you use it.

To remind developer that a function uses UTF-8 not Ansi so he needs to call SysToUTF8, UTF8ToSys or similar. All strings in Lazarus are UTF-8 anyway so this is only for show.

Will it use or won't it does not matter. If in one place you use UTF8String and in other AnsiString the compiler will convert that UTF8String in one place to default system codepage in another. For example on my windows it converted UTF8 string to 1251 codepage. Having all strings in default codepage prevents from automatic conversion.

Why change UTF8String to AnsiString and not String, like almost every other string parameter?

The argument was UTF8String. UTF8String previosly was declared as type AnsiString in RTL. Why should I choose a "string" type which depends on compiler switches and source code modes.

Why not apply the same to AnsiString and change all to String since Lazarus does not work with Ansi code pages anyway?

Lazarus works with strings which have 1 byte per element. If FPC later switch default string type to UnicodeString Lazarus will suddenly get many problems.

For example, if UTF8ToUTF16 was left to accept UTF8String I would think it would force the parameter to have UTF-8 code page, which would be more correct. And this is what I don't understand, how will it break when UTF8String is left.

Compiler adds implicit codepage conversion for string arguments. I had to avoid that. The better choise would be to use RawByteString type but I it is not defined in fpc 2.4.4 which we need to support.

Best regards,
Paul Ishenin.

Thank you, I think I understand it now. Currently in Lazarus String = AnsiString and by default it has DefaultSystemCodePage. If I use UTF8String type than it is AnsiString(CP_UTF8). So, to avoid conversions, which Lazarus does not want, I also need to remove usage of UTF8String from my application.

My confusion was caused by the fact that Lazarus uses AnsiString with default system code page but inside it always stores UTF-8, sort of fooling the compiler. Is this correct? If so, is it not possible to set Lazarus to use String=AnsiString(CP_UTF8) by default?

cobines wrote

My confusion was caused by the fact that Lazarus uses AnsiString with default system code page but inside it always stores UTF-8, sort of fooling the compiler. Is this correct? If so, is it not possible to set Lazarus to use String=AnsiString(CP_UTF8) by default?

This is not desired at the moment because compiler is not 100% ready to use UTF8String.

When compiler support for cp strings is ready Lazarus will probably switch to UTF8String type.

Best regards,
Paul Ishenin.

2011/9/19 Paul Ishenin :

I see, so the removal of UTF8String is sort of a workaround for current compiler condition. If in future Lazarus switches to UTF8String then applications using this type won't have to change it. But if I want to test now I need to remove it. I understand, thanks.

My statement about RawByteString need to be read in the context of code where I replaced UTF8String to AnsiString.

Best regards,
Paul Ishenin.

The choice of UTF-8 (for Delphi Ansi) strings is the first incompatibility. Shouldn't we cure it, by following the new Delphi Unicode model? Otherwise another string-type incompatibility is added to the string-encoding incompatibility :-(

When FPC starts to dictate inappropriate rules[1], I see no way around a fork into an Ansi/UTF-8 and an UTF-16/Unicode branch[2], according to the break in Delphi. This would mean that the old branch has to stick with an older compiler (current release), and the new branch requires the new compiler.

[1] The FPC developers currently try to find a model that fits both needs, compatibility with Delphi *and* Lazarus - we'll have to wait for its outcome.

[2] IMO it should be possible to separate user-land strings from platform/widgetset strings. Then all components can continue to use UTF-8 internally (in talking to the widgetsets), while the user accessible strings can be of another type. With a proper choice of that internal boundary, the number of excess conversions can be kept at a minimum, as well as the required changes to the LCL code.

IMO the use of RawByteString will not help much, except for (possibly) simpler code and less overloaded procedures. Avoiding implicit conversions instead will require *fixed* string types and encodings, for different tasks with different needs. E.g. a TFileName string type will allow to eliminate all conversions, when a string is known to hold file or path names (by design). Likewise an LCLString (widget, component) type could do the same for the LCL widgetset interface. The FPC decisions about string container classes (TStrings...) will tell where to put the break line, between user and widget string types.

DoDi

I don't suppose Lazarus can work with a cpstrnew version of moving Lazarus to cpstrnew will be a lot of work.

I suppose you can compile FPC in a way that does not provide cpstrnew.

-Michael

Work that needs to be done sooner than later, because cpstrnew was merged at the weekend (though it doesn't seem to be THAT much work to get Lazarus back in running shape; using the new feature is a different topic though).

I suppose you can compile FPC in a way that does not provide cpstrnew.

The only way I'm aware of: Use a revision of FPC before the merge.

Regards,
Sven

Just to point that RawByteString should be used in very specific cases like stated in the link below

http://stackoverflow.com/questions/498315/delphi-2009-rawbytestring-vagaries

Regarding change to UTF8ToUTF16 function RawByteString should not be used.

function UTF8ToUTF16(const S: UTF8String): UTF16String;

Currently does not matter if is UTF8String or AnsiString, but when the new string type starts to be used, there's no problem using UTF8String.

In fact the use of UTF8String will be necessary otherwise explicit conversions with extra hidden temporary variables inside the function will be necessary as is the case with RawByteString.

Also PChar should be changed to PAnsiChar since PChar can point to a word (UTF16 RTL) or a byte (UTF8 RTL)

I explained in a mail in the devel list (where you see UnicodeString read as UTF8String)

Luiz

Michael Schnell wrote

I don't suppose Lazarus can work with a cpstrnew version of FPC at all. Supposedly moving Lazarus to cpstrnew will be a lot of work.

fpc will provide two rtl (UTF8 and UTF16) Lazarus
should work without much changes with the UTF8 RTL. Changing to a UTF16 RTL (if desired) would in fact be harder, but also not much after the transition to UTF8.

Luiz

Luiz Americo Pereira Camara wrote

Regarding change to UTF8ToUTF16 function RawByteString should not be used.

BTW: as soon the new string type is used, UTF8ToUTF16 and alikes will not be necessary at all

Luiz

To be a decent New-String Unicode library, the LCL should use the New-String Type everywhere (supposedly in the source using some dedicated "LazarusString" Type that is an alias for the compiler's New-String Type) and drop any use of old style strings wherever possible (supposedly everywhere but at the lines directly interfacing the OS'es API).

Seems like a huge task to me.

-Michael

Luiz Americo Pereira Camara wrote

Nope. AFAIK String type will automatically map to UTF8 or UTF16 string depending of a compiler switch or directive and the RTL used (UTF8, UTF16).

What do you mean by "map to UTF8" ? AFAIK, there is no string type in FPC that decently denotes a dedicated encoding for the content so that library functions know how to handle them.

No need to "LazarusString". The unicode indenpendent code

If Unicode is possible at all there is no "Unicode independent code"as a user input can contain non-ASCII-characters

(most of it) will not need to be changed, just the code that interfaces with RTL. Probably some string handling like CodeTools and SynEdit will need to be updated also

If the compiler provides New Strings, it's not a decent way to do a library not using them. The end user ("naive" programmer) will use them and thus library calls will be problematic.

-Michael

2011/9/20 Michael Schnell :

What do you mean by "map to UTF8" ? AFAIK, there is no string type in FPC that decently denotes a dedicated encoding for the content so that library functions know how to handle them.

You will be able to alias String for UnicodeString (UTF-16) or AnsiString(CP_UTF8), and maybe AnsiString(Ansi code page) too, I'm not sure. And Lazarus will use String=AnsiString(CP_UTF8), from what I understand.

If the compiler provides New Strings, it's not a decent way to do a library not using them. The end user ("naive" programmer) will use them and thus library calls will be problematic.

If you mean UTF-16 UnicodeString then I don't think there will be a problem. If your program or library uses UTF-16 and Lazarus uses UTF-8 the conversion routines will be inserted automatically by the compiler where needed.

cobines wrote

String=AnsiString(CP_UTF8),

Meaning that a library function (or an FPC syntax candy feature) can't detect the code that is used for the string.

If you mean UTF-16 UnicodeString then I don't think there will be a problem.

AFAIK, Lazarus does not use UTF-16 (other than in the Windows API)

If your program or library uses UTF-16 and Lazarus uses UTF-8 the conversion routines will be inserted automatically by the compiler where needed.

If the application programmer uses any 8 bit coding (such as ANSI or UTF-8), as said, the type does not denote the coding, thus there can't be an automatic conversion.

AFAIK, automatic correct UTF-8 to UTF-16 conversion is not done by the compiler when assigning an "String" to a "WideString". The string type the compiler sees (after evaluation the type alias) is ANSIString, so maybe it does an ANSI to UTF-16 conversion.

-Michael

Michael Schnell wrote

On 09/20/2011 01:09 PM, cobines wrote: > String=AnsiString(CP_UTF8), Meaning that a library function (or an FPC syntax candy feature) can't detect the code that is used for the string.

Huh? What do you mean here? If String is declared as a AnsiString(CP_UTF8) then the a variable declared as String does contain UTF8 data... and any code can detect that by using GetCodePage which will return CP_UTF8.

Regards,
Sven

2011/9/20 Michael Schnell :

The code page will be stored in the record describing the string, you will be able to read it with something like StringCodePage(s).

AFAIK, automatic correct UTF-8 to UTF-16 conversion is not done by the compiler when assigning an "String" to a "WideString". The string type the compiler sees (after evaluation the type alias) is ANSIString, so maybe it does an ANSI to UTF-16 conversion.

With the code page aware strings it will be able to detect encoding and do UTF-8<->UTF-16 too.

Ah, I see. He was meaning a New String Type there. I did not understand this. (So supposedly the following messages are wrong are well. )

Thanks,
-Michael

Sorry, I suppose I misunderstood one of your previous messages.

-Michael

2011/9/20 Michael Schnell :

Sorry, I suppose I misunderstood one of your previous messages.

IMHO, almost every message in thread after the first couple of messages is premature, since nothing has been decided for the string extension is fpc 2.7.1. There everything said about how Lazarus or the LCL should use it is pure speculation, as most unicode threads seem to degrade to. :-(

Vincent

But at least we have the new code page aware string in trunk now, so the recent Unicode discussions were worth something :P

Regards,
Sven

2011/9/20 Sven Barth :

But at least we have the new code page aware string in trunk now, so the recent Unicode discussions were worth something :P

Paul, is that true? Did the recent Unicode discussions on this list and the fpc lists help you / motivate you to work on integrating the cpnewstr branch into trunk?

My estimate is that is more "in spite of these recent discussion" then "because of these recent discussions"

Vincent

In that case it would have been a very interesting coincidence ;)

Regards,
Sven

I just decided that cp string would probably be the best choise for me among the features listed here:
http://wiki.lazarus.freepascal.org/delphi_language_features_which_fpc_does_not_have. I look at this page from time to time when I want to do something in compiler.

Best regards,
Paul Ishenin.

Then an interesting coincidence it is... ^^

Regards,
Sven

Sure. The 2.6.0 branch will run for at least 1 1/2 year (and probably 2+) years.

cpnewstr is trunk/2.8.0