I started a separate thread for this lazarus part of the unicode talk.

Florian Klaempfl wrote

> And that's why I urge all core FPC > developers to try and finalize a Unicode design. Otherwise you leave > it up to developers to keep adding such shitty hacks They add it only because they insist on using utf-8 :)

I keep hearing this, but yet no one ever proposes a better solution for Lazarus.

It was already stablished what reasons made us choose this path, the most important of which being that it provides maximum compatibility with existing code.

I would like to hear of others actually have a better proposal for Lazarus.

And also, considering that Lazarus and Free Pascal are very related (I would bet most people using fpc use it througth Lazarus, and everyone using Lazarus uses fpc), it would be interresting if we actually work more or less in the same direction to provide a good unicode solution, instead of each part ignoring what the other is doing. And also remembering that Lazarus started working on Unicode before FPC and also before Delphi.

So, what kind of support could be implemented in Free Pascal to improve things for Lazarus and it´s users?

Maybe a real UTF8String?

if a real utf8string would be a solution for Lazarus (I am not saying it is, but it could be), we need to have a directive to change the default string into utf8string. To avoid a huge amount of code to need to be suddenly changed. Then only "ansistring" needs to be changed.

Felipe Monteiro de Carvalho wrote

using Lazarus uses fpc), it would be interresting if we actually work more or less in the same direction to provide a good unicode solution, instead of each part ignoring what the other is doing. And also

I fully agree.

So, what kind of support could be implemented in Free Pascal to improve things for Lazarus and it´s users? Maybe a real UTF8String?

My first take would be the following real string types: AnsiString, ShortString, UTF8String and UTF16String.

Then a compiler directive that decides what the String type actually represents. Just like the $H directive says String = AnsiString or String = ShortString. So the new compiler directive can toggle between any of the 4 string types.

This way Lazarus and fpGUI can toggle String to actually mean the real UTF8String. MSEgui could toggle String to actually be a UTF16String.

I don't see the point or need for a UCS2String type as suggested by others. UTF16String will cover UCS2 support and more.

As an optional extra, to prevent unneeded automatic conversion, I would suggest Linux and most other unix variants default String = UTF8String and Windows and WinCE default String = UTF16String.

Obviously with the compiler directive, the developer can override the default behaviour.

As for loading files. It's 99.9% that all files are in ANSI or UTF8 encoding and UTF8 being fulling backward compatible with ANSI makes this a good thing. So under Windows, TStringList.LoadFromFile() will default to UTF8 loading (or auto detect if possible). Once the content is loaded, convert it to the platforms default encoding UTF16 for Windows, or stay as is (UTF8) for most unix variants.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Maybe a real UTF8String?

Does this mean teach the compiler tell the type UTF8String from the type ANSIString and do the appropriate conversion automatically (and do the assignment of constants appropriately) ?

I suppose this in fact would solve a lot problems for Lazarus.

If on top of that the user can choose - e,..g. on a per-unit or even "on the fly" base - which string type he wants to use when he just writes "String" (ANSIString, UTF8String or WideString, with WideString again choosable as UTF16String or UCS2String, and the types "Char" and "WideChar" selected appropriately (you name it), it would be easily usable.

Regarding Lazarus LCL, it of course would be advantageous if it could be compiled and used with any String type as it's API, so that conversions can be minimized according to the project (needing locale independence or not, which OS, optimize for speed or for size (e.g. Win-CE or embedded Linux projects, needing WideString due to extended string operations in the user code, ...)

-Michael

Felipe Monteiro de Carvalho wrote

I would like to hear of others actually have a better proposal for Lazarus.

sorry, I have no idea since I am doing primarily embedded stuff. Speed and backward compatibility are the most important factors to me.

But it seems, that not everybody is happy with the current Codegear Unicode solution:

https://forums.codegear.com/thread.jspa?threadID=7140&tstart=0

After reading this thread, I am not sure, if Delphi 2009 compatibility is a must for FPC/Lazarus.

Regards, Bernd.

Op Thu, 20 Nov 2008, schreef Bernd Mueller:

The complaints there are about lack of backwards compatibility, i.e. people need to suddenly replace char by ansichar.

FPC will remain 100% compatible with old code. This is why there need to be separate platforms. Platforms where char=ansichar for full compatibility and platforms where char=widechar to take advantage of full Unicode.

Daniël

But it seems, that not everybody is happy with the current Codegear Unicode solution: https://forums.codegear.com/thread.jspa?threadID=7140&tstart=0

This is neither backwards compatible, nor nice, nor fast nor small :(

After reading this thread, I am not sure, if Delphi 2009 compatibility is a must for FPC/Lazarus.

I agree !!!!
But for future compatibility it would be nice if FPC/Lazarus could provide a _mode_ that allows for writing D2009 compatible code.

(You might also take a look at the string handling in Delphi-Prism which I believe is the way Embarcadero will be going in the future and where FPC/Lazarus is not sensibly / necessarily / easily / likely following. )

-Michael

Graeme Geldenhuys wrote

As for loading files. It's 99.9% that all files are in ANSI or UTF8 encoding and UTF8 being fulling backward compatible with ANSI makes this a good thing.

UTF-8 is not compatible with ANSI, but only with ASCII.

Petr

Op Thu, 20 Nov 2008, schreef Felipe Monteiro de Carvalho:

So, what kind of support could be implemented in Free Pascal to improve things for Lazarus and it´s users? Maybe a real UTF8String?

There will be a real UTF8string, i.e. ansistring with UTF-8 encoding as part of type information, this will help Lazarus users to get rid of the utf8encode/utf8decode.

On the other hand Lazarus may want to move to a "string depending on platform" too, to attract both Delphi < 2009 and Delphi >= 2009 users.

Daniël

Zitat von Felipe Monteiro de Carvalho :

if a real utf8string would be a solution for Lazarus (I am not saying it is, but it could be), we need to have a directive to change the default string into utf8string. To avoid a huge amount of code to need to be suddenly changed. Then only "ansistring" needs to be changed.

Compiler support for a unicode string is not enough for the LCL. As long as base classes like TStrings uses ansistrings, the LCL must use a string type, that does no conversion.

For example:
Items[i]:=Caption;

As long as Items[i] is treated as ansi, any automatic conversion can loose unicode characters.

So the roadmap from LCL pov is:
- compiler support for some unicode string on every platform: done in fpc 2.3, thanks to Florian
- a RTL using unicode strings
- changing the string types in the lazarus code
- a fpc release with the unicode RTL

Mattias

" ansistring with UTF-8 encoding as part of type information"

This sounds like a dynamic solution? As in the info that it is UTF8 encoded is part of the data, not of the type? Or do I misunderstand?

IMHO UTF8String should be like widestring. The encoding/decoding related parts are decided at compile time; not at runtime.

Op Thu, 20 Nov 2008, schreef Martin Friebe:

You do. I said "type information", not "data".

Daniël

Compiler support for a unicode string is not enough for the LCL. As long as base classes like TStrings uses ansistrings, the LCL must use a string type, that does no conversion.

Of course you are right that the RTL needs to be made up accordingly. Maybe TStrings and friends are needed in multiple versions.

Using a string type that dynamically can be used top hold multiple encodings (as discussed here, too) sounds nice, if it does not result in too slow code (due to the necessity of checking for the coding with any operation).

-Michael

Yes, I would have thought that's a given. All base classes need unicode support where appropriate - that includes TStrings.

Using a string type that dynamically can be used top hold multiple encodings (as discussed here, too) sounds nice, if it does not result in too slow code (due to the necessity of checking for the coding with any operation).

At the moment, I think first supporting Unicode is the most important step. Performance tuning comes afterwards. That how it seems to work now and I don't see a problem with that. FPC generates assembler code, and it's still an ongoing process in tuning the performance of the assembler code. I remember how things where tweaked a while back with all the language shootout tests.

We can implement Unicode shootout tests later. ;-)

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Daniël Mantione wrote

On the other hand Lazarus may want to move to a "string depending on platform" too, to attract both Delphi < 2009 and Delphi >= 2009 users.

I don't see this change any time soon, because it would break too much of existing code. If Delphi 2009 becomes really popular and there is a large need to migrate projects to Lazarus we may think of a solution.

At the moment a fully working UTF8String is what we need.

This is easily said, please create examples and descriptions how fully working is defined.

that we build our own set of routines/classes using UTF8String.

Later they could be added to Free Pascal. This is more or less what we are doing at the moment, so we should just continue in the same direction.

Daniël Mantione wrote

There will be a real UTF8string, i.e. ansistring with UTF-8 encoding as part of type information, this will help Lazarus users to get rid of the utf8encode/utf8decode.

When? Is this planned for 2.4?

thanks,

Op Fri, 21 Nov 2008, schreef Felipe Monteiro de Carvalho:

See Marco's e-mail from 10:21.

Daniël

Florian Klaempfl wrote

This is easily said, please create examples and descriptions how fully working is defined.

// Should actually convert from widestring to utf-8 when using encoding utf-8
programa utf8test1;
{$encoding utf-8} // or is it utf8?
var
  Str: UTF8String;
begin
  Str := "ção";
  if Length(Str) = 5 then Success
  else Fail;
end;
// Should work on all platforms. Passing the UTF8String to a routine
that requires
// ansistring should do the proper conversion
programa utf8test2;
{$encoding utf-8} // or is it utf8?
var
  Str: UTF8String;
begin
  Str := "ção";
  WriteLn(Str);
end;

// Should actually convert from widestring to utf-8 when using encoding utf-8 programa utf8test1;

In fact it should automatically convert ("as correctly as possible") between all available string types (ANSI, UTF8, UTF16).

Should provide appropriate char types for all available string types.

User selectable way of element counting in utf-strings (code element counting or code point counting)

The RTL would need to provide the appropriate objects (e.g. StringList) for all available string types.

-Michael

Of course correct automatic assigning of strings constants defined in the source file correctly regarding the coding of the source file (e.g. UTF8 or ANSI) Maybe here the coding of the source file needs actively be communicated to the compiler.

Of course assigning a constant should not do a conversion at run time.

-Michael

Big deal, I simply enable operator overloading for unique string types to get this working, then everybody is happy and we've unicode support?

Florian Klaempfl wrote

This is easily said, please create examples and descriptions how fully working is defined.

It would be really good if there was a guide, preferably in the wiki, to explain how to add a new test case to Free Pascal.

I have already some test cases in mind, like making sure static methods compile (an error in 2.2.2) and then after some discussion the utf-8 test cases. At the moment I can't add the test cases because I don't know how to.

thanks,

I'am quite sure I made a test case when I fixed it.

and then after some discussion the utf-8 test cases. At the moment I can't add the test cases because I don't know how to.

Just create a program which returns 0 if everything was ok and another value if it fails and attach it to a bug report. The program might only depend on FPC units, not LCL or anything else.

Florian Klaempfl wrote

Big deal, I simply enable operator overloading for unique string types to get this working, then everybody is happy and we've unicode support?

Indeed that could work. But the operator overloading would need to override the widestring managed. Actually it will be a bit confuse to have 2 methods to change the assignments: the widestring manager and the operator overloading.

We also need a {$ to set which string type "string" should be.

And then we need a set of RTL routines using utf8string, but Lazarus developers/users can write them after the other parts are working.

Michael Schnell wrote

In fact it should automatically convert ("as correctly as possible") between all available string types (ANSI, UTF8, UTF16).

And the compiler should produce a warning if you assign UTF8 or UTF16 string to a ANSI string. Mentioning that conversion is not 100% possible and you stand a chance to loose data.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

For everything but deep compiler stuff, I would think fpcUnit should do perfectly. After all, that is what fpcUnit (or unit testing in general) is for. And unit tests can have a GUI or Text (console) test runner. The latter being handy for automated runs - daily or hourly. tiOPF project does this and we have around 1600 unit tests.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Zitat von Graeme Geldenhuys :

... and a possibility to tell the compiler 'Thanks, I know. Don't bark about this place any longer'.

Mattias

Mattias Gärtner wrote

... and a possibility to tell the compiler 'Thanks, I know. Don't bark about this place any longer'.

:-) Yes definately! Like the wish for "Parameter not being used" or "Sender not being user" etc... Those drive me nuts!

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Felipe Monteiro de Carvalho wrote

It would be really good if there was a guide, preferably in the wiki, to explain how to add a new test case to Free Pascal.

It is documented here: http://svn.freepascal.org/cgi-bin/viewvc.cgi/trunk/tests/readme.txt?view=markup

You can find tons of examples under http://svn.freepascal.org/cgi-bin/viewvc.cgi/trunk/tests/

Description of the subdirectory names:

<stolen from Florian>
test: systematic tests, usually developed by test driven development webtbs: tests derived from bug tracker bugs requiring successful compilation and run
webtbf: tests derived from bug tracker bugs requiring failing of compilation
tbs: tests derived from non tracker reports or ideas while fixing something requiring successfull compilation and run tbf: tests derived from non tracker reports or ideas while fixing something requiring failing compilation
</stolen>

Jonas