Hello again,

We are seeing more and more "hacks" being applied to projects trying to scramble around the missing FPC feature - no built-in Unicode supporting.

A simple example in Lazarus.... Loading a UTF-8 encoded file into a TMemo.

Normally you would write code as follows (for ANSI text):

   Memo1.Lines.LoadFromFile('someunicodefile.txt');

Unfortunately that doesn't work if the file contains unicode content, so the following "hack" is required which is quite nasty:

   ls := TStringList.Create;
   ls.LoadFromFile('someunicodefile.txt');
   for i := 0 to ls.Count-1 do
      ls[i] := UTF8Encode(ls[i]);
   Memo.Lines.Assign(ls);
   ls.Free

All that crap just to load a simple text file that contains unicode content!!! :-( And the other problem is that the hack above assumes the files content is UTF-8 encoded. If the content is UTF-16 encoded, you need yet another hack. :-(
As far as I know the Unicode spec does mention a way of detecting the encoding type (educated guess). As a second stage backup for GUI toolkits, we can do the same as KDE or Qt has done in File Open/Save dialogs. The user can specify an Encoding type from a combobox overriding the auto-detect process.

So PLEASE lets finalize a Unicode strategy for Free Pascal!!! Like I said before, I'm willing to help where I can - be that writing unit tests or implementing some RTL functions. I don't know the deep internals of FPC, but the RTL seems relatively easy enough to code.

Marco summarized earlier discussions, but still no clear conclusion! FPC is getting more and more behind in this regard. And developers need to keep adding ugly hacks as shown above into there application. This causes double work for developers. Applying the hack now, then after FPC one day has Unicode support, remove the hacks.

http://www.stack.nl/~marcov/unicode.pdf

I'm also pretty sure I am not the only one that is willing to help with this feature - many developers require Unicode support. And this is just going to become more urgent as time goes by. So lets finalize a design and start implementation. Pretty please. ;-)

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

shorter (and faster) hacky crap:

  ls := TStringList.Create;
  ls.LoadFromFile('someunicodefile.txt');
  Memo.Text := UTF8Encode(ls.Text);
  ls.Free

Ok, two questions for the example above:
- how do you maintain backward compatibility?
- how do you load a plain old ansi file?

Op Thu, 20 Nov 2008, schreef Graeme Geldenhuys:

These instructions are highly unproductive. Work on being able to compile the RTL in either ansi/unicode depending on the platform has started. Full Unicode support is for FPC 2.4. If you need it today, widestrings are your best option.

If you want to help, we need to implement the Delphi 2009 encoding aware string type, both runtime support as well as the compiler support.

Daniël

Florian Klaempfl wrote

Ok, two questions for the example above: - how do you maintain backward compatibility? - how do you load a plain old ansi file?

If the file is UTF-8 or ANSI, the above should work. UTF-8 was designed to be backward compatible with ANSI. One of the beauties of UTF-8.

But yes, with other encodings it's a problem. And exactly the reason I think TStrings and the whole FPC for that matter should start supporting Unicode somehow. And that's why I urge all core FPC developers to try and finalize a Unicode design. Otherwise you leave it up to developers to keep adding such shitty hacks which could cause more issues in the long term - instead of solving issues.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

FPC supports Unicode, in 2.3.x is the UnicodeString type available being a ref. counted utf-16 string on all platforms.

And that's why I urge all core FPC developers to try and finalize a Unicode design. Otherwise you leave it up to developers to keep adding such shitty hacks

They add it only because they insist on using utf-8 :)

Op Thu, 20 Nov 2008, schreef Graeme Geldenhuys:

Wrong. It was designed to be compatible with ASCII, but incompatible with ANSI. A CP1250 encoded file won't show correctly in an UTF-8 encoded environment.

That's why you have to convert. In the current FPC, the compiler can infer the right conversion from type information only if the string is ansi encoded or UCS2/UTF16 encoded. It simply hasn't type information to do the other transformations correct. Therefore you have to call a function, there is nothing we can change about that.

Design is almost final. Just give us time to implement.

Daniël

Daniël Mantione wrote

These instructions are highly unproductive. Work on being able to compile the RTL in either ansi/unicode depending on the platform has started. Full Unicode support is for FPC 2.4.

Well, that's the first I heard of it. Last time I checked, there was no clear decision as to what design/implementation to follow? So where is the finalized design spec / wiki page etc, so I can see where I can help?

If you read Marco's PDF, there are a few ideas for Unicode implementation, and a few weeks ago, it seems nothing has changed. Everybody seems to have there own ideas.

If you need it today, widestrings are your best option.

At the moment I use UTF-8 and standard String type - as implemented in fpGUI. So I'm okay for now with our current projects, but there are still many outstanding issues and pitfalls (as my earlier example shows).

If you want to help, we need to implement the Delphi 2009 encoding aware string type, both runtime support as well as the compiler support.

Well considering that I don't have a copy of Delphi 2009 and don't want to fall into the same trap as before - where FPC had copyrighted code in the RTL, I would prefer to look at the design guidelines / spec as set out by the core FPC team. Where can I get hold of such a document or information?

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Can you summarize design in some document. For us developpers is important to see the future.

Petr

Op Thu, 20 Nov 2008, schreef Graeme Geldenhuys:

Marco's PDF was written prior to the Delphi 2009 information and not really updated. People still have a bit different ideas, but it is clear what needs to be done. As you may know, the UTF-16 string is already implemented in 2.3, no one disputes its need. No one disputes the need for a Delphi 2009 compatible ansistring.

While not everything has been settled yet, things are moving in a certain direction.

No document exists.

There will be Delphi 2009 styled ansistrings with the encoding as part of the type information. You can view how it works from the published Codegear information.

The rtl is being adapted so that it can be compiled with the desired string type, i.e. ansi, utf8string or utf16string. What platforms will get a utf8 or utf16 string as default has not been decided yet.

Daniël

I've used this ""hack"" in my projects so far (both FPC and C++), it works really well, and I don't have a bit of problem with it. The simple rules I use is:

- all internal strings in a project must be in UTF-8 (which I used)
- all localized strings are written in external files, never inside the code
- encoding/decoding is done in three situations:
1) before/after I/O operations (where I know/force encoding and format of files I use)
2) before/after system calls (if neccessary)
3) in several and well-defined places in GUI

As far as I can see, Lazarus team made similar decisions. Yes, this ab-uses AnsiString, but it works, and it works really well :-)

As of Unicode future inside FPC, after reading Marco's document, I'm sure FPC will be compiler with superb support for Unicode (both compiler- and RTL-side).

Greets,
Aleksa

Florian Klaempfl wrote

FPC supports Unicode, in 2.3.x is the UnicodeString type available being a ref. counted utf-16 string on all platforms.

OK, I'll try to switch fpGUI's TfpgString type to alias UnicodeString an see what happens. Obviously this will be done in a separate branch, not to screw with Trunk.

So where can I find out what has been implemented and what is missing? Some or other roadmap, like Lazarus has for it's LCL. http://wiki.lazarus.freepascal.org/Roadmap#Status_of_components_on_each_widgetset

eg:
Does 2.3.x with UnicodeString support the following (off the top of my head).

* TStringList.LoadFromFile() - what happens if file is UTF-8 encoded like under Linux.
* Does it support > 1byte Locale variables for Currency Delimiter, etc.. Russian locale
requires a >1 byte char.
* FindFile, FindFirst? Do they handle Unicode filenames? What about other file IO calls.
* Copy, Length, Pos etc...?
* What about usage like: SomeString[x] := 'A';
* What if we call external API's with string parameters? XLib, OpenGL, Win32 etc?
* Does UnicodeString work on all platforms? Linux, Windows for a start?

And the list goes on.... Is there a wiki page (like the Lazarus LCL

How to I enable UnicodeString in 2.3.x or do I just use UnicodeString
instead of String type?
eg:
   s: Unicodestring;
   ls: TStringList;
begin
  ls := TStringList.Create;
  ls.LoadFromFile('someunicodefile.txt');
  s := ls[2];   // s now contains a UTF-16 encoded string?

They add it only because they insist on using utf-8 :)

At this point I don't care what encoding, the final result is the same. UTF8 was just easier to support in fpGUI and automatically handled > BMP support. But it would be nice if there was something that wasn't very Windows centric. After all, FPC runs on more than just Windows nad there are lots of different unix environments, which all seem to use UTF-8 instead.

But like I said, the final result is the same -- supporting Unicode. Calling external library API must just be handled with care, but I already do that for fpGUI under Windows anyway.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

FPC supports Unicode, in 2.3.x is the UnicodeString type available being a ref. counted utf-16 string on all platforms.

Is same used by TStringList ?

I don't think so, otherwise LoadFromFile should need to be aware of several possible file encodings. And I suppose the utf8-API of the LCL would not work.

-Michael

Russian locale requires a >1 byte char.

Hmmm. We did lots of non-Unicode Delphi programs with a Russian ANSI variant.

-Michael

Op Thu, 20 Nov 2008, schreef Graeme Geldenhuys:

The file is assumed to be in system encoding (which can be UTF-8). Support for reading of other encodings has not been decided on about yet and is not part of the initial plan.

* Does it support > 1byte Locale variables for Currency Delimiter, etc.. Russian locale requires a >1 byte char.

It can remain a system encoded ansichar for 100% compatibility, or on selected platforms become a widechar. Conversion to multibyte (utf8) can in both cases be inferred by the compiler from type information.

* FindFile, FindFirst? Do they handle Unicode filenames? What about other file IO calls.

Win32 widestring platform for sure. Not really decided on non-Unicode platforms yet, they may remain in the system encoding. Autoconversion will in such case happen based on type information.

* Copy, Length, Pos etc...?

Yup.

* What about usage like: SomeString[x] := 'A';

String element based.

* What if we call external API's with string parameters? XLib, OpenGL, Win32 etc?

Auto-conversion based on type information.

* Does UnicodeString work on all platforms? Linux, Windows for a start?

Yes, but all platforms will get string=unicodestring.

Daniël

Op Thu, 20 Nov 2008, schreef Daniël Mantione:

> * Does UnicodeString work on all platforms? Linux, Windows for a start? Yes, but all platforms will get string=unicodestring.

There is a "not" missing:

Yes, but not all platforms will get string=unicodestring.

Daniël

Full Unicode support is for FPC 2.4. If you need it today, widestrings are your best option.

Unfortunately working with WideString in Lazarus is close to impossible as the LCL API is done with UTF8String and there is no correct automatic conversion between UTF8String and WideString, as the compiler can't tell UTF8String from ANSIString.

Moreover assigning a constant string to a WideString needs manual call to a conversion (in run-time) when using Lazarus.

-Michael

Well, I have a Russian user of fpGUI. He noted quite a few issues with FPC's locale variables and Russian language. We had to implement our own > 1 byte locale variables and create custom formatting strings to use those instead of FPC's ones. MSEgui had to do the exact same thing. A lot of effort, plus you need to notify the toolkit users to use the toolkit specific formatting functions, instead of the standard RTL ones.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

If you want to help, we need to implement the Delphi 2009 encoding aware string type, both runtime support as well as the compiler support.

A previous discussion showed that this also breaks a lot of old code and is not really nice.

So a better concept seems to have a dedicated type for any possible Coding (ANSISTring <of course locale-depending>, UTF8String, UTF16String, maybe UCS2String, too) and let the user choose (e.g. by a "{$" compiler option) which one he want to be used for "String" and "WideString". This would allow for simple compiler magic to perform any necessary conversion (including assigning constants).

-Michael

Op Thu, 20 Nov 2008, schreef Michael Schnell:

As I understand it, the incompatibility from Delphi 2009 comes from the fact that and char string by default has becomes 2 bytes, not by adding encoding information to the type.

Isn't this the same??

Daniël

The file is assumed to be in system encoding (which can be UTF-8). Support for reading of other encodings has not been decided on about yet and is not part of the initial plan.

What is "system encoding" regarding different OS, locale, ... ?

-Michael

Op Thu, 20 Nov 2008, schreef Michael Schnell:

System encoding is the encoding your files are written in when doing a "echo Hello > file.txt".

Daniël

Good explanation Daniël. :-) I always wonder that same question.

So on my systems:
Ubuntu 7.10 that's UTF8
Windows 2000 that's ANSI (I guess).

On both systems I have UK English locales setup.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

This also holds for Copy, Length, Pos, etc.

I thinks if would be a good idea to provide dedicated functions for the "element based" (fast) and the "character based" (old style compatible, easily readable) functionality. The user should be allowed to choose by means of compiler directions which of them should be used with the old-style notation. This would allow for not breaking old code.

type information could be static (different types) or dynamic (like with D2009)

-Michael

System encoding is the encoding your files are written in when doing a "echo Hello > file.txt".

nice point :)

I Suppose with my German WinXP system encoding is German ANSI

Does it hold only for files ? I suppose WinXP provides an OS API with WideStrings (supposedly UCS16).

But how do I have a program detect the current system encoding ?

-Michael

Isn't this the same??

I understand that D2009 uses dynamic code information, while my suggestion is based on several different (static) types.

I feel that static types are a lot easier to implement and if using them correctly, the user can tune the program to be as fast as possible or as small as possible. This helps a lot with embedded projects. dynamic types are due to much bloating regarding space and performance.

-Michael

UCS16

UTF16 :)

-Michael

For best backward compatibility, I would say Copy, Length, Pos etc should work by "character based" by default. Then introduce more optimised versions like ElementCopy, ElementLength, etc... Old programs will work out of the box, but might experience a minor speed penalty, until the developer has time to convert to the more optimise calls. Just a thought?

And to replace SomeString[x] code with a unicode safe / equivalent, we can implement a CharAt(str, x) method.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Op Thu, 20 Nov 2008, schreef Michael Schnell:

As I understand it is static.

type  cp850string=ansistring(CP_850);
       utf8string=ansistring(CP_UTF8);
var a:cp850string;
     b:utf8string;
begin
   a:=b;  {Compiler knows conversion to perform at compile time.

I feel that static types are a lot easier to implement and if using them correctly, the user can tune the program to be as fast as possible or as small as possible. This helps a lot with embedded projects. dynamic types are due to much bloating regarding space and performance.

I agree. However, FPC may still decide to do some things dynamic to prevent a blowup of rtl size.

Daniël

For best backward compatibility, I would say Copy, Length, Pos etc should work by "character based" by default.

The thing is we can't reasonablly provide functions based on what a user would see as a character because doing so would require huge lookup tables (one user visible character != one code point) so the best we can do is code point based which isn't really much better for most tasks than code unit based and would make certain operations MUCH slower. In other words it would be a major speed penalty for a minor gain.

For best backward compatibility, I would say Copy, Length, Pos etc should work by "character based" by default.

Agreed.

Agreed (but the speed penalty might be huge !) That is why I vote for a compiler option to safe the programmer the work to modify his code.

And to replace SomeString[x] code with a unicode safe / equivalent, we can implement a CharAt(str, x) method.

Here another problem might arises, as the "character" type needs to be defined accordingly (supposedly):
String Character
ANSIString ANSIChar (1 Byte)
UTF8String UTF8String
UCS2String UCS2Char (2 Byte)
UTF16String UTF16Char (4 Byte)/

-Michael

If a user in fact needs something else that code point counting, he of course needs to add his own procedures for that purpose. I feel that the vast majority of projects will be fine with what had been possible with ANSI (and I think that this is code point counting even with the silly "u plus Umlaut" coding that has been discussed last week).

-Michael

type cp850string=ansistring(CP_850); utf8string=ansistring(CP_UTF8);

Why not use the current locale for this ? Would that be just ANSIString ?

a:=b; {Compiler knows conversion to perform at compile time.

I suppose the conversion function is provided with the locale and this it as well could use the current one.

I agree. However, FPC may still decide to do some things dynamic to prevent a blowup of rtl size.

Maybe this really is an even better concept (when done in a way that it provides fast and small code for "embedded" projects).

If course it would be good to allow the user to write his code in a way that as little conversions as possible are necessary (e.g everywhere use the API coding of the OS he compiles for, or use especially fast UCS2 WideStrings for analyzing, building his Unicode information.

-Michael

Ok, two questions for the example above: - how do you maintain backward compatibility? - how do you load a plain old ansi file?

You could alter the LoadFromFile(), LoadFromStream(), SaveToFile(), SaveToStrwam() routines like below:

procedure TStringList.LoadFromFile(AFileName: TFilename; const ACharSet:
TCharSetKind = csSomeDefaultCharset);

This way, code using these would compile and work as usual.

Of course, descendants would need to be modified --a one time modification.

I think basing those functions on code points should suffice. I also think as soon as strings are assigned or loaded from file, they should be normalized. So two code points like the A and Umlaut code points would become one.

The .SaveToFile() methods could take an optional parameter to decide if the normalized version of the string gets saved, or if it must be split again - which I think Mac OS-X prefers.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Graeme Geldenhuys wrote

I think basing those functions on code points should suffice. I also think as soon as strings are assigned or loaded from file, they should be normalized. So two code points like the A and Umlaut code points would become one.

How would one know which code points were originally decomposed and which weren't? Should it be impossible to save a file that demonstrates the different possible UTF encodings of e.g. ö, and should a loaded/saved file which contained both encodings really be automatically entirely composed or decomposed when saved again?

I know of no text editor that handles UTF which automatically changes the encoding of pre-existing characters when saving the documents. And I would never want to use a text editor which does that by default.

The .SaveToFile() methods could take an optional parameter to decide if the normalized version of the string gets saved, or if it must be split again - which I think Mac OS-X prefers.

It doesn't. All OS functions that return file/path names return decomposed (UTF-8)strings. They accept both composed and decomposed strings. Text files are text files and can have any encoding you want, with any combination of composed and decomposed characters.

Jonas

Graeme

if the matter is only loading text from file, why don't you write your own utility function to load files content. After text is loaded from file, you could process them as WideStrings.

That's the way i'm doing. Since LCL handles UTF8 strings I always encode widestrings for LCL components, but for internal processing i use WideStrings.

For easy of use, i've written utility function, so loaded files (or streams) are first checked if they're unicode. Unicode text are then converted to required format (UTF8 or UTF16). See the attached unit.

procedure TForm1.Button1Click(Sender: TObject);
begin
  if OpenDialog1.Execute then
    Memo1.Text := ReadTextFile(OpenDialog1.FileName, true);
  // if file is UTF16 and utf16 mark is present, then ReadTextFile
will convert read WideString to the UTF8string
  // if i need WideString first, i'd use ReadTextFileWide, that reads
WideString (so no additional converting is used if UTF16 file is read)
  // but if file is UTF8 then ReadTextFileWide convert to UTF16.
end;
procedure TForm1.Button2Click(Sender: TObject);
var
  prc : TProcess;
begin
  prc := TProcess.Create(nil);
  try
    prc.Options := [poUsePipes, poStderrToOutPut];
    prc.CommandLine := Edit1.Text;
    prc.Execute;
    Memo1.Text := ReadTextStreamWide(prc.Output);
  finally
    prc.Free;
  end;
end;

Zitat von Graeme Geldenhuys :

Mac OS X (or better: HFS) auto normalizes. SaveToFile just needs unicode.

Normalization is only important when comparing filenames. That's why lazarus code always uses CompareFilenames.

Mattias

To make matters worse, things don't get clear, if mistakes like above are made.

For Lazarus it works if the file is UTF8 encoded and doesn't work if it has the system encodingUTF8.

If the file is UTF8 encoded,
Memo1.Lines.LoadFromFile('someunicodefile.txt');
works, currently, in Lazarus.

If the file is not UTF8 encoded - which implies, that it is not ASCII either - then you need to convert it to UTF8 before assigning it to the memo lines.

I hope, I didn't make a mistake myself here.

Vincent

That must name Convert not Hack
it is same when you work with Ansi version of Lazarus/Delphi and then try to load from unicode file.

I meant TStringList must not make Converting, convert string must be outside of TStringList (or add special methods to it), and without detecting the encode inside the file when LoadFromFile or Stream, Detecting may use Seek function in the stream, and that break load from tcp/ip connection or compressed stream that have not seek.

Zaher Dirkey wrote

That must name Convert not Hack it is same when you work with Ansi version of Lazarus/Delphi and then try to load from unicode file.

Zaher Dirkey wrote

I meant TStringList must not make Converting,

If it's known that a file is in some encoding and the instance of TStringList uses another one, I suppose LoadFromFile needs to do the re-encoding appropriately.

-Michael

In our previous episode, Dani�l Mantione said:

Full Unicode support is for FPC 2.4. If you need it today, widestrings are your best option.

Is it? Because that might mean yet another 2.2 fixes branch release to fix up the delay that this will cause to 2.4

In our previous episode, Florian Klaempfl said:

They add it only because they insist on using utf-8 :)

That's perfectly normal on *nix.

Op Fri, 21 Nov 2008, schreef Marco van de Voort:

People were complaining against the current FPC, not being aware of the new UTF16 string type in FPC 2.3. Perhaps it indeed needs postponing to an even later release, but at any Unicode support should not be expected for 2.2.

Daniël

In our previous episode, Dani�l Mantione said:

Yes. The added strings are perfectly compatible. (ansistring for 1 byte encodings unicodestring for 2 byte encodings) The incompatible part comes from switching the default string/char type to the 2 byte variants.

Typing wise it is the same. (except for the nonexistance of UCS2), implementation wise not.

But Michael obviously hasn't read the PDF.

Marco van de Voort wrote

Is it? Because that might mean yet another 2.2 fixes branch release to fix up the delay that this will cause to 2.4

Another 2.2 fixes branch release is a good idea, because it contains a fix for static methods which is necessary for Cocoa projects.

When Marco said "yet another 2.2 fixes branch release", he meant 2.2.6.

Vincent

Vincent Snijders wrote

When Marco said "yet another 2.2 fixes branch release", he meant 2.2.6.

Ah, ok ... =)

So my commend would then be changed to:

Unicode is what is most discussed and needed at the moment. What is the point in making a major release without any major change? For me it doesn~t matter if it will take time.

In our previous episode, Felipe Monteiro de Carvalho said:

Both branches are divergating, and merging gets more difficult. Also all changes (like a lot of alignment stuff for ARM) would be held up.

Note that I'm happy with either way. I just want a regular release schedule no matter what course (early 2.4 or late 2.4) is taken, and paint the consequences of declaring the uncodestuff.

I want to avoid self-delusion of painting optimistic timeschedules and saying "this time a major release preparation won't take as long as last time".

It is late november now, and 2.4 preparation hasn't started, so I have doubts we'll see 2.4 before summer, even _IF_ we don't wait for more unicode functionality (ansistring_with_UTF8, TEncoding)