Hello,

There is already another thread about that, but the thread got too long, and I would like to make a concrete proposal about unicode file routines.

It looks simple to me, there are just 2 ways to go, either utf-8 or utf-16. Correct me if I am wrong, but I beliave that FPC developers prefer utf-16, so we can have a widestring version of every routine in the RTL which involves filenames.

So let me start with a concrete example:

http://www.freepascal.org/docs-html/rtl/system/assign.html

We would need to add a:

procedure Assign(
  var f: ;
  const Name: widestring
);

Also for all this routines:

http://www.freepascal.org/docs-html/rtl/sysutils/filenameroutines.html

Under Windows it can be implemented like this with Windows 9x support:

procedure AnyFileRoutineInWin32(AFileName: widestring);
begin
 if UnicodeEnabledOS then SomeWin32APIW()
 else AnsiToWideString(SomeWin32ApiA())
end;

One can initialize UnicodeEnabledOS by reading the operating system version and the operating system type NT/9x very easily.

Under Windows 9x we won't support true unicode filenames, but this doesn't matter, because the operating system doesn't support them anyway. The widestring routines will keep working under Windows 9x for most code.

This method is used with great success in the LCL. Extended information here:

http://wiki.lazarus.freepascal.org/LCL_Unicode_Support#Guidelines

Zitat von Felipe Monteiro de Carvalho :

But what about all existing code?
For example the FCL?
How will TStringList.LoadFromFile be converted?

Mattias

There are more possibilities:
- native encoding (utf-8 on *nix, utf-16 on windows)
- have two types.
- an unified type (type contains encoding)

Even this has not been decided.

How can one write portable code with these options?

- an unified type (type contains encoding)

Vincent

2008/6/30 Felipe Monteiro de Carvalho :

It looks simple to me, there are just 2 ways to go, either utf-8 or utf-16. Correct me if I am wrong, but I beliave that FPC developers prefer utf-16, so we can have a widestring version of every routine in the RTL which involves filenames.

I thought UTF-8 was prefered. Hence the reason Lazarus followed the UTF-8 route in LCL and Unicode support.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Mattias Gärtner wrote

But what about all existing code? For example the FCL? How will TStringList.LoadFromFile be converted?

TStringList.LoadFromFile(AFileName: widestring); overload

The ansi version could call the wide version and just do the string conversion.

Graeme Geldenhuys wrote

I thought UTF-8 was prefered. Hence the reason Lazarus followed the UTF-8 route in LCL and Unicode support.

UTF-8 is much better for the LCL because it just fits much better in out existing codebase.

For the RTL we would also like to have UTF-8, but in previous conversations I got the impression that RTL developers prefer UTF-16.

How can you consider yourself portable by picking one systems encoding, and emulating it on others?

Note also that reliance on encoding is way less important, since fewer people will be parsing through strings manually (simply because it is more difficult)

At the borders of my I convert all strings to the 'internal type' and encoding and use it like that. Kind of like we are doing nowadays to convert the line-endings in text files.

I see what you are trying to say, but having a string type that is UTF8 encoded on one system and UTF16 encoded on another system, doesn't seem easy to work with to me, even if you name it for example RTLString. Even widestring is an example of bad portability, because they are refcounted everywhere except on windows.

Note also that reliance on encoding is way less important, since fewer people will be parsing through strings manually (simply because it is more difficult)

Right, but they rely on not having to convert it all the time.

ATM, all the client libs above the RTL have chosen one encoding, string type: LCL en fpGUI: UTF8, MseGui: widestring

So for those libs to interface with a platform dependent string type in the LCL, they would have to write platform dependent code. I don't feel much like writing a LCLSysutils.FileExists, like Graham already has done, to hide these conversions.

Vincent

Marco van de Voort schreef: At the borders of my I convert all strings to the 'internal type' and encoding and use it like that. Kind of like we are doing nowadays to convert the line-endings in text files.

I don't like this. This makes e.g. processing a database export on Unix unnecessarily costly

I see what you are trying to say, but having a string type that is UTF8 encoded on one system and UTF16 encoded on another system, doesn't seem easy to work with to me, even if you name it for example RTLString.

It should be possible to work in the native encoding. One doesn't want to wrap _every_ function in _every_ header with conversions procs.

Well, they will have to do that with one string type too, at every external barrier.

That also kills the benefit of choosing UTF-16 in the first place, since Delphi code won't work on Unix without manually inserting a lot of conversion code.

ATM, all the client libs above the RTL have chosen one encoding, string type: LCL en fpGUI: UTF8, MseGui: widestring

That has nothing to do with these decisions. They chose that in the absence of a good solution. This is about picking a good solution.

So for those libs to interface with a platform dependent string type in the LCL, they would have to write platform dependent code.

You will have to anyway for any solution that only supports one encoding.

Marco van de Voort wrote

It should be possible to work in the native encoding. One doesn't want to wrap _every_ function in _every_ header with conversions procs.

It is not possible to work with a ever changing encoding.

MyLabel.Caption := 'Lição';

How would that ever work with a ever changing encoding? It would not.

If you go to the real implementation level a changing encoding quickly becomes unmanagable.

And what about the LFM files? In which encoding will they be? What if you develop a software in one system and tryes to build it in another?

Ok, to go one step further: Has anyone ever seen a fully unicode system which works with changing encodings? I beliave there exists none, because this is not a good solution.

Well, they will have to do that with one string type too, at every external barrier.

This is already necessary.

That also kills the benefit of choosing UTF-16 in the first place, since Delphi code won't work on Unix without manually inserting a lot of conversion code.

Delphi code can use the ansi routines, which could just call the utf-16 routines with a string conversion, or you can implement every routine twice to maximize speed.

Encoding in source is something totally different. This is '\u1232\u2314' like syntax can be changed to utf8/16 by the compiler. In theory I think, practice might be else.

If you go to the real implementation level a changing encoding quickly becomes unmanagable.

That's why I don't believe the one string type two encoding helps. But if fileexists is utf-8 on unix and utf-16 on windows, and any utf-16 or UTF-8 string that you pass from Lazarus is auto converted, what is the exact problem? Everybody can maintain certain subsystems in a certain encoding, but doesn't force that choice upon others.

And what about the LFM files? In which encoding will they be?

The one you annotate in it? The loading code can decode both, since both systems have both ?

What if you develop a software in one system and tryes to build it in another?

What does that mean for the fully UTF-16 system? First you may start with wrapping all C api's that use utf-8 on Unix.

I understand the simplicity of one encoding is appealing, but you have to look at all aspects, and that is not just representation in the GUI.

It will mean that _every_ string transactie to the outside will have to be manually wrapped AND have a performance penalty. That is a heavy price to pay for not touching a bit of lfm loading code.

Ok, to go one step further: Has anyone ever seen a fully unicode system which works with changing encodings? I beliave there exists none, because this is not a good solution.

How many systems do you know have datafiles of like .lfm's over system borders?

> Well, they will have to do that with one string type too, at every > external barrier. This is already necessary.

But if you properly type them, some conversions maybe automatic. Something you don't have with a single type.

If the unicode code is not compatible with Delphi (UTF-16), there is no point in using UTf-16 in the first place.

This may have been discussed before - but should the encoding not be dependent on the locale? What would happen if I write a FPC program, if the internal routines are, eg., UTF-16, and my locale is set to en_US.UTF8?

Anyway, I have the impression that most of Linux is utf-8 oriented by now.

John

It won't need to be wrapped in platforms which nativelly support the choosen encoding. UTF-16 is natively supported in Windows and Windows CE. Not sure on unixes.

Because LCL uses a single encoding this performance difference disappears as soon as you need to convert the string in LCL.

How many systems do you know have datafiles of like .lfm's over system borders?

Gtk can load XML files, somewhat equivalent to our LFMs. They use UTF-8 everywhere.

Java is cross-platform and uses UTF-16 everywhere.

wxWidgets uses UTF-16 everywhere.

Let me try to sumarize my oppinion on multiple encodings vs single encoding:

multiple encodings:

* More complex
* Innovative solution, no known example of a implementation of this system exists = uncertainty if it works at all, or if it is convenient for developers
* Depends on a not yet implemented string type
* Potentially will have a higher performance then a single encoding system, but only if you use this new special string type

Single encoding:

* Simple, proved solution
* Does not need any new string type, can start being implemented immediately
* Potentially has a lower performance due to string conversions.

Actually for Lazarus the only advantage I see in the multiple encoding system does not exist, because we use a single encoding system in some platforms we will need conversion and in others we won't need, which just makes things worse for us.

En/na John Coppens ha escrit:

Well, yes, but that's the external representation. I'd say to take a look at how python managed to integrate unicode support:

http://www.google.com/search?domains=www.python.org&sitesearch=www.python.org&sourceid=google-search&q=unicode&submit=search

Bye

They have a UTF-16/UCS-2 internal representation, same as MSEgui which works very well and is fast and handy BTW.
What is missing is a reference counted widestring type on Windows. ;-)

Martin

En/na Martin Schreiber ha escrit:

And len, slicing, etc. work as expected.
Note that if you need characters beyond $ffff you have to compile it with wide unicode support, and in that case every character will use 4 bytes.

http://www.python.org/dev/peps/pep-0261/

I think the default is still to compile without wide unicode support (at least python under mandriva, that has no special configure option, is a narrow python build, while debian etch has a wide one).

Bye

GTK is unix centric on other systems. They don't have a firm leg in both the Unix as the Windows world as we do. I can't judge the wxwidgets situation, since I know nobody that uses it.

Java is cross-platform and uses UTF-16 everywhere.

Java has to emulate everything (read: put up a barrier) from the outside anyway, and not doing that is one of our fortes.

Needs to be done anyway, since widestring on windows is COM, and that must be also retained. So it is about adding 1 vs 2, and the work will be huge, with UTF-16 too, and to make it worthwhile the best, not the quikest solution should be sought.

* Potentially will have a higher performance then a single encoding system, but only if you use this new special string type

Certainly. Can you imagine loading a non trivial file in a tstringlist and saving it again and the heaps of conversions?

Moreover, there is an important reason missing:

* Being able to declare the outside world in the right encoding, without manually inserting conversions in each header.

* Does not make one of the two core platforms (Unix/windows) effectively second rate.

* Can be done phased, IOW in the beginning lots of conversion, but later have more and more routines in the right encoding ready.

Single encoding: * Simple, proved solution

Simple solution, complex implementation (needs conversions anywhere).

* Does not need any new string type, can start being implemented immediately

It does. And you can start making UTF-16 routines anyway

* Potentially has a lower performance due to string conversions.

That's IMHO a faulty system. It requires you to choose between an incomplete solution or making strings a horrible memory hog. But maybe that doesn't matter for mere scripting languages (though I wonder then why they didn't chose UTF-32 directly)

Surrogates are not nice, but they were invented for a reason.

En/na Marco van de Voort ha escrit:

OTOH using variable length characters will make string operations expensive (since you can't just multiply the index by 2 or 4 but you have to examine the string from the beginning, and the length in bytes isn't the same as the length in characters).

Well, yes, they're a trade-off between performance and memory consumption, but I fear we're losing one of the advantages that pascal has over C: fast and simple string handling.

Bye

It's amazing that this argument come up again and again. But I know hardly any code that need this index to char mapping. And the code, that need it is seldom time critical.
(I must admit, I feared the same some years ago. But the extra cost is practically a myth.)

Most code only needs the number of bytes. And this still cost under pascal O(1).
In fact if a UTF8String or UTF16String would be added, then I would say, it would be a waste of memory to store an extra PtrInt for the number of characters.

Mattias

Yes. In the routines where you do random access on elements. Half of that can be gained back since most string routines iterate from start to end anyway.

We also don't want to slip of the other end and turn into a scripting language unsuitable for major programming.

[...] > multiple encodings:

Are we talking about one encoding per platform or two encodings for all platforms?
Under Unix the encoding preference is clear: UTF-8. Under Windows there are a lot of current code page texts and the UTF-16 W functions. So, what encoding is the preference under windows? UTF-16 plus Ansi like the A and W functions?

Auto conversion of the strings in a TStringList does not make much sense (and will break a lot of code). That's why I propose to keep one default string type. If almost everything uses one string type, then no conversion will take place.

I think the main problem is that the RTL calls the Ansi functions under windows. Maybe we should not loose the focus.

Windows need per se at least two encodings. So whatever is decided, the windows part need some more work.

Mattias

A good example is text layout calculation where it is necessary to iterate over characters (glyphs) over and over again. MSEgui uses widestrings directly, fpGUI converts to widestrings before processing (or use they the slow utf-8 routines ?). I once switched MSEgui to utf-8 because of the widestring problems in FPC, one or two months later when I implemented complex layout calculation with tabulators and justified text I switched back to widestrings...
This belongs to a GUI framework, for a RTL are possibly other priorities.

Agreed.
I think the best compromise for a GUI framework are referencecounted widestrings where normally physical index = code point index. If one needs characters which are not in the base plane, he must use surrogate pairs and more complicated and slower processing. I assume this will be seldom used.

Martin

On Tue, 1 Jul 2008 09:23:52 +0200 (CEST)

(note that this is all IMHO, not necessarily core viewpoint)

Are we talking about one encoding per platform or two encodings for all platforms?

My proposition was: Two encodings, two stringtypes for all.

Florian's stand was thinking about one stringtype that supports both encodings. I don't like this, but we can only discuss that if Florian has more details about his ideas.

Maybe to not get the RTL unwieldy don't implement the more exotic routines (like soundex and some of the more complicated formatting routines) for all encodings. This will also allow to optimize binary size a bit. (don't want all the routines two fold? Add a define when compiling the RTL, and the relevant's encoding includefiles are not included and the compiler inserts conversions). Not that I think that is that important (except for CE maybe).

Note that I want to do some (if not most) of the RTL stringroutine work. I like doing them, they are like little puzzles. Though it will take some time to get proficient in unicode string coding.

Under Unix the encoding preference is clear: UTF-8. Under Windows there are a lot of current code page texts and the UTF-16 W functions. So, what encoding is the preference under windows? UTF-16 plus Ansi like the A and W functions?

Split the win32 target into
- a win9x compat + legacy codepages, using A
- NT unicode port that strictly -W.

The ports can share nearly all code, and use/perfect the already remaining IFDEF UNICODE remains. (IOW the NT/UNICODE port defines UNICODE, the other not)

It will on every communication with the external world. IOW all my db exports will generally be UTF-8 on Unix and UTf-16 on Windows.

This one size fits all attitude works fine for Lazarus, with only human latency to worry about, and small amounts of data, (and that is already a challenge to keep performant) but not for FPC as a whole, as all processing is hit severely by it.

Most notably, in the single string case, the only way to avoid the forced encoding is to everything OS specific and manual. That is IMHO too poor.

I think the main problem is that the RTL calls the Ansi functions under windows. Maybe we should not loose the focus.

This is not about loosing focus but gaining it. Out with the evolutionary workarounds and start making decisions.

See above. If we have to support two totally different OS api's (A and W) they are two different targets. Period.

This also avoids the mess of changing all windows routines to be dynloaded, and hopefully lessen the mutual breaking a bit.

Text layout nowadays need to consider font widths and unicode specials. Iterating from character to character should be hardly measurable compared to this. For example synedit does not yet care much about font widths and unicode specials and the UTF-8 stepping is negligible.

It depends if your code should solve a special problem or if you write a library that should work for all. The RTL and FCL should work for all. So they must support UTF-16 and can not use a limited widestring.

Mattias

> On Tue, 1 Jul 2008 09:23:52 +0200 (CEST) (note that this is all IMHO, not necessarily core viewpoint)

Same for me: mine are not lazarus core.

> Are we talking about one encoding per platform or two encodings for > all platforms? My proposition was: Two encodings, two stringtypes for all.

Both at the same time?

Florian's stand was thinking about one stringtype that supports both encodings. I don't like this, but we can only discuss that if Florian has more details about his ideas.

I think, Marc had a similar idea. Adding an encoding field (e.g. in front of the length). But IMO it has some drawbacks.

I guess, that means only one at a time.

Maybe you misunderstood me here. This section is about multiple encoding proposal. So I was proposing to use only one string type in RTL/FCL. It can be a different one for each platform. As long as almost everywhere only one string is used no conversion can take place and you can therefore store UTF8 in widestrings or UTF-16 in strings or whatever binary data. Just as it is at the moment. Strings are not only text. I think this concept is very important in pascal and breaking this will create a bigger incompatibility than Codegear does with it string to widestring move.

ok

Two different windows targets. Wow, a big step.

Mattias

I did it with utf-8 and UCS-2, beleave me, it was not negligible.

That's why I wrote "for a GUI framework". There we have always the possibility to access the OS with optimized routines independent from RTL and FCL and to provide the optimozed stringhandling routines for the chosen internal string representation. What is necessary for the toolkit user is automatic conversion from the GUI framework internal string type to the system encoding. That already exists for widestrings.

Martin

Yes, utf8string and utf16string. Whatever Tiburon introduces aliased to utf16string, so that will be compat on non-windows too. And the utf16 tiburon code can easily communicate with the outside world.

Yes. Any manual string handling, that already gets more difficult, gets more expensive. Also because array dereference (which ignores surrogates, but is still a baseblock for string routine implementation) becomes expensive, or needs to be done with pointers.

Ok. That is somewhat different. One size fits all (UTF-16 everywhere) is not an option for me. It's the way of the least resistance, but is more for languages that have an ivory tower concept and want to keep the real world at arms length.

So then different platforms, different encodings. Actually that was my first thought/proposal too, but that precludes any possible solution for Tiburon compability before we even start, and introduce a portability barrier. (want to recompile for linux ? First fix all your UTF16 string routines so that they support UTF-8 under ifdef. That is a hard sale)

IMHO that is no long term sustainable situation, so which is why I changed to the two stringtypes solution.

That has some disadvantages too, most notably adding even more string types and possible auto-conversion pitfalls. But I think it is an experiment that should at least have been tried.

Note that this is totally separate from what Lazarus should do. Lazarus can IMHO happily use the UTF16 string type exclusively. I'm concerned with the base system.

As long as almost everywhere only one string is used no conversion can take place and you can therefore store UTF8 in widestrings or UTF-16 in strings or whatever binary data.

It still requires manual conversion at the borders (any input or output to system, libraries,disk). But a lot less since only sources in an encoding "foreign" to the system need manually conversion code inserted.

Just as it is at the moment. Strings are not only text. I think this concept is very important in pascal and breaking this will create a bigger incompatibility than Codegear does with it string to widestring move.

???

Yes, but longterm unavoidable IMHO, to avoid the situation we had with Dos in years past, where the port is always trailing the Tier 1 ports. (though Giulio and Tomas managed to keep it working again I saw, but only after releases of it were postponed)

W9x support is being dropped on all sides. However for me that is not necesary if we split the stuff now, while the w9x support is still qualitively ok. Even though w9x and NT are both windows, in some ways they differ more than e.g. FreeBSD and Linux.

Doing the split before major NT requiring changes (read:unicode, but also e.g. symlink support?) will make the change more evolutionary, and the branching from a moment where the codebase is still proven to work on w32 will assure that it will have decent quality for quite some time.

In the long term it will also save a lot of work, like crazy attempts tomaintain the status quo with insane workarounds like dynloading all api routines etc.

Zitat von Martin Schreiber :

Where is the code in msegui? (the code that was formerly UTF-8, not the old UTF-8 code)

Ah, ok. Yes, a gui framework is a special case.

Mattias

lib/common/kernel/msedrawtext.pas, mserichstring.pas, msestrings.pas.

http://sourceforge.net/projects/mseide-msegui

Martin

Zitat von Martin Schreiber :

Thanks. Can you be little bit more specific? I see a lot of functions. Most of them can treat UTF-8 as 8bit encoding. Unless you want to do something special.

Mattias

I read most of the discussion and I think there is no way around a string type containing an encoding field. First, it allows also to support non utf encodings or utf-32 encoding. Having the encoding field does not mean that all target support all encoding. In case an encoding is not supported, the target might either use some default operation as the current widestring manager does or it might spite out an exception. Having such a string type requires some manager which does not only store the procedures to handle this string type but which also contains some information which encoding to prefer or even use solely. Combining this with several ifdefs and compiler switches makes this approach very flexible and fast and allows everybody (FPC people, Lazarus, MSE) to adapt things to their needs.

Just an example: to overcome the indexing problem efficiently when using an encoding field (this is not about surrogates), we could do the following: introduce a compiler switch {$unicodestringindex default,byte,word,dword}. In default mode the compiler gets a shifting value from the encoding field (this is 4 bytes anyways and could be split into 1 byte shifting, 2 bytes encoding, 1 bytes reserved). In the other modes the compiler uses the given size when indexing. For example, a Tuberion (or how is it called?) switch could set this to word.

The approach has the big advantage, that you really need all procedures only once if desired. For example e.g. linux would get only utf-8 routines by default, utf-16 is converted to utf-8 at the entry of the helper procedures if needed. Usually, no conversion would be necessary because you see seldomly utf-16 in linux applications so only the check if the input strings are really utf-8 is necessary, this is very cheap because the data is anyways already in a cache line.

Even more, this variable encoding approach allows also people using languages where utf-8 is more memory expensive than utf-16 (this is in numbers the majority of mankind) to use utf-8/utf-16 as needed to save memory only with a few modifications.

I know this approach contains some hacks and requires some work but I think this is the only way to solve things for once and ever.

In this routines length(widestring), widestring[index], pwidechar^, pwidechar[index], pwidechar + offset, pwidechar - pwidechar and inc(pwidechar)/dec(pwidechar) are used often. This can't be done with utf-8 strings.

Martin

Marco van de Voort wrote

Certainly. Can you imagine loading a non trivial file in a tstringlist and saving it again and the heaps of conversions?

And how do you know that the file to be loaded will be in the system encoding? We should simply not do any conversion or any assumption when loading a file in a TStringList, so nothing changes here.

We are talking about strings like the filename in LoadFromFile, and not about the string to hold the contents. This would always be an ansistring and if someone needs to load a utf-16 file he needs to build a TWideStringList.

Moreover, there is an important reason missing: * Being able to declare the outside world in the right encoding, without manually inserting conversions in each header.

This has nothing to do with this. With a fixed encoding you can also have automatic conversions.

I bet you would convert automatically from whatever to ansi when going to a ansistring, but Lazarus uses utf-8 in ansistrings.

We do manual conversions in Lazarus because FPC misses a solution for automatic conversion using utf-8 in ansistrings.

Florian Klaempfl wrote

I read most of the discussion and I think there is no way around a string type containing an encoding field.

[cut]

I know this approach contains some hacks and requires some work but I think this is the only way to solve things for once and ever.

I think it is the most promising and extensible proposal, so I'm all for it.

Michael.

is the problem of the programmer, and depends on what he knows (or can detect), it is not the systems choice.

Point is that if I only have UTF-16, I must convert to UTF-8 data coming in manually, do some minor processing (like append a string to a tstringlist), and then convert it back before writing.

And I have to, since UTf16 is my only type.

We should simply not do any conversion or any assumption when loading a file in a TStringList, so nothing changes here.

You have an assumption, since your TStringList would be UTF16 strings only.

We are talking about strings like the filename in LoadFromFile, and not about the string to hold the contents. This would always be an ansistring and if someone needs to load a utf-16 file he needs to build a TWideStringList.

A solution for unicode should be for everything, not just for UIs and filenames. I should be able to carry data within it also, because otherwise we are having this dicussion next week again if Joost needs unicode for DB related issues etc.

How? I can't express the foreign encoding because I have no type for it. I only have ansistring that can mean pretty much everything, and that constitutes no compiletime safety.

I bet you would convert automatically from whatever to ansi when going to a ansistring, but Lazarus uses utf-8 in ansistrings.

But that is lazarus specific.

We do manual conversions in Lazarus because FPC misses a solution for automatic conversion using utf-8 in ansistrings.

Because the decision to put utf-8 in ansistrings is too fundamentally flawed to implement such a thing, since it perfectly legal if an ansistring does not contain utf8

I read it shortly, and I still don't like it. I need more time to prepare a reponse though.

A string type which you don't know the encoding is very inconvenient, because you need to convert it to something else anytime you wish to do any routine which will require knowing the encoding.

How will Pos be implemented? And UpperCase? Any cross-platform string manipulation routine will sudenly become single-platform. Any one implementing a string routine will need to implement it several type, not to mention that no string routines currently exist for such new type.

The more I think about it the more I am sure that a string type whose encoding we know nothing about is very inconvenient for developers.

About UCS-2 this is absurd. We certainlly cannot have half the chinese characters ignored in the Free Pascal RTL.

Keep in mind in your response, that we want also handle other formats than utf-8 or utf-16 if needed :)

Marco van de Voort wrote

A solution for unicode should be for everything, not just for UIs and filenames. I should be able to carry data within it also, because otherwise we are having this dicussion next week again if Joost needs unicode for DB related issues etc.

Ok, but how do you know that everyone wants to store data in the "system" encoding?

What if I want to store data using ansistring in Windows because my file is UTF-8?

In my system I propose that simply a TWideStringList be implemented, so both ways of storing data are available everwhere.

How? I can't express the foreign encoding because I have no type for it. I only have ansistring that can mean pretty much everything, and that constitutes no compiletime safety.

ansistrings don't mean everything. They mean either ISO or utf-8. They can never hold a utf-16 string (or at least there are no routines to cover this case).

> I bet you would convert automatically from whatever to ansi when going > to a ansistring, but Lazarus uses utf-8 in ansistrings. But that is lazarus specific.

Lazarus is by far the largest project using Free Pascal?

Because the decision to put utf-8 in ansistrings is too fundamentally flawed to implement such a thing, since it perfectly legal if an ansistring does not contain utf8

We concluded that utf-8 in ansistrings is a very convenient solution for us which works very well today. It provided a smooth migration path and keeps the vast majority of code working.

We may some day migrate to a possible utf8string type when it gets implemented.

I think that if you put the encoding field at a negative offset, as length for ansistrings, that this code should be relatively compatible to current code if you assume that encoding=0 (or whatever tag value) means ansistring: You just have an extra field; you could even make that 2 fields: in addition to byte length, add character length: it should keep operations fast, as most conversions and other operations will end up with a character length of some kind anyway.

Michael.

Felipe Monteiro de Carvalho wrote

ansistrings don't mean everything. They mean either ISO or utf-8.

This assumption is wrong. ansistring means the system encoding which uses 8 bit chars.

Even if the system encoding is UTF8?

Vincent

2008/7/1 Felipe Monteiro de Carvalho :

In my system I propose that simply a TWideStringList be implemented, so both ways of storing data are available everwhere.

I have a TWideStringList implementation if you are interrested. I got the code somewhere and kept it for a rainy day.

> But that is lazarus specific. Lazarus is by far the largest project using Free Pascal?

If I dare comment - that's a bold statement to make. ;-) I know a few more "large" projects using Free Pascal. Off the top of my head Pixel32 being one.

We may some day migrate to a possible utf8string type when it gets implemented.

In which case I suggest Lazarus start using a custom/alias string type to ease mirgation. Something like was done with TTranslateString. Maybe TLCLString = String

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Well, euh, the main reason is that euh, most programs and data on the system uses the system encoding?

What if I want to store data using ansistring in Windows because my file is UTF-8?

Then I'd say you convert. But that is the point. The need for conversion should be the exception (different from the default system encoding), not the rule.

In my system I propose that simply a TWideStringList be implemented, so both ways of storing data are available everwhere.

But I don't have an utf-8 type in your system to operate on.

Yes. Which is why there is a need for a separate UTF-8 type as well as the UTF-16 type. So that the compiler knows for sure something is UTF-8, and can insert conversions. And can error/hint/warn you to insert manual conversions if you assign an unicode type (either) to an ansistring.

FPC itself?

Anyway that doesn't matter. A solution for FPC must be carried broadly, not just by lazarus.

Because you had no choice.

Then it means utf-8 of course. What I wanted to say, it _means_ something: 8 bit system encoding.

Why not just introduce a set of utf-16 routines with utf16string type like the new Delphi?

This proposal is at least better then the one from Marco as we at least can get the encoding somehow, but is still inconvenient for cross-platform software.

Felipe Monteiro de Carvalho wrote

Why not just introduce a set of utf-16 routines with utf16string type like the new Delphi?

Because it's not cross platform.

Felipe Monteiro de Carvalho wrote

About UCS-2 this is absurd. We certainlly cannot have half the chinese characters ignored in the Free Pascal RTL.

???
Where did you get the information that half of the Chinese characters won't fit in base plane? And utf-16 surrogate handling is much simpler than utf-8 variable character length handling.

Martin

Martin Schreiber wrote

Where did you get the information that half of the Chinese characters won't fit in base plane?

http://unicode.org/roadmaps/sip/index.html

CJK means Chinese Japanese Korean

Why not just introduce a set of utf-16 routines with utf16string type like the new Delphi? This proposal is at least better then the one from Marco

My is having both an UTF8string and a UTF16string, on all platforms that support unicode. So I don't get this remark.

It is just that on unix, the fileroutines will be defined as utf8string and on windows with utf16 strings.

SO if I pass a different type, the compiler will insert a conversion.

The only problem with this scheme is that it doesn't support utf-32, or it needs yet another additional type. But at least it is a proper compiletime option, and in a procedure I can see from the declaration what unicode type I'm gonna get.

I had hoped that the implementation of a clean orthogonal set of unicode types (2 or 3, 4 if you count the COMstring) might be not as bad as 3 new string types, since we know they can always be converted to eachother, so maybe we could reuse and share implementation here and there.

But since Florian tries to cast it out of the compiler and to the runtime, I suspect he thinks that is not possible.

Marco van de Voort wrote

Well, euh, the main reason is that euh, most programs and data on the system uses the system encoding?

So you are saying that FPC should privilege platform-specific software development to cross-platform software development? This is in the inverse direction of all other cross-platform development platforms in existence.

If you are writting cross-platform software you will wish to avoid as much as possible the system routines, and a known encoding is good.

Florian's proposal shines here. You get the string with no conversion and a marker for the encoding, so you can convert it to whatever you want easily.

But it doesn't solve the TStringList problem, because there you have no parameters to know the encoding of the file being loaded.

Then I'd say you convert. But that is the point. The need for conversion should be the exception (different from the default system encoding), not the rule.

I think there should be no conversion at all (unless explicitly asked) in the contents of the stringlist.

> In my system I propose that simply a TWideStringList be implemented, > so both ways of storing data are available everwhere. But I don't have an utf-8 type in your system to operate on.

How do you know what I want to do with the data? What if I just want to use some string routines in it to extract data? Or save them back to another file? (or any operations which don't involve system routines which need a specific string encoding)

Ok a quick pointwise comment then.

I don't like the runtime want to be able to say "hey look, I've a bunch of units here, and they only accept utf16, (e.g. because they were ported Tiburon code). Convert if necessary"

So we need at least one directive in that case, one that says "all unicodestrings under this directive are in encoding type <n>, convert if necessary".

I don't understand how this can work, how can I have a compiletime solution for a runtime problem?

procedure mystringproc (s:FlorianUnicodeString);
begin
  if encodingof(s)=utf-16 then
    begin
      // utf-16 code here with shiftsize 2 [] needed
    end
  else
    begin
      // utf-8 code here with shiftsize 1 [] needed
    end;
end;

I wonder if having 2,3 (utf-8,16 and maybe -32) straight simple unicode types isn't easier than this polymorphic beast.

At least then you have one procedure, one encoding, and since they all guaranteedly convert (and to comstring too), the conversion code might be not as painful as when ansistring and widestring were introduced. It could be parameterisable in the compiler. With the added advantage of compiletime decisions.

Florian Klaempfl wrote

> Why not just introduce a set of utf-16 routines with utf16string type > like the new Delphi? Because it's not cross platform.

Why isn't is cross-platform?

Marco van de Voort wrote

I don't like the runtime want to be able to say "hey look, I've a bunch of units here, and they only accept utf16, (e.g. because they were ported Tiburon code). Convert if necessary"

Tiburon code will never run in this case because in var parameters the exact type must match. And it will not match, doesn't matter how many compile directives you use.

No, we should privilege cross-platform software development over an portable emulation of a 3rd platform (the Java principle).

This is in the inverse direction of all other cross-platform development platforms in existence.

I'm only having FPC and Lazarus requirements on the table here. I don't care about the others. They have other starting points (being very unix or windows centric, or work with portable sandboxes)

And in my case you specify you want UTF-16 by making the parameter "utfstring16", and the compiler inserts a conversion for you if sb calls it with a utfstring8. No manual runtime check necessary.

But it doesn't solve the TStringList problem, because there you have no parameters to know the encoding of the file being loaded.

No there is no solution for that except making the string type really fat. Which is not our way.

Well, that means the tstringlist is a blind store without any methods. It isn't since any operation requires knowledge about the insides.

Does it matter? I just want to be able to tailor to the most common scenario's. See my other msg that restates the proposal in simpler terms.

Or save them back to another file? (or any operations which don't involve system routines which need a specific string encoding)

You've really lost me now. I think you are still confusing general unicodestrings with unicodifying a few filename using routines.

Marco van de Voort wrote

My is having both an UTF8string and a UTF16string, on all platforms that support unicode. So I don't get this remark.

Unless I understood your proposal wrong it involves a TMarcoString which will be declared like this:

{$ifdef Linux}
  If SystemEncoding = utf-8 then TMarcoString = Utf8String
  else TMarcoString = ansistring;
{$endif}
{$ifdef Windows}
  If WindowsNT then TMarcoString = utf16string
  else TMarcoString = ansistring;
{$endif}
{$ifdef Darwin}
  TMarcoString = utf8string;
{$endif}

Just how do you implement a string routine with TMarcoString? Fill it with ifdefs?

It is just that on unix, the fileroutines will be defined as utf8string

So you are going to convert in non utf8 unix?

So I must add some glue to the outside of the system for those few cases. When you start tying code together with different encodings this is always the case.

But at least you can do this, contrary to your proposal, where I must do all my communication with an UTF-8 system with heaps of slow manual conversions, or even worse, manually using pchars.

Because using utf-16 on linux is very unnatural, same for utf-8 on windows. Platforms like go32 even don't have any unicode. Coding platform independent but fast applications is really ugly having fixed types.

[ Charset ISO-8859-1 unsupported, converting... ]

No. Just utf8string and utf16string, with tutf16string aliased to the identifier that Tiburon nems it.

But on Linux the RTL is mostly utf-8, and on Delphi the RTL is mostly utf-16. And if you pass the utf-16 filename that you got from the Lazarus .dfm (that is apparantly already set to remain utf-16) to a filename routine a converrsion will automatically happen.

People that make a FPC distro can decide if their Linux version contains a full complement of overloaded utf16 routines or not. If not more conversions will happen if you have pure utf-16 code, but that can be worthwhile for embedded/minimalist distributions if those ever emerge.

And only the few special cases with var are a problem, as you correctly pointed out, and a these few cases can be fixed for the RTL by overloading them aslo for utf16.

> It is just that on unix, the fileroutines will be defined as utf8string So you are going to convert in non utf8 unix?

Maybe I should have said "in the native encoding" then. So if the it's a utf-16 unix it will be utf-16. In will have to see how this fares with the shared character of the unix rtl.

Note that both is also possible, e.g for most used string routines (like extractfilename etc) they can be simply overloaded to support both.

Florian Klaempfl wrote

Because using utf-16 on linux is very unnatural, same for utf-8 on windows. Platforms like go32 even don't have any unicode. Coding platform independent but fast applications is really ugly having fixed types.

Well, then you mean that it requires conversion in some platforms rather then it not being cross-platform.

What I am trying to say is that the new proposed systems will be harder to use, trying to please everyone everywhere with a perceived performance gain without any indication that this gain will actually be significant in real world applications. It uses an exotic solution, never tested before.

The speed difference in LCL-Qt apps and LCL-Gtk apps is negletible, althougth we do string conversions when using Qt. Because the manipulation of strings is usually not a bottleneck.

And the ansi routines will not be removed, they will be kept for those really interrested in speed.

I for one prefer simplicity and easy of use to speed.

I mean Unixes with iso encoding and not utf-16

C/C++ support the native encoding on all platforms.

The speed difference in LCL-Qt apps and LCL-Gtk apps is negletible, althougth we do string conversions when using Qt. Because the manipulation of strings is usually not a bottleneck.

That's because it doesn't do that much string processing, compared to e.g. iterating through a db-export and transforming it. That should be the norm for a native unicode type, not an UI.

I don't see what is difficult about Florians proposition. On the contrary, it is the simplest possible solution, and quite elegant in my eyes.

For the LCL/fpGUI/MSEGui programmers, nothing changes, you can even throw away your own conversion routines. You need only a single call just prior to passing a string to the OS/GUI system: ForceEncoding(). No ifdefs needed, all is transparant.

Michael.

In my opinion: no automated support. Simply since the target "encoding" can't represent all characters. In theory one could throw an exception, but that would only require to guard all string routines in even more exception handling, moreover there is not really an alternate path to take then.

So all ansistring to unicode handling must be done by proper conversions procedures, manually.

And I assume long term, these will die out anyway

On Tue, 1 Jul 2008, Felipe Monteiro de Carvalho wrote: I don't see what is difficult about Florians proposition. On the contrary, it is the simplest possible solution, and quite elegant in my eyes.

To be honest, I flabbergasted that the two of you agreed on such a runtime construct. It goes IMHO against Pascal principles.

That's one of the problems. Having to check and insert code for a Tiburon solution. (that simply will expect UTF-16). The least it should do is have a way to flag a routine (e.g. by directive/ Tiburon mode) to only accept UTF16 and insert that call itself.

Marco van de Voort wrote

C/C++ support the native encoding on all platforms.

I did some googling and they don't support unicode filenames. So we are back to zero systems using this method again =)

http://www.google.com/search?q=C%2B%2B+unicode+filename&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:pt-PT:official&client=firefox-a

I think that we should keep in mind that we already have a fast ansi set of routines which fits the operating system encoding (but is never utf-16).

What we wish to have is a unicode set of routines, the ansi routines will keep existing if you really need speed.

On Tue, Jul 1, 2008 at 10:50 AM, Felipe Monteiro de Carvalho

I did some googling and they don't support unicode filenames. So we are back to zero systems using this method again =)

Actually I think that Carbon uses a system very similar to the one proposed by Florian. The string is an opaque type, which can be in any encoding. All string routines uses this opaque type and if you wish to get the string contents in a certain encoding you use a routine for that.

But Carbon is not cross-platform.

A very similar system is used in Cocoa, but the string is a class there.

Why ?
In your opinion, we must get rid of Array of Const, Variants as well, as well as RTTI ? They all serve the same purpose.

No-one will be forced to use the new type, so...

Let's first wait to see what Codegear comes up with, and then worry about compatibility. In my opinion we'll have to write exactly 0 lines of code for this compatibility, with the solution of Florian.

Michael.

If compiler magic is at work, wouldn't all this reduce to s[1] giving the first char no matter the char size? If you do something like c := s[1] and c is defined as char, it gets converted to a standard 0-255 value, but c could be defined as FlorianChar and be the native char size. Or am I smoking crack?

Jeff.

No, you understand it correct.

Obviously, with Florian's type, simple low-level access is out of the question. That's the price you pay.

Michael.

Those are explicitely meant as layer over existing compiletime systems to handle exceptions where that is not possible. (respectively normal parameter arrays, normal typed vars and pointer based method execution)

Here we are talking about adding runtime construct without typed alternative.

No-one will be forced to use the new type, so...

No one is forced to use FPC, but that is also an open door. We both know that this is a pretty crucial decision, since the only alternative is handcoding using pointers.

Codegear has an UTF16 type, for .NET compability. See also http://blogs.codegear.com/abauer/2008/01/28/38853

In my opinion we'll have to write exactly 0 lines of code for this compatibility, with the solution of Florian.

Why don't you point out what I misunderstood above? If I only have a UTF16 routine with a parameter of type "unicodestring" (like T. has), how do I make sure that it only gets passed UTF-16 system without code changes?

Marco van de Voort wrote: > If compiler magic is at work, wouldn't all this reduce to s[1] giving the first char no matter the char size?

Where does the "magic" gets its information is my point.

Huh, since when is [] lowlevel access? It is a normal string operation. And Tiburon will keep supporting it.

You can still do C:=S[i]. What you cannot do is
  P:=PChar(S);
  While (P^#0) do
   SomeByteSizedOperation;

if the string type has some multibyte values, obviously the code for

  C:=S[i];

will be rather cumbersome. But it will guarantee you the I-th character from the string. Since C will be FlorianChar, it'll be at least an integer (2 bytes encoding info, 2 bytes actual value)

Michael.

I described this already in detail in my first mail: just in one of the four bytes available for storing the encoding.

Why you cannot? PChar(S) should represent S as raw bytes. If you know what you are doing - it will not harm. In other case, if you corrupt the string then you are responsibile for all problems you get.

Best regards,
Paul Ishenin.

Obviously you can :-)
But what I meant was that you shouldn't expect old code that relied on 1-byte characters to work.

Michael.

It is supposed to break on utf-xx or whatever anyways.

I think you can still do the byte-size operations this way:

ForceEncoding(S, iso-xxxx)
P:=PChar(S);
While (P^#0) do
 SomeByteSizedOperation;

Similarly for any other code supposing an encoding.

Absolutely.

Michael.

Would this new multiencoding string replace a reference counted widestring type on Windows?
I'd like to repeat the need for a "as fast as possible" (reference counted) widestring on all platforms which offers all possibilities of optimized low level pointer stuff like widestrings on Linux, which are ideal for MSEgui unicode handling.

Martin

Zitat von Florian Klaempfl :

The above works normally for UTF-8. UTF-8 was designed for this. That's why most ansistring code works with UTF-8. Switching to UTF-8 was easy. Switching to UTF-16 needs more work.
And a multi encoded string will break even more things. Means: more work. The question is: how much more?

Mattias

Zitat von Martin Schreiber :

Ehm, do you know, that UTF-8 has the advantage, that many ascii functions work without change?
For example ReplaceChar or searching a substring?

Mattias

What will break? As I said, the tflorianstring manager will get some variables which allow to controll the behaviour of this string. For example you could tell it that all strings should be utf-8 encoded. Of course, you get into trouble if some user plays unfair but you could still protect your code with some EnforceUTF8Encoding. It's exactly the same as with the current lazarus solution. If the user messes with the abused ansistrings, you're in trouble but with the tflorianstring you have a runtime mean to detect the mess (wrong encoding).

Sure, but for layout calculation and the like we need fast access to codepoints.

Martin

Mattias G?rtner wrote: example you could tell it that all strings should be utf-8 encoded. Of course, you get into trouble if some user plays unfair but you could still protect your code with some EnforceUTF8Encoding. It's exactly the

See earlier mail. Tiburon code shouldn't need mods. That will make sharing projects with Delphi too difficult. (and make FPC look cumbersome).

Note that while I don't like the polymorphic string, it doesn't have to kill that, just in some compiler mode (possibly Tib compat) it should be possible to control the encoding that goes into routine. IOW if a module is declared in this mode, the compiler will call the force routine before the call or in the routine.

Florian Klaempfl wrote:

[..some of my thoughts..]

this suits a construct I saw somewhere:

type
   SomeString = type String(CP_KOI8);

Marc

The only way to be sure is using utf-32 in this case. (or not supporting unicode)

Marc

I'd like to repeat:
We talk about the MSEgui framework here, not about FPC RTL or FCL. In MSEgui we need fast internal string and character handling routines which support UCS-2. UCS-2 is enough even for our single active Chinese user I know of. I don't want to slow down MSEgui for 100% of the MSEgui users because of the theoretical possibility that someone needs code points which don't fit into the base plane. If someone needs the whole unicode range he can use surrogate pairs. They will not show correct on screen, but all other tasks can be done. It is the same situation as with ansistring/utf8string. The use of 16bit instead of 8bit as storage base of the MSEgui string representation has the big advantage, that 100% of the MSEgui users can access characters by a simple linear index. Because MSEgui is mainly used by Russian speaking people, this would probably be less than 20% in case of 8bit. Most of the European users wold be out of luck because of the umlauts and accents.
Another need of the MSEgui users and the MSEgui routines is converting internal string representation to the current 8bit system encoding. FPC supports this perfectly by the widestringmanager already. Xlib and gdi both have a widestring interface. The only drawback I see is that there is no reference counted FPC widestring type in Windows at the moment. The upcoming new Delphi version uses a simple reference counted widestring as string base type too AFAIK.
So if FPC decides to implement a referencecounted widestring on Windows for Delphi compatibility, it should be available in OBJFPC mode too. Conclusion:
MSEgui, and propably most of the MSEgui users too, has no need for a multi encoding string type at the expense of slower code and more memory consumption, a referencecounted widestring on Windows would be enough.

Martin

Can you point me to an example function, where this is critical?

Mattias

This isn't the case with florian's type?. Because the first copy from a source with an other encoding would force it to the encoding of the source?

For example lib/common/kernel/msedrawtext.pas:223, procedure layouttext.

Martin

Zitat von Martin Schreiber :

Nice code.
As far as I can see, it handles tabs, linebreaks, c_softhyphen and charwidth. It uses single array element per character optimizations, like the charwidths array.
I think I would simply keep that and define the characterwidth for the follow up elements as 0.
Then you only need to change the places where you check for the c_softhyphen. And because this is a constant you can even use some tricks here. I don't see how this have a big impact on the performance.

Mattias

The code is complicated enough, don't you think? ;-)

Martin

Zitat von Martin Schreiber :

Ah, sorry, now I understand.
You meant, the performance penalty of the *programmer* is not negligible comparing ASCII/UCS-2 and UTF-8/16.
Yes, it's true, that for layout code it is often better (readability, maintance) to refactor the code first, before extending it for UTF. Especially if it is optimized code.
BTW, where is the layout code that handles Right-To-Left, kerning and sub pixel rendering?

Mattias

:-))))

Not supported. For wisiwig layout calculations of Truetype and Postscript printer fonts a scaled up shaddow font is used for emulation of sub pixel placement.
Full Unicode handling is much more than to have the ability to encode all possible Unicode points...

Martin