Hi,

Please check this discussion:

http://community.freepascal.org:10000/bboards/message?message_id=172880&forum_id=24092

Short summary:
* Many places in the rtl use single character strings, i.e. ansistrings.
* To make them Unicode proof they need to be changed into wide strings.
* But this could break a lot, and add a lot of overhead.

To be short, Juras B. wants to add a Unicode Win32 target, so in the standard RTL things like Tlist etc. use ansistrings, while in the Unicode RTL they use widestrings.

IMHO this is indeed a good solution, but one with consequences. Could somebody give his opinion?

Daniël

IMHO the problem is that the principle of the target dependant dir structure is that there is an separation between OS and non-OS dependant code, allowing to reuse the OS independant code.

However this doesn't work fully this way, since unicode internally also hits OS-independant code hard. IOW the separate target only solves the windows unit defaults.

Op Wed, 16 Nov 2005, schreef Marco van de Voort:

Indeed. What is proposed is that the system independent RTL uses an abstract string type (Tnativestring) which is widestring or ansistring depending on target.

What should be done on Linux/FreeBSD/MacOS is still unknown to me, it is a wild west, but likely something similar, internally a widestring rtl is used that converts to the right encoding when communicating externally.

Daniël

Does that mean string operations will be twice a slow?

Vincent

Daniël Mantione wrote

To be short, Juras B. wants to add a Unicode Win32 target, so in the standard RTL things like Tlist etc. use ansistrings, while in the Unicode RTL they use widestrings.

Why not use ansistrings with UTF-8 ?

IMHO this is indeed a good solution, but one with consequences. Could somebody give his opinion?

You postpone a decision that no one wants to make. It is not a solution to anything. The RTL will have duplicate code. I don't like it.

Micha

That only solves the typing. Not the duplication of all string code in the RTL. (accessing strings with [] e.g.), unless you choose a fixed width character encoding and semantics that are 100% ansistring compatible.

What should be done on Linux/FreeBSD/MacOS is still unknown to me, it is a wild west, but likely something similar, internally a widestring rtl is used that converts to the right encoding when communicating externally.

Problem is that this is a costly system that decreases string performance significantly. Moreover I don't like solutions that don't tackle all platforms at once, since that means the other platforms might be pushed into an unsuitable model when they start implementing.

Daniël Mantione napsal(a):

Big overhead (double maintenance efforts for all targets supporting this schisma). :-( I'd say it's better to successively identify the weak points and address these case by case.

Tomas

Op Wed, 16 Nov 2005, schreef Tomas Hajny:

Big overhead (double maintenance efforts for all targets supporting this schisma). :-( I'd say it's better to successively identify the weak points and address these case by case.

I know, I'm all for abolishing Chinese (and perhaps Korean), the only language(s) that absolutely cannot be written in with an 8 byte code.....

However, it won't work that way in this world and also people in other scripts tend to like Unicode.

Now, take for example the FCL. Tstringlist uses ansistrings, for example.

Now there are several solutions:
1. Make a Twidestringlist
2. Make Tstringlist use widestrings internally and add methods for both ansistrings and widestrings.
3. Make an FCL with ansistrings and an FCL with widestrings.

I have been convinced that 3 causes the least maintenance trouble and the least overhead for people that don't need it. The reason is strings are used everywhere in about any library. It cannot be reduced to a few weak points :/

Daniël

Op Wed, 16 Nov 2005, schreef Micha Nelissen:

Because then you will have to modify routines like pos, insert, delete. Since that is not possible, you would get a pos_utf8, insert_utf8, etc.

I don't like it either.

Daniël

Op Wed, 16 Nov 2005, schreef Vincent Snijders:

I don't think so. A string copy operation for a wide string needs to move twice as much data as a string copy operation for an ansistring. However, when walking through it, (take the widestring version of pos for example) the amount of work per character does not change.

The big trouble is the conversion stuff when you need communication with the outside world.

Daniël

You have to anyway. Windows XP+ supports surrogate pairs (IOW true UTF16) see http://www.unicode.org/faq/utf_bom.html#UTF16

One could argue that codepoints over 40k are rarely used, but if we really do major work like this, it should be done good. And it should be a good compromise for all OSes, not just windows.

Daniël Mantione wrote

I know, I'm all for abolishing Chinese (and perhaps Korean), the only language(s) that absolutely cannot be written in with an 8 byte code.....

why not remove all languages that do not fit ASCII next???

Now seriously.

Why not use 1? It can be implemented on a separete unit (wideclasses or something like that) so it won't introduce overhead for programs that don't use it. It can be implemented by those who need it.

Yes.

That's what's the use of polymorphism.

Did you ever release building, try to get regression test results in shape etc.?

The reason is strings are used everywhere in about any library. It cannot be reduced to a few weak points :/

I think it can.

No, why? When working with utf-8 strings, you don't use character positions.

Modern CPUs are memory bandwidth limited usually so widestrings a lot slower.

Afaik, on Mac OS X internally pretty much everything is UTF-8.

Jonas

Well, then we need some Chinese or Korean people who contribute the necessary code :)

Op Wed, 16 Nov 2005, schreef Florian Klaempfl:

pos('ë','Daniël');

... has a different implementation for utf-8 and 8-bit code pages.

Daniël

Dani?l Mantione wrote:

y> >

That's pretty much only when using utf-8 in the error handling only (iow localising UI and errormsgs etc).

However if you use strings in your business logic it changes.

Why? With utf-8 a string is searched, with 8-bit cp one char. No other char/sequence of char other than ë can generate the byte sequence representing ë

const s : 'Dani?l';
var accent : utf8char;
x:=pos('i','Dani?l');
accent:=s[x+1];

Well, I didn't suggest abolishing of any languages (or maybe... - C#? ;-) ).

However, it won't work that way in this world and also people in other scripts tend to like Unicode. Now, take for example the FCL. Tstringlist uses ansistrings, for example.

This is one particular weak point which needs to be addressed, IMHO.

You're right that strings are used everywhere, but I don't think that this really means that you need to add special support for widestrings everywhere. In many places you can pass a DBCS/MBCS string to it today (e.g. encoded using UTF-8) and it wouldn't cause any harm. From my point of view, you need some kind of special support mainly for sort operations (which includes your TList) and then for visual classes (length of text for controls, etc.). In addition, you certainly need to have a proper routines for I/O. However, e.g. your particular example in the forum discussion is IMHO conceptually wrong. Turning a string around just cannot be performed this way (this is unsupported by design for DBCS/MBCS texts; not even mentioning the fact that the example is "somewhat" artificial). People who want to perform such an operation need to analyse and design the implementation properly, probably by translating the ansistring to a widestring first in this case. How this translation is performed is another question and it depends on programmer's decision. It could be that the string already _is_ an UCS2 string (and "translation" to widestring means that you just copy it byte by byte), it could be UTF-8 and it could be even a simple string created in particular codepage (SBCS). This is programmer's decision (trade-off between the widest support and the best performance); the same way that he has to decide whether he'd use multi-platform APIs or native API of a particular platform, or whether he'd use/import XxxxW or XxxxA API function for his Win32 application.

Maybe I'm still overlooking the real issues. Please, give me more concrete examples which cannot be resolved at the moment, we could discuss them (and then possibly come to a conclusion that separate RTL would be better/necessary).

Tomas

We've already had a (South) Korean user at least (under OS/2 target). However, I suspect he wouldn't be really interested in supporting DBCS in FCL. ;-)

Tomas

Marco van de Voort napsal(a):

We could have special support for assignment to type utf8char, couldn't we?

Tomas

It would be horribly slow, since this would apply to length too, and think of while i<length(x) do inc(i); like constructs.

I think the avg delphi code simply assumes 100% that chars are fixed width.

Marco van de Voort napsal(a):

I'm afraid that you don't get too far with that assumption. "Existing Delphi code" most probably isn't DBCS/MBCS safe.

Regarding constructs like "while i<length(x) do" - I'd say that most common use of these are comparison, copying, translation to uppercase/lowercase and combinations of these. All these operations should be performed using dedicated (RTL) functions, otherwise they will fail in DBCS/MBCS environment anyway (or at least result in suboptimal implementation).

Tomas

But still, only if the string operations are a bottleneck. That might not be the case.

Op Wed, 16 Nov 2005, schreef Tomas Hajny:

*Sigh*, this going to be a long e-mail for a subject I don't interrest myself not much. Here we go.

There are a few models you can use:

Model 1: Be ignorant about multibyte character sets. ----------------------------------------------------

UTF-8 was designed to behave well with programs that assume US-ASCII, therefore you reasonable results.

If you assume nothing about the ordering of characters in the string, do not try to break it into pieces, do not modify them (i.e. uppercase), things work out in many situations.

The limitations of this model is that there are situations were the ordering is important, strings need to be broken up into pieces etcera. Reversing a string is an extreme example where strings need to be broken into pieces, but there are way more examples.

Obviously, if code should be ignorant about the charset, people wouldn't be asking about Unicode support.

You can also be partially ignorant about charsets. I.e., you leave pos, insert etc as is and leave it up to the programmer not to do tricks like reversing strings.

In the case you are ignorant pos('ë','Daniël'); is a substring search of a string of 2 bytes into a seven byte long string.

Model 2: Use an internal encoding
---------------------------------

The UCS-2 widestring stuff is an example of this. You could also use UCS-4. The advantage is you can do any operation on the string you like, you do not get into trouble. Trouble is a lot of conversions when talking with the external world.

In this case pos('ë','Daniël'); is a widechar search into a widestring.

You can also decide that your internal encoding is UTF-8, no problem. In that case pos('ë','Daniël') is a widechar search into an ansistring.

The desirability versus UCS-2 and UTF-8 is a matter of taste. You can walk through UCS-2 strings with for-loops. You cannot do this with UTF-8, unless we would implement [] in O(n) time (or you are ignorant, model 1).

With UCS-2 you can reuse a lot of code by just changing the string type. You must include all stuff twice (either by making a separate Unicode rtl, or doubling code in your units).

With UTF-8 you can reuse a lot of code that is ignorant about character sets.

However, there is one big caveat. It is up to one level.

Take the Tstringlist. We can make a Twidestringlist, or we can add methods that do UCS-2/UTF-8 sorting and other operations.

The consequence is that all the code that uses the Tstringlist, must make a difference between 8-bit and UTF-8. Any code that uses Tstringlist should decide wether it is going to call the 8-bit or UTF-8 methodes. It is even worse, if that code is to be reused it needs to provide the option to the programmer as well, in other words, it needs to have both 8-bit and utf-8 methods as well.

In other words, you still need to duplicate an awfull lot of code.

What convinced me two rtl's might be a better choice, is that many of the source code remains intact and does not need to be duplicated. New code could take advantage immedeately. The decision wether the code is going to be used in an 8-bit environment (i.e. MS-DOS) and will be 8-bit, or in a Unicode environment (i.e. Windows NT) and will be 16-bit a character, is solved by a few ifdefs. There won't even be any overhead on the MS-DOS executables (allthough the programmer can use widestrings if he wishes so).

Daniël

Delphi named these AnsiPos, Ansi..., no?

LOL! First you say we can't use utf-8 strings because we need to introduce pos_utf8 etc., but then you propose to introduce a crippled string type which is not standards compliant, cannot represent all asian charsets and cannot be implemented well on anything non-win32 *and* you propose to introduce procedures for this new string type after all :-D.

Where's the logic in that?

I think it's better to keep with the ansistring which either has specific charset ISO-8859-X (so the programmer uses regular Pos), or has UTF-8 and programmer uses AnsiPos etc. TStringList can then simply keep using ansistrings.

Micha

Op Wed, 16 Nov 2005, schreef Micha Nelissen:

I propose nothing, I'm trying to help someone else. It is better to stop discussing I think.

Daniël

pos('ë','Daniël'); ... has a different implementation for utf-8 and 8-bit code pages.

one little desgin feature of utf-8 is that is was carefully designed to be friendly to byte-orientated code. No special precautions are needed for substring matching in utf-8!

Op Wed, 16 Nov 2005, schreef peter green:

Which is the "be ignorant about multibyte character sets" model. Nothing wrong with that model, but it has its limitations.

Daniël

Why? What he says is correct?

UTF-8 however is far more friendly to that model than most legacy multibyte character sets. Most importantly you CAN'T get a false match when doing byte-orientated substring matching on utf-8 strings and if some code does chop a UTF-8 string mid-character only the chopped character will be lost.

I don't understand, why you connect UTF8 with 'ignorant of MBCS'. UCS-2 can be used as ignorant as UTF8.
Even UCS-4 and UTF32 will not solve all problems. Think about arabic RTL.

You must extend old souce code if you want to support all languages anyway. Widestrings lets you keep some old code and introduces some new problems. The same is true for UTF8.
That's a matter of choice and heavily depends on the old code. What's more important, is that widestring needs sometimes two widecharacters for one character. So, you have MBCS problems too. For Lazarus we decided to use UTF8, because
- UCS-4/UTF32 is too much memory overhead. That means we must use a 1 or 2 byte encoding, which implies, that we have to implement MBCS functions anyway.
- UTF8 works with ASCII without conversion

That is the same for 8bit and widestring.

Please: No two RTLs.

Mattias

Op Wed, 16 Nov 2005, schreef Florian Klaempfl:

*sigh* Yes, what he says is correct. Now to do something with strings. I.e. reverse them, or any other operation that needs to split the string into pieces.

Try to do any operating with has to do with the order of characters (i.e. compare strings).

Now the model "be ignorant" no longer works.

If all you did need to do was nothing, people wouldn't be begging for Unicode support, right?

Daniël

*sigh* Yes, what he says is correct. Now to do something with strings. I.e. reverse them, or any other operation that needs to split the string into pieces.

reversing a string properly requires a very deep understanding of unicode and huge lookup tables (reversing the code point order will break down when combining characters come in). Lukilly it is very rarely required outside toy examples.

tokenising a UTF-8 string does not require any special precautions because substring matching doesn't require any.

blind truncation (say to fit a length limit) can cause partial characters on the end but the nature of utf-8 means that theese are much less of a problem than with more traditional multibyte encodings and are easy to clean up later.

Try to do any operating with has to do with the order of characters (i.e. compare strings).

sorting/comparing utf-8 strings with a byte-orientated sort routine will have the same effect as sorting/comparing by unicode code point. If you wan't a culturally appropriate support you will need special code and you would have done even with a single byte encoding in most cases.

If all you did need to do was nothing, people wouldn't be begging for Unicode support, right?

ms made the understandable but highly annoying descision to use a totally seperate unicode api. This means that apps that don't use unicode are limited to whatever ansi code page the system is using (which btw can be multibyte and a lot nastier to nieve apps that utf-8).

Op Wed, 16 Nov 2005, schreef Mattias Gaertner:

I don't understand, why you connect UTF8 with 'ignorant of MBCS'. UCS-2 can be used as ignorant as UTF8. Even UCS-4 and UTF32 will not solve all problems. Think about arabic RTL.

Sure. I am not against UTF-8 or something (if you got that impression). What I did note though is that adding UTF-8 (or widestring, it doesn't matter) would mean doubling a lot of code, while with a separate RTL would prevent, there would be single code that can be compiled for multiple targets.

> In other words, you still need to duplicate an awfull lot of code. That is the same for 8bit and widestring.

No, that is not true. There would be two rtls based on the same code.

A separate RTL is not my proposal, but the idea isn't that stupid.

I am not going to push anything.

Daniël

Can you give some examples, what parts of the RTL should change for widestring?

Mattias

All typecasts to OS parts, libraries and functions. E.g. accessing mysql is now done with pchar(ansistring)

And they should be replaced by what?

Mattias

Op Thu, 17 Nov 2005, schreef Mattias Gaertner:

Pchar(Tnativestring(widestring_variable));

Daniël

Op Thu, 17 Nov 2005, schreef Daniël Mantione:

Apparently I got too little sleep or something tonight :) It would be this:

Pchar(ansistring(wide_string_variable));

If the RTL would then be compiled without a {$define unicode} the ansistring() would be a dummy.

Daniël

This makes no sense. If such a thing is done, the encoding of the actually used mysql db must be taken into care.

Nothing. You can't. It was meant to illustrate that there is more to it than simply declaring an "tbasicstring" or so alias and fixing some internal routines. It will "just" mean full checking and ifdeffing of/in every part of the entire fpc/lazarus repository, and maintaining that, the added support burden etc.

Op Thu, 17 Nov 2005, schreef Florian Klaempfl:

This makes no sense. If such a thing is done, the encoding of the actually used mysql db must be taken into care.

Nothing is fun with charsets... The above just demonstrates the idea. File system encoding would be another of those wild west scenarios.....

Just to repeat, be ignorant about charsets is by far the most convenient scenario. For us, that is. Providing an environment were people (i.e. programmers/end users) don't need to worry about charsets is a different philisophy. That is what is being asked (by some users). Lets stop confusing those.

Daniël

:)
That's exactly why I asked the trick questions.

Mattias

Hi

Following this discussion, I want to throw in my 2 cents as well: On a real long term (like 5 or 10 years from now), the solution should be as "clean" as possible with as little awkward parts because of backward compatibility. This favors of a more separate solution with a compatibility layer. Sure enough, this means more work to set it up and maintain it. Therefore, it will take longer to have it running, but in the end it will prevent the situation, I'd like to call the A20 gate situation.

Michael

From: "Marco van de Voort"

When unicode RTL will be used with existing non-unicode applications or libraries there will be a problems in any case.
But it does not mean that unicode RTL is not RTL is needed for people who want to develop unicode applications from the beginning. In such case the developer will be aware of unicode and will write correct code.

There is no magic solution to make unicode RTL compatible with existing apps or libs. Unicode RTL will be an option for people who really need it.

IMHO RTL code base must be the same for ansi and unicode. To build unicode RTL some define must be used.

Yury Sidorov.

Who knows the future?
Maybe in 10 years 16bit non multi 'byte' encoding is sufficient for all remaining languages.
Or maybe 32bit encodings will become the default.

Speaking for lazarus: we want to support the whole unicode and UTF8 is the easiest to achieve that.

Mattias

Correct.

All correct, BUT

1) it is not Unicode RTL but Unicode FPC/Lazarus.
2) it is not backwards compatible. We have to take care of more people's need than just the ones that urgently need Unicode to be as easy as possible.
3) thus creating an huge support problem

4) as mentioned before (and forgotten by me earlier) even UTF32 is not 100% fixed width

There is no magic solution to make unicode RTL compatible with existing apps or libs.

Correct. But what is then the use of stuffing it into the entire codebase? That extra work is then useless

Unicode RTL will be an option for people who really need it.

Unicode is. I'm not sure "Unicode RTL" is.

IMHO RTL code base must be the same for ansi and unicode. To build unicode RTL some define must be used.

As said, since legacy RTL code doesn't become magically unicode, maybe it is better to keep it separate, and overload a few funcs (like filesystem) if necessary at all (on *nix, use utf8).

Op Thu, 17 Nov 2005, schreef Mattias Gaertner:

Deciding between UCS-2 and UTF-16 could be left up to the implementers of such a runtime library; all special UTF-16 positions are unallocated in UCS-2, so they do not bite each other. IMHO the advantage of UCS-2 is that you can use the [] array index safely. Or can indeed decide that you want a super wide string... Variable length strings have their problems tough, I've got some problems in my collection which would be quite hard to solve Unicode proof with UTF-8 strings.

I checked the situation a bit. There is one alive language that has symbols allocated beyond plane 0 and that is not surprisingly Chinese. These are 40000 words allocated in plane 2, archaic words though. Dead languages are allocated in plane 1.

You can decide to support them, or you can not. This is also up to font designers, try if you can render #&173733; in your browser. Mine did not in several fonts.

Speaking for lazarus: we want to support the whole unicode and UTF8 is the easiest to achieve that.

Fair enough; there is nothing wrong with that. What you need to consider though is the following:

* How many Delphi apps in the wild are Unicode proof?
* How many apps would be Unicode proof if the libraries would use widestrings?

That is what is IMHO going on in the minds of people that are asking for this.

So, I can tell Juras B. to go whining somewhere else and not bother, but I cannot deny validity of his points.

Daniël

From: "Marco van de Voort"

I agree. There is no need to make existing RTL APIs unicode aware, because there will be no compatibility with existing apps in any case. So how about to add alternative unicode versions of RTL APIs via overloading or via adding some suffix (eg. Assign and AssignW, TStringList and TStringListW - like in WinAPI)?

There are following benefits in such approach:
1. It will be possible to develop unicode apps using these APIs.
2. There will be only one RTL.
3. There will be single source tree.
4. It will not break existing RTL code.

Yury Sidorov.

Op Thu, 17 Nov 2005, schreef Yury Sidorov:

If you look at the discussion I tried to convince Juras that such an assignw solutions was best. However, he convinced me instead of that I convinced him.

IMHO the "hope" is that if there is a Tnativestring, people will start to design their libraries using Tnativestring, which will result in that those libraries can be compiled for 8 of 16 bit strings.

With assignW, TstringlistW etc., that won't happen, i.e. everyone who writes a library has to decide if he uses 8 bit, 16 bit or both code paths. Practice will be that people design libraries with 8 bit in mind, if you are lucky utf-8 aware. People who have an interrest in Unicode would have to add Unicode support to way more code than they can manage.

Daniël

Mattias Gaertner wrote

Speaking for lazarus: we want to support the whole unicode and UTF8 is the easiest to achieve that.

I particulary like this solution.

* It doesn´t break existing code
* It makes it easy to make a program unicode. Just change the encoding to utf-8!
* It includes no overhead to existing apps

Only a few RTL functions would have to be created to support utf-8, as has being said here. Many functions could remain the same.

The programer should know what he is using and take the necessary precautions (use special utf-8 function for example, if needed)

Daniël Mantione wrote

IMHO the "hope" is that if there is a Tnativestring, people will start to design their libraries using Tnativestring, which will result in that those libraries can be compiled for 8 of 16 bit strings.

This does not fix everything. Some RTL functions would still have to be modified even with this Tnativestring.

I think this can lead to slower compiles and overhead on the RTL.

----- Original Message -----
From: "Felipe Monteiro de Carvalho"

I think RTL functions can use UTF-8 stored in regular ansistrings. But string type name must be different from String or AnsiString. It must be named like Utf8String or StringUtf8 to specify that the function or procedure is UTF-8 aware.

Also conversion between WideString and Utf8String need to be transparent (automatic UTF-16 (UCS-2) <-> UTF-8 conversion).

This will allow to assing Utf8String to WideString and perform indexing or per character operations easily if needed. Also developer can work only with WideStrings in his application and pass them to functions that accept Utf8String and that will work good (in non time critical places of code of course).

Yury Sidorov.

Op Thu, 17 Nov 2005, schreef Felipe Monteiro de Carvalho:

The RTL needs modifications in any solution.

I think this can lead to slower compiles and overhead on the RTL.

There would be consequences, even big ones, but not these.

You won't get slower compilations; it doesn't matter much if the compiler has to consider a type Tnativestring or ansistring. Overhead is zero for non-Unicode rtl. Overhead would be huge when interfacing with non UCS-2 stuff.

The widestring is slower than ansistring, but handling utf-8 inside ansistring can turn O(N) into O(N^2), which has terrible effects on performance.

No, I think implementation of this could put a big wall between Pascal and external libraries. That is going to hurt the most. Inside Pascal it could be a very comfortable solution.

Libraries important, very important. It might result in more Pascal libraries, but also makes using foreign libraries more difficult. For the Unicode RTL only, then, but I don't get comfortable with the idea that libraries get hard to use.

Daniël

Who knows the future? Maybe in 10 years 16bit non multi 'byte' encoding is sufficient for all remaining languages. Or maybe 32bit encodings will become the default.

the unix and web worlds are going utf-8 the ms and java worlds are going utf-16 both are variable width