Hello,

we have placed a new major release of the Free Pascal Compiler, version 2.4.2 on our ftp-servers.

Changes that may break backwards compatibility are documented at: http://wiki.freepascal.org/User_Changes_2.4.2

Downloads are available at:

the main FTP server at

ftp://ftp.freepascal.org/pub/fpc/dist/2.4.2/

and mirrors like

ftp://freepascal.stack.nl/pub/fpc/dist/2.4.2/
ftp://ftp.hu.freepascal.org/pub/fpc/dist/2.4.2/

and sourceforge.

Enjoy!

The Free Pascal Compiler Team

Free Pascal Compiler

Version 2.4.2

****************************************************************************** What's New in 2.4.2
******************************************************************************

Free Pascal 2.4.2 contains many fixes and a few new features. Most bugfixes in the RTL and packages before june 2010 have been merged.

Please also see http://wiki.freepascal.org/User_Changes_2.4.2 for a list of changes which may affect the behaviour of previously working code, and how to cope with these changes.

Some highlights are:

Compiler:
* Support D2006+ FOR..IN, with some FPC specific enhancements. Refer to http://wiki.freepascal.org/for-in_loop for more information
* Support for sealed and abstract classes.

Packages:
* Many improvements to the XML units
* Many improvements to the database units
* Many improvemnets to the chm units

Platforms:
* Long term bug in OS/2 implementation of unit Video finally fixed which among others allows inclusion of the text-mode IDE (without debugger) for this platform as part of the distribution again.

See http://bugs.freepascal.org/changelog_page.php for the list of reported bugs which have been fixed in this release.

Congratulations!
It was a hard work.

Marcos Douglas

we have placed a new major release of the Free Pascal Compiler, version 2.4.2 on our ftp-servers.

Great :-)

A few notes on enumerators:

What restrictions apply to enumerators in general? IMO the loop variable should be treated as "const", any change to an enumerated structure (tree...) may result in obscure bugs. Like traversing strings seems not to allow to change the string, by e.g. replacing '/' by '\'.

In the array enumerator, is it possible to use an pointer into the array, instead of a copy of the element? This would allow to use arrays of Class, Record and Object types in the same way (by reference).

The comparison in the UTF-8 string example is very questionable. First ch(i) is not equivalent to ch, not even closely related, and the claim of O(N^2) operations deserves an proof - IMO it's simply wrong.

In general, what's the benefit of using enumerators? IMO a for loop executes faster on (linear) string and array types, where enumerator calls occur in for-in (see also my note on the UTF-8 string example).

Thus enumerators are useful in detail with non-linear data structures, like trees, linked lists, datasets(?) or collections.

DoDi

Hans-Peter Diettrich wrote

In general, what's the benefit of using enumerators? IMO a for loop executes faster on (linear) string and array types, where enumerator calls occur in for-in (see also my note on the UTF-8 string example).

I'd say they simplify the code. They might not make the code faster (or even make it slower), but they make it easier to read/understand.

Consider the following:

without for-in:

procedure DoSomethingWithSet(aSet: TSetOfEnum);
var
   e: TEnum;
begin
   for e := Low(TEnum) to High(TEnum) do
     if e in aSet then
       // do something
end;

with for-in:

procedure DoSomethingWithSet(aSet: TSetOfEnum);
var
   e: TEnum;
begin
   for e in aSet do
     // do something
end;

The second one is a bit easier to understand.

Basically for-in is syntactic sugar (like "case of string"), nothing more, nothing less.

Regards,
Sven

I never saw the first post, but I'll reply from here..

If you use full-blown Iterator classes (instead of just for-in style) you get a lot more too:

* full control over iteration
- move forward
- move back
- reset iteration
- peek forward/back
- skip, etc...
* you can extent it with regular expressions to only iterate what is applicable, and what
matches the regex.
* you completely hide the container implementation (TList, TObjectList, TFPList, etc)
and simply supply the developer with a consistent iteration interface. So even if the
container implementation changes, the higher level iteration code will not.

I have implemented such Iterator classes for all the most used container classes in the RTL, and they work a treat!

Yes, this caught my eye as well: O(N^2) seems only the case if "length" would be evaluated every time. Since you should not modify the index (although this is possible with some tricks in Delphi) the compiler should optimize it away.
If you look at the algorithm, however, the statement is correct: it implies/reads: "evaluate length every iteration", whereas "for..in" evaluates the index value only once.

In our previous episode, Thaddy said:

True, but the difference is that in the for..in case it is guaranteed, while in the for ..to case the compiler devel must make sure that it is the special case of uniform iteration.

A more grave reason though is that Delphi does not have low() and high() on sets and a request to add it by me in 2006 was closed with their equivalent of "won't fix".

So without for..in there is no comfortable way to iterate over a set in both fpc and delphi.

2010/11/14 Thaddy :

the O(N^2) stems from the fact that it is hard to get the ith character in a a UTF8String in O(1). Suppose it is o(N), then the loop is O(n^2).

Vincent

"Hard to" is implementation detail and not part of any algorithm. A translation table goes a long way in implementing your example efficiently.

2010/11/14 Thaddy :

I did not have in mind such a sophisticated UTF8 string implementation, that included a translation table for easy indexing.

Vincent

In our previous episode, Thaddy said:

What kind of translation table ? Do you have an example? What is the key and what is the value?

This would apply to zero-terminated C strings, but not to counted Pascal strings. Most probably the text was copied from a C site, and that essential difference was not taken into account in the Pascal adaptation.

This also is a weak argument, since the behaviour of the iterator is not taken into account. When the iterator checks the length before moving to the next element, and the determination of the length would be O(N), then the order of the iterated loop still is O(N^2).

But you spot an essential pro of iterators: the user does not have to know about the implementation of the container, and can not choose a wrong function for the loop termination condition.

DoDi

A more grave reason though is that Delphi does not have low() and high() on sets and a request to add it by me in 2006 was closed with their equivalent of "won't fix".

I wonder how FPC defines low() and high() for sets. The static bounds can be obtained from the underlying enumerated or subrange type, even in Delphi. So it would only make a difference, when FPC sets would track the actual (dynamic) first and last element in an set, with special handling for empty sets where both low() and high() cannot return valid values.

DoDi

With regards to UTF-8 (or other MBCS) strings, what does Length(s) return in these cases? IMO other functions have to be used for the determination of the true character count (as opposed to the char=byte count).

At least the example code has to be made work, i.e. the nonsense statement

   DoSomething(ch(i));

has to be changed into something like

   DoSomething(GetUTF8char(s,i));

before we can can talk honestly about the order of the loop.

DoDi

In our previous episode, Hans-Peter Diettrich said:

See the source.

The static bounds can be obtained from the underlying enumerated or subrange type, even in Delphi.

If defined separately.

So it would only make a difference, when FPC sets would track the actual (dynamic) first and last element in an set, with special handling for empty sets where both low() and high() cannot return valid values.

No, it just handles the values from the basetype (range or enum), they are still compiletime expressions. The main use is that this means that you don't have to recompile a library if sb was foolish enough to define a set of an anonymous type.

In our previous episode, Hans-Peter Diettrich said:

The base size of the encoding (1 for shortstring/ansistring, regardless of encoding, 2 for unicodestring/widestring regardless of encoding is UCS2 or UTF16).

The other of the algorithm is then still O(n^2), since UTF8Char will already be O(n)?

I agree that a more powerful iterator interface would be a good thing too. However, the clarity and brevity of for..in interface should not be underestimated.
I eagerly await a moment when Lazarus switches to FPC 2.4.2 so I could use for..in in the Lazarus code ;-)
If your extended iterator implements enumerator interface, you can get the best of both worlds, especially if the "index" extension I proposed at the end of the page is implemented.

Yes, that was the idea.
Additionally I hypothesized that if/when FPC supports such type as UTF8String, it will surely have overloaded Length and [] to work properly on it ;-)

I have written this example during for..in loop discussion as an illustration, and did not update it since then.
Indeed, to work in the current FPC for..to loop would be something like

for i := 1 to GetUTF8Length(s) do
  Writeln(GetUTF8Char(s, i));

except I am not sure about the "canonical" function names to work with utf-8 strings.
If the above version is correct, I will update the wiki.

With UTF-8 strings and friends would it not be appropriate to have the loop variable be a string containing the multiple bytes of the Unicode character ?

-Michael

Hans-Peter Diettrich wrote

I wonder how FPC defines low() and high() for sets.

IMHO it should not. An "in" loop on sets should not use a defined sequence. Relying on on an "order" of the elements of a set mathematically is erroneous.

-Michael

Vincent Snijders wrote

I did not have in mind such a sophisticated UTF8 string implementation, that included a translation table for easy indexing.

I don't think you need a translation table to walk through an UTF-8 String Unicode-Character by Unicode-Character (and create an UTF-8 String as a loop variable containing this single Unicode Character in UTF-8 coding, This seems like a simple algorithm (disregarding the notorious Mac-Style double-characters that exist in Unicode independently of the UTF-coding style).

Of course the compiler needs to know that the "string" in fact is an UTF-8 String (I don't know if now there are different types for ANSIString and UTF8String.)

-Michael

Hans-Peter Diettrich wrote

With regards to UTF-8 (or other MBCS) strings, what does Length(s) return in these cases? IMO other functions have to be used for the determination of the true character count (as opposed to the char=byte count).

Of course its possible without a code table (and maybe even desirable) to calculate the length of an UTF-8 String in terms of Unicode Characters rather than in terms of bytes. But to do this the compiler would need to know if in this place the string is an UTF8String or an ANSIString. I don't know if those different types are predefined.

Maybe we will need to wait for this comfort until the "New Derlphi String" implementation (right now only existing in the svn) is up to speed.

-.Michael

Actually, I do not think so. I believe that an integer containing the codepoint is preferable implementation. Either way, of course, the quadratic complexity of for..to loop remains.

2010/11/15 Michael Schnell :

Maybe I did not understand Thaddy, but to give you O(1) access to the ith character, I was thinking about a a translation table of the utf8 string, with key=index (1..length) and value=offset in bytes to the ith character. Such a translation table would need to created once for the complete loop and can be done in O(N).
I cannot imagine another way that a translations table can give you o(1) access.

Vincent

Vincent Snijders wrote

I cannot imagine another way that a translations table can give you o(1) access.

Maybe I don't understand the o(1) correctly. Do you think it should be necessary to access each character in the string with in each iteration in this way.

What I meant was:

A pointer to the first UTF8-Code of the next Unicode character is the internal invisible loop variable.

A string containing the multi-Byte code of the current Unicode character is the visible loop variable.

Constructing the string and the pointer at the start of each loop is straight forward (regarding the UTF-8 definition), fast, o(1) and does not need a table.

-Michael

Alexander Klenin wrote

Actually, I do not think so. I believe that an integer containing the codepoint is preferable implementation.

OK, Unicode always blows up the complexity of the code greatly ;).

Your suggestion would result in an UTF-8 -> UCS32 translation and thus force

Either way, of course, the quadratic complexity of for..to loop remains.

I don't thinks so. see the other mail regarding implementing a hidden loop pointer and a published loop variable (e.g. as an UTF-8 string).

-Michael

2010/11/15 Michael Schnell :

I agree, and that is why you need enumerators to make it work.

Vincent

unintentionally deleted;

..forces the programmer to work with both UTF-8 and UCS32 coded Unicode characters. This might blow his mind even more (regarding that e.g. the Lazarus LCL forces him to work with UTF-8 coded Unicode in a string type called "ANSIString" :( )

-Michael

Vincent Snijders wrote

I agree, and that is why you need enumerators to make it work.

OK, in fact this _is_ an implementation of an enumerator, but same is hidden and so the application programmer is not forced to bother. He just sees the Unicode character in the loop variable (e.g. coded as an UTF-8 String).

-Michael

In our previous episode, Michael Schnell said:

No, since that wouldn't describe the position of that char in the string that is being iterated.

Marco van de Voort wrote

No, since that wouldn't describe the position of that char in the string that is being iterated.

Is this really wanted ?

I suppose this would ask for a full blown iterator....

-Michael

In our previous episode, Michael Schnell said:

Typically I'd iterate by means outside the language (I've used simple iterators based on a record with a few inline methods in the past), and review the places where you iterate by char through strings, and reduce it signficantly.

Since the latter is needed for optimal result in nearly every solution.

Correct. That was my perception.

The total order will be something between O(n^1) and O(n^2), depending on many factors (what is "n"?...).

I also doubt that UTF8char is a reasonable type for the loop variable - IMO UTF32char were a much better choice...

DoDi

In our previous episode, Hans-Peter Diettrich said:

Doesn't matter, utf8 chars or encoding granularity.

I also doubt that UTF8char is a reasonable type for the loop variable - IMO UTF32char were a much better choice...

First you would have to come up with a workable model for s[x] being utf32chars in general that doesn't suffer from O(N^2) performance degradation (read/write)

And for it to be useful, it must be workable for more than only the most basic loops, but also for e.g.

if length(s)>1 then
  for i:=1 to length(s)-1 do
    s[i]:=s[i+1];

and similar loops that might read/write more than once, and use calculated expressions as the parameter to []

Michael Schnell wrote

..forces the programmer to work with both UTF-8 and UCS32 coded Unicode characters. This might blow his mind even more (regarding that e.g. the Lazarus LCL forces him to work with UTF-8 coded Unicode in a string type called "ANSIString" :( )

You can also use the UTF8String type, but the result of this change is only psychological at the moment.

Huh? O(f(n)) has a precise definition, and of course we are talking worst-case complexity here (although average complexity would be the same in this case). n is the string length, measured in whatever units you choose.

I also doubt that UTF8char is a reasonable type for the loop variable - IMO UTF32char were a much better choice...

I think I meant exactly the same thing (an integer holding the codepoint value), but just named the type carelessly. In fact, I think the best name would be something like "UnicodeChar" or even "Codepoint", to avoid association with any particular encoding.

First you would have to come up with a workable model for s[x] being utf32chars in general that doesn't suffer from O(N^2) performance degradation (read/write)

Right, UTF-32 or UCS2 were much more useful in computations.

Okay, essentially you outlined why UTF is not a useful encoding for loop body results in O(N^2) for every loop
type, be counted or iterated, on data structures with non-uniform elements.

UTF encodings have their primary use in data storage and exchange with external API's. A useful data type and implementation would use a SxCS encoding internally, and accept or supply UTF-8 or UTF-16 strings only when explicitly asked for. All meaningful UTF/MBCS implementations already come with iterators, and only uneducated people would ever try to apply their SBCS-based algorithms and habits on MBCS encodings. At least they'd change their mind soon, after encountering the first bugs resulting from such an inappropriate approach.

BTW, I just found a similar inappropriate handling of digraphs in the scanner, where checks for such character combinations occur in many places, with no guarantee that all cases really are covered.

Furthermore I think that in detail Unicode string handling should not be based on single characters at all, but instead should use (sub)strings all over, covering multibyte character representations, ligatures etc. as well. Then the basic operations would be insertion and deletion of substrings, in addition to substring extraction and concatenation.

DoDi

For matrix multiplication there exist algorithms from O(N^3) downto O(N^2.807) [Strassen] and even O(N^2.378) [Coppersmith-Winograd]. Similarly for other problems (hash tables...), where the *amortized cost* can differ from the agnostic "worst case" assumptions.

A "worst case" IMO applies to concrete procedures only, while algorithms or problems only have an upper limit, that can be lowered by further research. Applied to procedures, the coder *establishes* a certain complexity of his implementation, that may be far away from the complexity of the problem to solve.

In any case it's essential to verify the determined complexity, in order to detect inappropriate definitions of N or other factors, that show up as deviations from the assumed formula.

As outlined in another message, (Unicode) string handling IMO should be based entirely on substrings, so that a special single-character data type is not required anywhere. Wirth may have anticipated the problems with character sets in his Pascal design, and did not distinguish between string and character literals.

DoDi

In our previous episode, Hans-Peter Diettrich said:

I said s[x] _returning_ utf-32. I didn't say S was an array of utf32 chars.

And no, IMHO UCS2 as array structure is unwise as it is only a subset, and UTF32 is too wasteful.

expressed with array indexing is not a good idea in the first place. But that might be considered a legacy problem.

Above loop body results in O(N^2) for every loop type, be counted or iterated, on data structures with non-uniform elements.

Yes, but the realisation should be that the holding on array indexing is what makes it expensive. The problem could be strongly reduced by removing such array indexing skeleton from all routines where it is not necessary.

UTF encodings have their primary use in data storage and exchange with external API's.

And in memory.

A useful data type and implementation would use a SxCS encoding internally, and accept or supply UTF-8 or UTF-16 strings only when explicitly asked for. All meaningful UTF/MBCS implementations already come with iterators,

Depends on what you mean with iterators but yes some simple inline routines, one that goes to the next char, and one to load a char will do.

and only uneducated people would ever try to apply their SBCS-based algorithms and habits on MBCS encodings.

That's the main problem. That will have to change anyway.

Furthermore I think that in detail Unicode string handling should not be based on single characters at all, but instead should use (sub)strings all over, covering multibyte character representations, ligatures etc. as well

This is dog slow. You can make such library for special purposes, but for most day to day use this is overkill.

The most common stringoperations that the avg programmer does is searching for substrings and then split on them, something that can be perfectly done in UTF-8.

Then the basic operations would be insertion and deletion of substrings, in addition to substring extraction and concatenation.

Basic operations with capital B yes, string support like in a Basic interpreter.

Indeed. Javascript uses the 'substrings only' approach. Can't get more slow than that :(

Probably the reason they injected regular expression treatment in the language;
To make some things at least acceptably fast.

But in general there is something to say for Hans-peter's statements. Fixed-length character encoding simply make more sense from a computing point of view.

Michael.

Yes, but the realisation should be that the holding on array indexing is what makes it expensive. The problem could be strongly reduced by removing such array indexing skeleton from all routines where it is not necessary.

Why fall from one extreme into the other one? Traditional For loops have their use with array structures, iterators have their use with other data structures.

> UTF encodings have their primary use in data storage and exchange with > external API's. And in memory.

That's the design flaw, IMO. When UTF-8 strings are re-encoded before processing, even insertion and deletion require only linear time. All that could be encapsulated in a class, with flexible internal string representation.

I don't think so, and you don't either:

The most common stringoperations that the avg programmer does is searching for substrings and then split on them, something that can be perfectly done in UTF-8.

:-)

DoDi

In our previous episode, Hans-Peter Diettrich said:

I don't consider it an extreme, on the contrary. Trying to fix this is extreme IMHO.

Traditional For loops have their use with array structures, iterators have their use with other data structures.

Yes. But this fits neither. Array of encoding granularity remains used, and for fundamental, speed sensitive routines there are the inline procedures.

I don't see the need to create another first class type to iterate a first class type.

How, when? How do you avoid repeated encoding/decoding/encoding cycles?

(and even if it succeeds it would be very, very slow, since strings are not infinite in length, the constants still matter)

All that could be encapsulated in a class, with flexible internal string representation.

I don't see why a string should be a class. In FPC it is already a first class type.

True, but I'm not an avg programmer :-)

Sorry, I understood that you want to replace all for loops by iterated loops.

The encoding is only changed on demand. Once converted into a computational format (e.g. UCS2), the internal representation deserves no more changes. Another conversion may occur when a copy in a different encoding is requested; since a copy operation already is O(n), a conversion will not increase that order.

The result were a general Unicode string class, not bound to a specific encoding, similar to the Delphi Unicode string representation.

I don't see why a string should be a class. In FPC it is already a first class type.

Name it as you like, but a first class string type is only a container, that can contain data of any kind - every data type or structure can be seen as a collection of bytes or chars.

Take Unicode, zip or graphics encodings as examples for data types, that can be stored in such a container, and you'll see that every different encoding deserves different support functions - the data type becomes polymorphic. Having reached that point I see a need for a base class, that reflects the *nature* of the information, from which classes for specific encodings can be derived. A user will be happy with the abstract type (see TStrings), and must not bother with specific encodings, that can result in bad behaviour when not choosen properly.

DoDi

In our previous episode, Hans-Peter Diettrich said:

Only the ones where surrogates really matter.

This sounds awfully like the argument long ago that ansistrings wouldn't eventually be slower than strings, since the optimizer would optimize nested inc/decref calls away, so that there would be no more than one call per string per procedure. Afaik, I'm still waiting (and realize now that it is a lot harder since the refcount has to match pretty closely due to e.g. the possibility of exceptions)

Yes, you can throw it all on future optimization, but I more a one bird in the hand kind of guy.

The result were a general Unicode string class, not bound to a specific encoding, similar to the Delphi Unicode string representation.

I see no implementation details here at all that supports such statement. I see a wish, not a plan. The devil is always in the details.

> I don't see why a string should be a class. In FPC it is already a first > class type. Name it as you like, but a first class string type is only a container,

... of chars.

that can contain data of any kind - every data type or structure can be seen as a collection of bytes or chars.

I don't see the any kind stuff. Yes, every type that has a variable memory footprint can store everything in theory, but that is as far as it goes.

Take Unicode, zip or graphics encodings as examples for data types, that can be stored in such a container, and you'll see that every different encoding deserves different support functions - the data type becomes polymorphic.

1. I don't see zip or graphics encodings
2. Since there are some variable requirements on the type, I don't see a good reason to scale that up to "all"

You totally lost me here.

While I of course do see that in many cases it does make sense to use a full blown enumerator to to a loop that traverses an UTF-8 (or other Unicode) string in terms of Unicode characters, with a loop variable that represents the position of the Unicode character in the string (in therms of what ? ), I feel that there are also many cases that only ask for a simple loop that just offers a loop variable coded as a 32 bit UCS Unicode character or as an UTF-8 string.

-Michael

Marco van de Voort wrote

Only the ones where surrogates really matter.

Is is really viable to have the compiler/RTL try to automatically handle these ugly beasts, rather than presenting them to the poor user as two separate Unicode characters (and only handle the UTC/UTF coding issues automatically) ?

-Michael

In our previous episode, Michael Schnell said:

> > Only the ones where surrogates really matter. Is is really viable to have the compiler/RTL try to automatically handle these ugly beasts,

It is not viable not to. Either you implement unicode or not.

It's an users own choice to not be unicode compliant in his apps (e.g. if he knows he never goes to the Eastern Asiatic market etc), but a runtime should be as unicode compliant as reasonably possible.

Regarding that handling surrogate pairs needs tables while UTF/UCS handling can be done by simple algorithms and that (AFAIK) surrogate pairs are used only in certain environments (Mac and what else ?) I think that viability is not an easy decision.

-Michael

Michael Schnell wrote

Regarding that handling surrogate pairs needs tables while UTF/UCS handling can be done by simple algorithms and that (AFAIK) surrogate pairs are used only in certain environments (Mac and what else ?)

Surrogate pairs have nothing to do with Mac OS X. Surrogate pairs are required when encoding any codepoint in UTF-16 whose UTF32 value is >= $10000.

You are probably thinking of are decomposed characters (where e.g. "e" and "¨" are encoded separately, instead of as "ë"). The RTL will never do anything special about them, since they are two regular separate codepoints. And then there's of course the fact that more than one composed character can map to the same decomposed character, see e.g. http://unicode.org/reports/tr15/#Primary_Exclusion_List_Table , and many other issues listed on that page.

In general: if you want to assume that a unicode string is in a particular form, convert it to a particular canonical form and operate on that (and keep in mind that you may destroy data in the process, like with most code page conversions).

Jonas

In our previous episode, Michael Schnell said:

Surrogate pairs enumeration can be done with simple algorithms. Maybe you are confusing surrogate pairs with decomposition?

(AFAIK) surrogate pairs are used only in certain environments (Mac and what else ?) I think that viability is not an easy decision.

Anything that is not strictly UCS2 (afaik Win2000 and older, XP is already UTF16 afaik) has surrogate pairs.

Regarding OS X, iirc I saw a mention somewhere that some components of Mac OS X prefer decomposed characters. (aka UTF-8Mac). I assume routine all unicode routines through the Mac libc on OS X makes sure this will work fine mostly.

In fact I was not aware of the UTF-16 coding scheme. I _supposed_ it would work similar as UTF-8 (highest bit set => 32 bit value composed from the 31 remaining bits of this and the next word and bit 31 reset) and thus could be decoded algorithmically. Seemingly I was wrong and a huge table is needed to decode UTF16.

But in fact we have just been discussing UTF-8. If here, surrogates are unnecessary, and thus using them seems "funny" to me.

-Michael

Marco van de Voort wrote

Regarding OS X, iirc I saw a mention somewhere that some components of Mac OS X prefer decomposed characters. (aka UTF-8Mac).

In another forum I saw this mentioned as surrogate pairs. Sorry for the confusion :(.

-Michael

You don't need any table, see utf16toutf32 in http://svn.freepascal.org/svn/fpc/trunk/rtl/inc/wstrings.inc

Jonas

It's an users own choice to not be unicode compliant in his apps (e.g. if he knows he never goes to the Eastern Asiatic market etc), but a runtime should be as unicode compliant as reasonably possible.

IMO there exist levels of compliance.

The bottom level supplies storage facilities for Unicode. Strings are only stored and processed as a whole, never analyzed or modified. This level supports e.g. text display and storage in databases. A single internal Unicode representation is sufficient, e.g. UTF-16 for Windows or UTF-8 else.

In the next level dedicated string handling is added, for everydays use, like for splitting and composing filenames; this is where basic iteration and character classification support enters the scene, for mostly internal use. Separator characters can be assumed as ASCII, so that they can be found by a dumb byte/char scan; only few encodings have to be recognized and handled, based on the char size: MBCS (UTF-8...), WideChars (UTF-16/UCS2) and UTF-32.

Beforementioned levels IMO can be considered part of the RTL.

Next comes codepage specific handling, usable by coders which are familiar with the specific language. Here more basic parsing features are added, like for whitespace, punctuation, words, numbers etc. Added are character classification and conversion (upper/lower), what requires an implementation of Unicode character sets, as a replacement for the restricted 256-element sets. The Unicode BMP eventually can be treated as one such codepage, so that the many Unicode separators (spaces, dashes...) can be handled in a unique way.

This level can be implemented in language/codepage specific packages, whose maintenance requires more than only coding skills.

The top level is text processing, where knowledge of the language is inevitable, and special libraries are required. Only at this level string composition and decomposition at single character level is required, taking into account ambiguous encodings, ligatures, and the other hazzles introduced by full Unicode. The grammar of a language becomes important, for e.g. proper spelling and breaking of words.

DoDi

In fact I suppose that for UTF-8 ("pure UTF-8" without surrogates) pos() works for all strings and an UTF-8 "character" is a string. It's just not allowed to use the result of pos() other than in the position argument of copy() or delete() and to calculate the length argument for copy() or delete() as a difference between pos() results or Length(String)-values. this makes it hard to extract a single Unicode character from an UTF-8 string, but of course it's easy to create a library function that gets a pos() result and - decoding the UTF-8 code
- creates an UTF-8 string containing the next Unicode character. (UTF-8 coded surrogate pairs may need additional attention)

-Michael

In our previous episode, Michael Schnell said:

Either you have UTF-8 with surrogates, or you have ASCII (since UTF-8 without surrogates means that only char 0..127 are valid, which is ASCII)

Marco van de Voort wrote

Either you have UTF-8 with surrogates, or you have ASCII (since UTF-8 without surrogates means that only char 0..127 are valid, which is ASCII)

In another post surrogate pairs have been denoted as a specialty of a 16 Bit coding (UCS-2), and I did not understand why this was introduced in a discussion about UTF-8. I just accepted that this somehow would leak into UTF-8 as a special (alternate) way to code certain Unicode characters.

I did not think about calling the up to four bytes of a normal UTF-8 "character" "surrogates" (to me these are "codes" or something like this).

Sorry if I introduced any confusion. :(

-Michael

In our previous episode, Michael Schnell said:

Surrogates are characters that can't be encoded in one encoding space.

About 40000 chars can be encoded in a 16-bit value in UTF16, and 127 in 8-bit UTF8. The rest must be encoded in multiple encoding spaces, and these are called surrogates.

I did not think about calling the up to four bytes of a normal UTF-8 "character" "surrogates" (to me these are "codes" or something like this).

Anything larger than 1 is surrogate in UTF-8. And since UTF-8 uses a bit to signal a "larger" character, you end up with low ASCII 0..127 values.