I fully aggree on this

http://www.utf8everywhere.org/

and therefore want to use UTF8 in all my programs. But the problem is sorting UTF8 strings.
According to

http://forum.lazarus.freepascal.org/index.php?topic=15256.0

UTF8CompareText would be the best choice and it runs quite fast. But it does not obey sorting by locale (i.e. german umlauts end up at the end of the list although they need to be sorted together with their corresponding non-umlaut characters (? at U, ? at A, and so on).

Does *any* Pascal UTF8 string compare function exist that does this?

Such functions should be provided by the system's
Unicode/internationalization library. When they can be located there, a wrapper can be added to the RTL.

DoDi

Yes, it exists. But on Windows for example is not available for UTF8, only for UTF16. When I store millions of file names I already have the problem to convert from UTF16 to UTF8 and now I need to convert back again only to get a correct sorting? That would be a real performance hit. It's strange that so many UTF8 string functions exist but none that sorts correctly dependend on locale.

The OP seems to clam that with Unicode, localization is obsolete.

If this is not the case, why then use Unicode ?
- Michael

It's strange that so many UTF8 string functions exist but none that sorts correctly dependend on locale.

Sorting can be done not only by locale and alphabetically, but also in phone book and more sort orders. Just in German you have a couple of options to sort umlauts, and such national stuff is covered by unicode standard libraries.

I didn't realize that such functions are supplied only for the target Unicode representation, thanks for mentioning this topic.

DoDi

Hans-Peter Diettrich wrote

Sorting can be done not only by locale and alphabetically, but also in phone book and more sort orders.

But I don't know any sort order that sorts german umlauts at the end of the whole list (as UTF8CompareText does). Such a sort order is not usable for me.

Just in German you have a couple of options to sort umlauts, and such national stuff is covered by unicode standard libraries.

As it seems these libraries are not available for UTF8 directly (only after time consuming conversion to UTF16).

Michael Schnell wrote

The OP seems to clam that with Unicode, localization is obsolete.

Who claims this?

If this is not the case, why then use Unicode ?

I thought Unicode is just for international *coding* of characters but not for sort order definition.

Most probably it sorts by codepoints, or by bytes.

Such a sort order is not usable for me.

Such sorting is useless in almost every language.

Perhaps the new UnicodeStrings will provide more useful solutions?

DoDi

The Unicode Consortium found many tools and functions required for handling and display of Unicode texts. Many are implemented as (open source) libraries, to be supported and provided by a Unicode-aware platform.

DoDi

You are looking for Collation support,
Comparing/Sorting requires additional infos to load the proper collation for your language (which is not provided in the string encoded as UTF8) -------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20131018/20379436/attachment.html>

J?rgen Hestermann wrote

Who claims this?

Sorry if I over-interpreted your wording.

In a Unicode aware programming language, the handling of Unicode encoded strings needs to provides compare (besides many other string operation, potentially including conversion between multiple Unicode and non-Unicode encoding schemes. )

If string compare only allows for "equal" vs "not equal" results (in some imaginary language) this is complicated enough, as there can be multiple different encodeings for the same "visual character". Additionally, it might be viable to do a "case aware" and/or a "not case aware" operation. To me it's not clear what "case aware" might mean with characters for ancient Egyptian language,

If string compare also allows for "greater" vs "smaller" results the programming language needs to impose some sort order (and maybe a lot more "locale"-depending complex algorithms). This to me seems horribly complicated. Rather obviously you can't define a natural sort order for the complete set of Unicode characters. Thus a kind of "localization" is necessary and supposedly needs to be selectable/definable by the user via "locale" or whatever.

-Michael

this is the purpose of "Collations"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20131021/99dd1ccf/attachment.html>

Jy V wrote

this is the purpose of "Collations"

I see:

http://www.unicode.org/reports/tr10/

As expected: horribly complicated.

-Michael

DUCET support has been submitted by clever developers in FPC and/or Lazarus source tree,
http://bugs.freepascal.org/view.php?id=24856
and I guess it will become available,
it may require the user to provide 1 additional parameter to compare 2 strings
it should not be that difficult to use.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20131021/77480a78/attachment.html>

Jy V wrote

it may require the user to provide 1 additional parameter to compare 2 strings it should not be that difficult to use.

Yep.

Only that traditional Pascal programmers are not used to do

if compareUTF8String(s1, s2, comparemode) < 0 then ...

but

if s1 < s2 then ....

Thus Delphi and FPC introduce quasi-dynamically encoded Strings with automatic encoding-type handling.

Unexpected compare-results are hardly avoidable here :-( .

-Michael

I myself am more used to "CompareText/Str" than "<" or ">" as I didn't know until around 1 or 2 years ago that "<" and ">" are supported on strings at all ^^

Regards,
Sven

Sven Barth wrote

I didn't know until around 1 or 2 years ago that "<" and ">" are supported on strings at all

Nice try O:-)

-Michael

And (probably) overloaded operators are your friends here?

operator < (S1, S2: UTF8String) b : boolean;
begin
  b := (compareUTF8String(s1, s2, comparemode) < 0)
end;

-L.

Nope, because Strings already have the "<" and ">" operators defined you can not overload them.

Regards,
Sven

I get the feeling that _Closed_/_Open_Strings_ (-> http://en.wikipedia.org/wiki/String_theory#Strings ) are easier to understand and of more practical use than _Unicode_Strings_ .

Thus an IDE / Language / Library that not completely hides the complexity behind Unicode (and it's different encoding schemes) should provide a "non-Unicode" Mode that allows for happy programming (at least in Europe, America and Australia).

-Michael

Michael Schnell wrote

I get the feeling that _Closed_/_Open_Strings_ (-> http://en.wikipedia.org/wiki/String_theory#Strings ) are easier to understand and of more practical use than _Unicode_Strings_ .

That link gets me to an interesting but completely unrelated (except for the word string) subject ;-)

"string theory is a theoretical framework in which the point-like particles of particle physics are replaced by one-dimensional objects called strings."

Bart

Bart wrote

"string theory is a theoretical framework in which the point-like particles of particle physics are replaced by one-dimensional objects called strings."

And it's grade of complexity resembles that of Unicode ;-) . And, IMHO,both can be avoided in (my) every-day life unless in the rare cases I am especially interested in the things they are invented for.

-Michael

I forgot to mention that in fact I like to busy myself both with Unicode Strings handling in Pascal and with String theory in theoretical physics.Otherwise I would not take part in this discussion.)

But in every-day life - mostly because of Unicode - I prefer using the old Turbo-Delphi rather than Lazarus.

-Michael

There is a long thread on the forum regarding sorting. http://forum.lazarus.freepascal.org/index.php/topic,17454.0.html <http://forum.lazarus.freepascal.org/index.php/topic,17454.0.html>