Hello FPC,

Working in a new piece of code I had noted that:

A: WideString;
A:=UpCase(A);

returns exactly the same string (WinXP FPC 2.2.2), as this functions seens to not be implemented right now I started to work in some functions for this job at least for the simple cases, this means one low case char has one upper case char, which should solve 95% of the cases.

My question is if this code is of interest to the compiler team, if the code will run only in my programs it will be coded in a way to fit my needs and no more, but, if the team is interested in them I'll put a bit more efforce in write them in a more readable way and even try to match as much as possible the Unicode standards, up to my skills let me, of course.

Currently I have UpCase and LowerCase functions for simple chars and I'm now working in the "SameText" and "Sort" functions.

If the team is interested and can point me to the most important functions it will prevent me for looking for unsupported functions throught the code flow.

I'm just curious how this works with Unicode characters. I guess the Unicode website should cover this in detail (though I haven't searched for it yet). How do you convert from lower to upper case? Is there a set formula per Unicode Block?

The ASCII (standard English alphabet) is easy. Low char minus 32 decimal = Uppercase char

What do you do for other non-English languages and special characters? I noticed the "Latin-1 Supplement" block has the same formula as the standard English alphabet.
ö (u+00F6) -> Ö (u+00D6)

"Latin Extended-A" block doesn't. There the upper and lower characters are next to each other. ę (u+0119) -> Ę(u+0118)

I can't even imagine how things like Hebrew, Greek etc works.... I guess the Unicode UpperCase() function could become quite complex.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Op Tue, 26 Aug 2008, schreef Graeme Geldenhuys:

There is only one way to case convert Unicode characters: tables. A trivial solution would be an array[widechar] of widechar, but one might want a more memory efficient solution.

Daniël

Daniël Mantione wrote

There is only one way to case convert Unicode characters: tables. A trivial solution would be an array[widechar] of widechar, but one might want a more memory efficient solution.

Ah okay - thanks. I saw similar usage of tables in the old LPTK project. Maybe we could use something there - it's written in Object Pascal as well? The LPTK project used a BSD license.

Or, instead of building the tables ourselves (if they are not available on the Unicode website), could we maybe look at something like the OpenOffice source code and get the tables from there? OpenOffice seems to handle Unicode quite well (from what I could see).

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Graeme Geldenhuys wrote

project. Maybe we could use something there - it's written in Object Pascal as well? The LPTK project used a BSD license.

I had a quick look at the LPTK project source... So I gather this implementations is totally rubbish? First of all, a Unicode char is defined as a word (2 bytes), so that means it's only UCS2 compliant and not full Unicode UTF-16 (which is what we want). Plus, it uses no lookup tables as far as I can see - frankly, I don't understand the code at all. :-)

function UpCase16(const s : string16) : string16;
begin
  if length(s) < 2 then Exit;
  if s[2] = #0 then Result := UpCase(s[1])+#0
  else
  begin
    result := chr(ord(s[1]) and $FE) + s[2];
  end;
end;
function Upper16(const s : string16) : string16;
var
  n : integer;
begin
  result := '';
  for n := 1 to length16(s) do
  begin
    result := result + UpCase16(s[n*2-1]+s[n*2]);
  end;
end;

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Op Tue, 26 Aug 2008, schreef Graeme Geldenhuys:

For uppercasing/lowercasing it is correct to define a Unicode char as 2 bytes.

Plus, it uses no lookup tables as far as I can see - frankly, I don't understand the code at all. :-)

The code is simple: If the char is in [#0..#255], it does a normal upcase, otherwise it sets the least significant bit to zero. Seems not correct to me.

Daniël

Lets not limit ourselves again! Lets plan ahead this time. After all, I'm busy learning Klingon [wink], which falls outside the BMP range. UCS2 or word size is limited to the BMP only.

The code is simple: If the char is in [#0..#255], it does a normal upcase, otherwise it sets the least significant bit to zero. Seems not correct to me.

Hence the reason I say I don't understand the code...

I spoke to Martin (from MSEgui). He uses the underlying OS to do the case conversion, but I don't know if that's the way to go, or how portable that is (MSEgui only supports Linux and Windows). It would be nice if the lookup tables could be used by frameworks that use UTF-8 encoding as well (Lazarus LCL and fpGUI for a start).

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Op Tue, 26 Aug 2008, schreef Graeme Geldenhuys:

Klingon does not have capitals, nor does any script outside the BMP. And this is intentionally to keep people their code simple. So there is no "limiting ourselves".

And by the way, Klingon's is not part of Unicode, and unofficially #f8d0 to #f8ff which is inside the BMP.

Daniël

Daniël Mantione wrote

Klingon does not have capitals, nor does any script outside the BMP. And

See the character table (image)...
http://www.kli.org/tlh/pIqaD.html

q and Q. Klingon doesn't have a lower case D, H, I, P and S either. ;-) But you are kinda correct regarding Klingon. Punctuation appears to be only used in Latin transliteration. I cannot comment on other languages/scripts outside the BMP.

And by the way, Klingon's is not part of Unicode, and unofficially #f8d0 to #f8ff which is inside the BMP.

Proposal to encode Klingon in Plane 1 of ISO/IEC 10646-2 http://www.dkuug.dk/JTC1/SC2/WG2/docs/n1643/n1643.htm

5a. Characters should be encoded entirely in BMP?
No. Positions U+0001 CDD0 - U+0001 CD7F are proposed for the encoding.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Also many languages such as Hebrew Arabic and more does not have upper/lower case thingy (Arabic have for most but not all chars 3 types of appearing one at the beginning of the word/next to a non combined char), one in the middle of the chars (combined on both sides) and one for the end, if it does not combine after that char like end of a word or char just not part of the combined chars.

So it's not simple to Math it, unlike ASCII regarding regular latin chars.

Ido

Op Tue, 26 Aug 2008, schreef ik:

Indeed. Of all the scripts daily used in computer usage, only Latin, Cyrillic and Greek have upper/lower case characters. There are a few more not so often used scripts that have upper/lowercase, but that's about it. Other scripts like Hebrew/Arabic have their own cultural conventions, putting everything into upper/lowercase boxes is just another short-sighted western view.

Daniël

Hello Graeme,

Tuesday, August 26, 2008, 8:59:36 AM, you wrote:

GG> I'm just curious how this works with Unicode characters. I guess the GG> Unicode website should cover this in detail (though I haven't searched GG> for it yet). How do you convert from lower to upper case? Is there a GG> set formula per Unicode Block?

After reading, more or less, a big bunch of documentation I finally ended with a +/- 950 entries table lookup to perform binary search. Most letter do not have lowercase/uppercase representation, specially Arabic and ideographic representations, so for the most common alphabets the lookup solution seens to be the easiest way, and should be unicode versioning compatible.

GG> The ASCII (standard English alphabet) is easy. Low char minus 32 GG> decimal = Uppercase char

I'm thinking in hadling them directly instead the lookup table, as it will be the most common case. But currently everything is written in the same "complex" way.

GG> What do you do for other non-English languages and special characters? GG> I noticed the "Latin-1 Supplement" block has the same formula as the GG> standard English alphabet.
ö (u+00F6) ->> Ö (u+00D6)

No, no > "Latin Extended-A" block doesn't. There the upper and lower characters GG> are next to each other. e (u+0119) -> E(u+0118)

The real problem is that some characters are different in amount when in upper case and lower case, in example something like:

"A in lowercase ai" but "ai in uppercase AI", this situations are not currently handled, fortunatly this are not very common.

GG> I can't even imagine how things like Hebrew, Greek etc works.... I GG> guess the Unicode UpperCase() function could become quite complex.

Uppercase and lowercase is quite simple compared with "SameText" ;)

Hello Graeme,

Tuesday, August 26, 2008, 9:19:14 AM, you wrote:

GG> Or, instead of building the tables ourselves (if they are not GG> available on the Unicode website), could we maybe look at something GG> like the OpenOffice source code and get the tables from there? GG> OpenOffice seems to handle Unicode quite well (from what I could see).

In my case I'm building tables from the unicode database which generates pascal array tables.

Current tables are not the very best in memory efficience, but they will be better if the routines works as expected. Now keep things simple :)

Hello Daniël,

Tuesday, August 26, 2008, 11:05:58 AM, you wrote:

DM> For uppercasing/lowercasing it is correct to define a Unicode char as 2 DM> bytes.

If surrogates are handled ;) (my code does not handle them currently).

JoshyFun wrote

Uppercase and lowercase is quite simple compared with "SameText" ;)

Apparently sorting is also a bitch! :-) Depending on the country, sorting of special chars like é or ß appear before or after standard alpha (a-z) characters.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Hello Graeme,

Tuesday, August 26, 2008, 1:57:03 PM, you wrote:

GG> Apparently sorting is also a bitch! :-) Depending on the country, GG> sorting of special chars like é or ß appear before or after standard GG> alpha (a-z) characters.

That's the reason that I'm only thinking currently in the general implementation. Locales should be added as some kind of plugin for the functions, maybe passing a runtime array.

Are you certain those are upper and lower case variants of the same character, and not just completely different characters with a different meaning?

Jonas

See:

http://fundementals.sourceforge.net/cUnicodeChar.html

The "Unicode letter table" contains upper/lower case codes.

2008/9/1 Jonas Maebe