Hello,

I'm porting to Lazarus a Kylix application, where I need to display the special symbols in the range $A0 to $BF such as: ? ? ? ? ? ?, etc. (Actually I'm using a font where those symbols are replaced by other symbols related to the application, but that's not relevant). But while, on the same platform, the Kylix application displays them properly, the Lazarus application using the same code doesn't: with GTK2 ws they display as question marks or as crossed boxes, with Qt ws they display as blanks.

The code is sort of:

aText: string;
.....
aText := 'SomeString' + char(ord($A0));
.....
canvas.TextOut(Pos.X,Pos.Y,aText);
Label1.caption := aText;

Changing type from string to UTF8String has no effect.

What I'm doing wrong?

Giuliano

The solution lies in your own mail. Your mail is UTF-8 and so are the above characters. Just copy the above characters from your mail to the IDE:

aText := 'SomeString?';

Note: Keep in mind that "char" is a single byte, but ? in UTF-8 is multiple bytes.

Mattias

Il 08/03/2014 14:05, Mattias Gaertner ha scritto:

I was aware of that. My problem is that the char I must add to the Utf8 string is calculated run time, and is in the range Unicode $A0-$BF.

I had assumed (wrongly) that the compiler was smart enough to convert a type "char" to UTF8, when concatenating it to an UTf8 string. Instead it turns out that the character is appended as it is, which leads to an invalid UTF8 character (above 127), which displays as a crossed box. IMHO that's an FPC bug.

When I realized that, I then tried to explicitly convert the Unicode char to UTF8, but again I failed, this time because of the default behavior which is to map char <-> Unicode only in the range 0-127. Anything above 127 becomes a question mark.
Therefore my symbol displays as a question mark.
IMHO that's a silly FPC limitation.

The only way out I found was to build a table of UTf8 strings in the correct range (corresponding to the Unicode symbols from $A0 to $BF), and use my calculated value as an index to that table.

Rather cumbersome and inelegant, if you think that in the unit ustrings there's an UnicodeToUtf8 routine, which performs exactly the required conversion, but which can't be used because of the above silly limitation.

Giuliano

Are you aware of $CODEPAGE directive? And that Lazarus, unless told otherwise, saves the source files in UTF-8 and tells FPC they are encoded in UTF-8?

Regards,
Fl?vio

[...] I was aware of that. My problem is that the char I must add to the Utf8 string is calculated run time, and is in the range Unicode $A0-$BF.

The Unicode ranges are given in "code points". These are abstract values that must be encoded in bytes. The most common encodings are UTF-8 and UTF-16.
Code point $A0 has two bytes in UTF-8: $C2$A0.

I had assumed (wrongly) that the compiler was smart enough to convert a type "char" to UTF8,

A char is not a code point. A char is an element of string. Every byte encoding consists of chars and so does UTF-8.

when concatenating it to an UTf8 string. Instead it turns out that the character is appended as it is, which leads to an invalid UTF8 character (above 127), which displays as a crossed box. IMHO that's an FPC bug.

It's not a bug.

When I realized that, I then tried to explicitly convert the Unicode char to UTF8, but again I failed, this time because of the default behavior which is to map char <-> Unicode only in the range 0-127.

That's because UTF-8 maps Unicode 0-127 to one byte with the same value as the code point.
Above that it uses a different mapping.

Anything above 127 becomes a question mark. Therefore my symbol displays as a question mark. IMHO that's a silly FPC limitation.

Maybe you underestimate FPC.
FPC supports various source encodings. Lazarus uses by default UTF-8.

[...]

There are some useful UTF-8 functions in unit LazUTF8 and LazFileUtils.

Mattias

There was specific discussion about conversions relating to characters in the range 0x80 through 0xff a few months ago. For parsing things like APL and ALGOL character sets I specifically use UTF-16 internally, and only convert to/from an 8-bit OEM codepage and/or UTF-8 at the margins of the program.

See discussion in http://mantis.freepascal.org/view.php?id=21195 and note change in the 2.7.1 compiler around revision 23613.

Il 09/03/2014 09:36, Mattias Gaertner ha scritto:

Thanks to everybody for the suggestions.

Giuliano