Hello,

I am having a problem with string conversion from utf-8 to utf-16:

procedure TForm1.Button6Click(Sender: TObject);
var
  WideText: PWideChar;
  AText: string;
  Size: Integer;
begin
  AText := 'Meu longo texto iso-8859-1';
  Size := Utf8ToUnicode(nil, PChar(AText), 0);
  WideText := GetMem(Size);
  Utf8ToUnicode(WideText, PChar(AText), Size);
  FreeMem(WideText);
end;

When I execute this procedure above, I will get an error on the end of the program:

External: Sigsegv

System_remove_from_list_fixed

any ideas?

thanks,

Did you include cwstring as first unit in your project?

Marco van de Voort wrote

Did you include cwstring as first unit in your project?

thanks, I didn't know I had to.

Marco van de Voort wrote

Did you include cwstring as first unit in your project?

Oh, that doesn´t work. I forgot to mention I am on Windows, where this unit isn´t available.

And isn't needed. The windows system unit does the intialisation of the widestring manager.

Vincent

This should be

   WideText := GetMem(Size*2);

because you get the number of characters, and the number of bytes 2* number of characters.

BTW: I didn't get an access violoation, probably I was just lucky.

I got one when I compiled with -gh.

Vincent

This question scares me a little because the Threading Applications require you include them first... is this also true for Uni-Code Stuff?

Does this possibly suggest Threaded Applications Using Uni-Code May be problematic by chance?

Jason P Sage

Vincent Snijders wrote

This should be WideText := GetMem(Size*2); because you get the number of characters, and the number of bytes 2* number of characters.

Thanks, it works, but I still have doubts.

1) On linux I will need that cwstring unit, right? This was a utf-8 test to be used on fpGUI, and possibly LCL. So can´t we just add cwstring on another unit instead of the first of the program?

2) Does this work in case my string contains characters bigger then #FFFF ? I mean, it seams that we suppose that each character will have 2 bytes, but this may not be true.

3) Shouldn´t we allocate Size * 2 + 2? I mean, we did not allocate space for the null-terminator.

4) Here is this function on action:

procedure TGDICanvas.DoTextOut(const APosition: TPoint; const AText: String);
var
  UnicodeEnabledOS: Boolean;
  WideText: PWideChar;
  AnsiText: string;
  Size: Integer;
begin
  UnicodeEnabledOS := True;
  NeedFont(True);
  if UnicodeEnabledOS then
  begin
    Size := Utf8ToUnicode(nil, PChar(AText), 0);
    WideText := GetMem(Size * 2);
    Utf8ToUnicode(WideText, PChar(AText), Size);
    dynWindows.TextOutW(Handle, APosition.x, APosition.y, WideText, Size - 1);
    FreeMem(WideText);
  end
  else
  begin
    AnsiText := Utf8ToAnsi(AText);
    Windows.TextOut(Handle, APosition.x, APosition.y, PChar(AnsiText),
Length(AnsiText));
  end;
end;

Notice the Size - 1 on the TextOutW call. If I use Size instead of Size -1 it will display a wrong character as the last of my string. Even if I clean the string filling it with zeroes before I pass it to the conversion unit. Why is that? Size already counts the null-terminator?

Umm, I think that Utf8ToUnicode needs to be documented. Even some comments on the code would help, currently there are no comments at all.

5) When I try to convert strings that contain Line-Endings it doesn´t seam to work. What happens to Line-Ending marks on UTF-16? I mean, if we are on linux, a utf-16 line ending marker cannot have just 1 byte, can it?

If I convert a string with a line ending and pass that to ExtTextOutW, a wrong character will appear on the place of the line-ending marker.

thanks,

First of all, I am perfectly happy with using English in all texts, so all this utf-8 and Unicode is just needless ballast and ascii is just fine. So I did not delve too deep into it, nor I am going to in the near future.

1) On linux I will need that cwstring unit, right? This was a utf-8 test to be used on fpGUI, and possibly LCL. So can´t we just add cwstring on another unit instead of the first of the program?

As long as you don't need the widestring manager and any functions that use it, there is no problem putting in any other unit. You just have to make sure the widestring manager is set (which is what the cwstring unit does in its initialization section), before you start using the widestring functions.

2) Does this work in case my string contains characters bigger then #FFFF ? I mean, it seams that we suppose that each character will have 2 bytes, but this may not be true.

That depends on the definition of the char count, check the docs of the function you used (or if the docs are missing, the source of the function you used and the docs of the function it uses).

3) Shouldn´t we allocate Size * 2 + 2? I mean, we did not allocate space for the null-terminator.

That depends on the definition the size parameter. See 2. You earlier version didn't have +1 either and I assumed you had checked that to be ok.

Vincent

Ok, I looked at the source code of Utf8ToUnicode and I can answer some of my questions:

3) Shouldn´t we allocate Size * 2 + 2? I mean, we did not allocate space for the null-terminator.

Utf8ToUnicode returns a size that already counts the extra null-terminator it appends to the string.

I confirmed that size already counts the null-terminator.

Some other questions are still unanswered, namely 2 and 5

thanks,

IMO, Utf8ToUnicode should return the number of bytes the unicode string requires and not the number of characters. Since even for unicode, one char may take more than one word.

I don't know what it returns atm.

Marc