Hi,

I edited my source code with TextWrangler (a macintosh text editor), setting the encoding to utf-8, and when I opened with Lazarus it would show the beginning of the file like this:

Ôªøunit mainform;

Notice the first 3 funny characters (actually on lazarus I see different characters, but they changed on copy+paste). FPC didn't seam to care about this, and the file compiled without problems, so I ignored them.

But then, utf-8 strings stoped working.

This, for example:

procedure TForm1.FormCreate(Sender: TObject);
var
  MyStr: string;
  i: Integer;
begin
  MyStr := 'Texto ł ñ ø ß á';
  WriteLn('[TForm1.FormCreate] Printing string values');
  WriteLn('Length: ', Length(MyStr));
  for i := 1 to Length(MyStr) do
   Write(IntToHex(Integer(MyStr[i]), 2) + ' ');
  WriteLn('');
  Self.Caption := MyStr;
  Label1.Caption := 'átomo tômo não';
  Button1.Caption := 'łñø˘ðßßăŏ';
end;

Would result in non-sense results, like this:

[TForm1.FormCreate] Printing string values
Length: 15
54 65 78 74 6F 20 20 20 20 20 20 20 20 20 20

However, if I remove the 3 funny characters, everything work normally again, and I see my UTF-8 characters on screen, and the text output is:

[TForm1.FormCreate] Printing string values
Length: 20
54 65 78 74 6F 20 C5 82 20 C3 B1 20 C3 B8 20 C3 9F 20 C3 A1

Does anyone know what are those funny characters? (I suppose some kind of encoding setting), and why they make utf-8 strings sudenlly stop working?

thanks,

They are the standard marker to identify a file as UTF-8. This is not Mac-specific in any way, it's part of the unicode standard.

(I suppose some kind of encoding setting), and why they make utf-8 strings sudenlly stop working?

You said things did initially work with the UTF-8 marker in place. The default code page used by FPC is 8859-1. However, the scanner detects the UTF-8 marker if present, and when it finds it then it switches the code page to UTF-8. You can also set the code page manually to UTF-8 using {$codepage utf-8}.

The UTF-8 marker maybe got mangled somehow by Lazarus or so. I don't know why it worked again afterwards when you removed the marker.

Jonas

Felipe Monteiro de Carvalho wrote

Does anyone know what are those funny characters? (I suppose some kind of encoding setting)

Your Text editor saved those files with byte order mark (BOM) - see http://en.wikipedia.org/wiki/Byte_Order_Mark. Dunno about the second part of your question though, could be because the compiler treats the characters differently because of this.

David

Jonas Maebe wrote

You said things did initially work with the UTF-8 marker in place.

I didn't say they worked. I said that the source code compiled =)

What happens is that lazarus can't handle utf-8, so I open TextWrangler, edit the strings, close it, and go back to working with Lazarus.

I did some more tests. This time I didn't open the source file with Lazarus, to make sure that it doesn't alter it.

When I use TextWrangler to save my file to UTF-8, the utf-8 strings don't work. If I set it to save my file as UTF-8 without BOM, they work.

The default code page used by FPC is 8859-1. However, the scanner detects the UTF-8 marker if present, and when it finds it then it switches the code page to UTF-8. You can also set the code page manually to UTF-8 using {$codepage utf-8}.

Why does the codepage matter in this case? I would imagine that FPC just reads my string as a bunch of bytes... fpc doesn't need to perform any operations with it.

The compiler internally stores such strings as widestrings. I don't know the details of what the scanner does exactly with utf-8 and why it does so, but there's quite a few utf-8 specific code in the scanner.

Jonas

Jonas Maebe wrote

The compiler internally stores such strings as widestrings. I don't know the details of what the scanner does exactly with utf-8 and why it does so, but there's quite a few utf-8 specific code in the scanner.

Should I submit a bug report?

You can try, but as I said I have no idea whether what you are doing is correct or not. Maybe you're just supposed to wrap all your constant strings with UTF8Encode() before passing them to a routine which expects utf-8 strings.

Jonas

The compiler takes care of the codepage when reading string constants as widestrings.

Daniel Mantione wrong some coments on my bug report:

http://www.freepascal.org/mantis/view.php?id=9058

Could someone elaborate on this? I did't really understand

Looks somewhat illogical to me ... so I write a UTF-8 string and need a widestring managed? But I am not using widestrings ...

And then it will convert the string to iso?? Where does he get iso from?? My app is using utf-8 ..

thanks,

You're. String constants containing chars > 127 are obviously as widestrings because when you give a code page in the source, you want another encoding then the system's default i.e. the compiler needs to remap.

And then it will convert the string to iso??

... or utf-8 ... depending on the setting of the system running the program.

Where does he get iso from??

On windows by default, on unixes by using cwstrings.

My app is using utf-8 ..

This seems too limited to me, as you may be communicating with an API that requires a different encoding than whatever is detected as default. A Mac OS X GUI app does not execute any .profile or so, so libiconv may very well default to some ISO encoding. That does not mean that people converting widestrings to ansistrings want to use this ISO encoding in all cases.

There's utf8encode/decode, but it's quite annoying if you have to replace all widestring->ansistring assignments and parameter passing code with that call (especially since the type conversion from widestring to ansistring is supposed to do everything for you anyway).

Jonas

You should simply leave away the codepage stuff then.

And the BOM... Still, I think it's strange that there is no widestring manager routine to change the output codepage.

Jonas

My current understanding of the issue is this:

I don't tell fpc how my file is encoded, so it suposes it's latin iso. Then fpc detects the encoding of the operating system, and it sees that it's also latin iso, so no convertion is necessary.

I suppose that if a different operating system encoding was found, fpc would, at run-time, convert my string, but I don't really want that, I would like to always get a UTF-8 string.

This is probably a good behavior for console applications, and maybe applications written for gtk 1 too, where the expected encoding is the one of the system.

But if I am using for example Gtk 2, or LCL, or fpGUI, or Carbon, or many other libraries, I will want to always pass a UTF-8 string, regartless of the encoding of the operating system.

How would I then be sure that my string is never converted (or always converted from utf-8 to utf-8 if prefered), but just passed like I wrote it to the library that I am using?

In the current situation, it seams to me that the output of my software would be influenced by factors like changing the implementation of the mac rtl so it starts detecting utf-8 (or anothing different then latin iso) as the system encoding, or by using a c widestring manager to do something unrelated.

thanks,

Op Wed, 13 Jun 2007, schreef Felipe Monteiro de Carvalho:

How would I then be sure that my string is never converted (or always converted from utf-8 to utf-8 if prefered), but just passed like I wrote it to the library that I am using?

Add the cwstring unit, and run it in an utf-8 terminal.

Daniël

Sorry, but this view is too "terminal-centric" as far as I am concerned. That's not something you want to tell users of a GUI app. Or even programmers, for that matter. I really don't see a reason why this should not be configurable by the programmer himself.

Jonas

Op Wed, 13 Jun 2007, schreef Jonas Maebe:

I fully agree one should be able to convert to any encoding one wishes;
in the HTML build tool I use a hacked widestring manager for this purpose;
we need to more structural solution.

I don't agree it is terminal centric; converting to the system encoding is usually what is wanted it ensures that regardless wether the system uses utf-8 or any other encouding, the program displays correct output (for the characters that can be displayed).

Daniël

Well, then something with the design is wrong. Ansistrings are per definition strings which use the default 8 bit encoding of the environment. Putting always utf-8 into them is abusing them and that's why there is an utf8string type in the system unit.

The problem is the definition of "environment". What libiconv considers as environment (some terminal environment variables) does not necessarily match the api's you are using in your program.

Putting always utf-8 into them is abusing them and that's why there is an utf8string type in the system unit.

I'm not saying that they should always contain utf-8, but that the programmer should be able to control this. It's also not just about explicitly using ansistrings, but also about constant strings. If you have

api_which_expects_utf8_string(p: pchar)

and do

api_which_expects_utf8_string('łóżka')

then you'd like a way for that constant string to be passed as utf-8 in all cases without needing to put utf8encode() calls everywhere in your program (especially if all api routines I use expect utf-8, which is pretty much the case on Darwin/Mac OS X).

Jonas

This is a matter of the cwstrings unit now.

The problem is that the compiler needs conversion rules how to interpret this and the compiler internally knows only widestrings and no utf8 or whatever strings. Maintaining different encodings in the compiler is imo also too much.

Then you have to use the utf8string type. If it doesn't work good enough, we've to fix it.

Florian Klaempfl wrote

Then you have to use the utf8string type. If it doesn't work good enough, we've to fix it.

changing on my program from strint to utf8string didn't make any difference. The string is ok when I don't have BOM and is wrong with a BOM

program utftestbom;
{$mode objfpc}{$H+}
uses SysUtils;
var
  MyStr: UTF8String;
  i: Integer;
begin
  MyStr := 'Texto ł ñ ø ß á';
  WriteLn('Printing string values');
  WriteLn('Length: ', Length(MyStr));
  for i := 1 to Length(MyStr) do
   Write(IntToHex(Integer(MyStr[i]), 2) + ' ');
  WriteLn('');
end.

Op Wed, 13 Jun 2007, schreef Felipe Monteiro de Carvalho:

How hard is it to add that widestringmanager? Correct version of this program:

program utftestbom;
{$mode objfpc}{$H+}
uses SysUtils,cwstring;
var
  MyStr:ansistring;
  i: Integer;
begin
 MyStr :='Texto ? ñ ø ß á';
 WriteLn('Printing string values');
 WriteLn('Length: ', Length(MyStr));
 for i := 1 to Length(MyStr) do
  Write(IntToHex(Integer(MyStr[i]), 2) + ' ');
 WriteLn('');
end.

Daniël

Daniël Mantione wrote

How hard is it to add that widestringmanager? Correct version of this program:

This also doesn't change the output of the program. With either BOM or not.

Felipe Monteiro de Carvalho wrote

changing on my program from strint to utf8string didn't make any difference. The string is ok when I don't have BOM and is wrong with a BOM

I've in the mean time discovered that nl_langinfo always returns US- ASCII (or an empty string) under Darwin, regardless of what your LANG/ LC_* settings are. Forcing it to utf-8 fixes some problems, but there is another error in the generic code for writing a widestring. I'm fixing that currently.

Jonas

What happens if you redefine your program as follows?

program utftestbom;
{$mode objfpc}{$H+}
uses SysUtils;
const MyStr: UTF8String = 'Texto ł ñ ø ß á';
var
  i: Integer;
begin
  WriteLn('Printing string values');
  WriteLn('Length: ', Length(MyStr));
  for i := 1 to Length(MyStr) do
   Write(IntToHex(Integer(MyStr[i]), 2) + ' ');
  WriteLn('');
end.

If MacOSX uses always utf-8 for 8 bit strings, you can hardcode it of course in cwstrings and don't use iconv.

Florian Klaempfl wrote

If MacOSX uses always utf-8 for 8 bit strings, you can hardcode it of course in cwstrings and don't use iconv.

Well, it's a bit more complicated than that, see
http://cvs.gnupg.org/cgi-bin/viewcvs.cgi/trunk/intl/config.charset? rev=4343

(search for "Darwin 7.5")

But I'm indeed going to force utf-8, though still using iconv (I don't feel like implementing the translation manually).

Jonas

widestring to utf-8 can be done by the system unit?

Jeff Wormsley wrote

What happens if you redefine your program as follows?

Doesn't change anything.

The core problem is (like jonas said some posts ago) that we need a way to define the output of the widestring manager.

One possible way to do this would, when converting from widestring to UTF8String always convert to UTF-8, despite what the operating system or the terminal expect. Based on my tests, this is not the case today.

Op Wed, 13 Jun 2007, schreef Florian Klaempfl:

Yes, but widestring comparisons cannot, so a utf-8 only widestring manager would still be a huge job.

Daniël

Both issues are now fixed.

Jonas