Hi, i have some units shared between 2 Project First in Delphi 2007 and the second in Lazarus for Wince
for unicode support on WinCE i converted my my files to UTF8, but i discovered that Delphi not take my local language in shared units as will, and if convert the files in Delphi IDE it work in Delphi but lazarus not take my strings.
Delphi take UTF8 with BOM mark, Lazarus Without BOM.

Is there any possible idea to make my shared units compiled in both compiler, like as make FPC compile this units as Ansi and in compile time convert it to "UTF8 without BOM" depend my windows locale.

I test -Fmcp1256.txt but no chance.

My language Arabic Win1256
FPC 2.2.2
Lazarus 0.25 beta

FPC automatically sets the input encoding of the scanner to utf-8 when it encounters a BOM.

Jonas

I notice UTF8 from Delphi not Compatible with Lazarus/FPC and vise versa. It Corrupt the characters.

Zaher Dirkey wrote

I notice UTF8 from Delphi not Compatible with Lazarus/FPC and vise versa. It Corrupt the characters.

When you set the encoding of an FPC source file to UTF-8 (either by adding a BOM or by using {$codepage utf-8}), then
a) all constant strings containing utf-8 characters will be decoded and converted to utf-16 (widestring)
b) at run time, these widestrings will again be converted to the active code page when assigning them to ansistrings or shortstrings

If a file does not contain a BOM nor a {$codepage yyy} statement, then constant strings are not parsed in anyway and will be appear literally in the compiled program (and when assigning them to a widestring, they will be "converted" from ansi to utf-16 and hence contain garbage at the end).

This means that if you (ab)use ansistrings to store utf-8 strings (rather than strings in the current ansi-encoding), you either have to a) not use a bom/codepage, or
b) use UTF8Encode(widestringconstant)

Otherwise characters not representable using the active code page will disappear during the run time conversion from widestring to ansistring (and you won't end up with an utf-8 string in any case).

Jonas

Thanks for your answers, and just another few questions :)

1- Is "fpc .... -FcUTF-8" same as {$codepage utf-8}?

2- If i put {$codepage utf-8} in my unit and i have string A variable, that variable will be widestring and any ansistring assigned to it will converted to it?

3- if i has another unit with ansi text not UTF 8 and there is a B variable in that unit, then i assigned to the A := B is B string converted to widestring?.

4- How can i show the content of string (Wide, Ansi) (save it to file without any converting), is there any example.

Thanks in advanced

Zaher Dirkey wrote

Thanks for your answers, and just another few questions :) 1- Is "fpc .... -FcUTF-8" same as {$codepage utf-8}?

Yes.

2- If i put {$codepage utf-8} in my unit and i have string A variable, that variable will be widestring and any ansistring assigned to it will converted to it?

No, A will be a shortstring (in {$h-} mode) or ansistring (in {$h+} mode). Only *constant* strings containing non-ASCII characters will be converted into a widestring at compile time if a codepage is set, so they can be converted to the current code page at run time (since ansi/ shortstrings are always supposed to be encoded using the current code page, and this code page can vary from execution to execution).

3- if i has another unit with ansi text not UTF 8 and there is a B variable in that unit, then i assigned to the A := B is B string converted to widestring?.

No. All (ansi)strings are always supposed to be already encoded using the current ansi encoding, so no conversion happens when assigning one to another.

4- How can i show the content of string (Wide, Ansi) (save it to file without any converting), is there any example.

uses
   sysutils;
var
   f: file;
   s: string;
   w: widestring;
begin
   assign(f,'test.txt');
   rewrite(f,1);
   s:='abc'+sLineBreak;
   w:='def'+sLineBreak;
   blockwrite(f,s[1],length(s)*sizeof(s[1]));
   blockwrite(f,w[1],length(w)*sizeof(w[1]));
   close(f);
end.

Jonas

You fixed my mistakes :)

Is that mean when i want to compile my project to UTF-8 (or Unicode), that mean must change my variables to widestring, A:string to A:widestring (i remember old discuss somewhere about this), is there a string type switched between two types when i choose the unicode in compile, or make "string" type be a widestring (that my mistake in fact)?

There has been a lot of discussion about this problem. What happens is that FPC wishes to always have ansistrings holding system locale encoded strings, it's impossible to have strings which store utf-8 data as far as FPC is concerned.

In Lazarus we consider that the best solution is storing utf-8 data in ansistrings for a series of reason, but the fpc team didn't agree.

The current proposal of the FPC team is to have a new string type which would hold any kind of string, so that you can set it to the desired encoding before doing some use which requires a given encoding. The type is still theorical.

To share code at the moment I do a IFDEF. If in Delphi, then I convert the string from utf-8 to ansi and I don't use the BOM.

To share code at the moment I do a IFDEF. If in Delphi, then I convert the string from utf-8 to ansi and I don't use the BOM.

Ah, by the way, the solution may be more permanent then one imagines, because there is no guarantee that the new type will solve anything for Lazarus, and one of the main reasons we use ansistrings is compatibility with current code, so I don't expect a jump to a new string type.

Take a in this file here to
see how I manage to have strings in both Delphi and Lazarus:

http://magnifier.svn.sourceforge.net/viewvc/magnifier/trunk/translationsvmg.pas?view=markup

The software is aiming to Delphi 2007 or older

I expect things to change in the new Delphi.

When i started to build my projects for unicode, i expect it  is like
{$h-} {$h+} when we moved from Turbo Pascal to Delphi.

BTW my agent ask me to make my project multi language and that solve my problem, but for small projects already made for Ansi we need that feature.

What for {%encoding utf-8} in the first line?

No, that is not possible.

Jonas

Felipe Monteiro de Carvalho wrote

There has been a lot of discussion about this problem. What happens is that FPC wishes to always have ansistrings holding system locale encoded strings, it's impossible to have strings which store utf-8 data as far as FPC is concerned.

And the reason is that
a) if you mix system and non-system encodings in ansistrings, then a bunch of string conversions between ansistrings and widestrings will go horribly wrong
b) if you only use a particular non-system encoding for ansistrings, then interfacing with OS routines will break down completely

It is possible to solve b) by manually adding necessary extra string conversions everywhere in the RTL where ansistrings are passed to OS routines, but that is a lot of work (both to implement and to maintain) and very error prone. Then it's indeed much cleaner to simply introduce a new string type which does not have to be compatible with the OS encoding.

Jonas

Then it's indeed much cleaner to simply introduce a new string type which does not have to be compatible with the OS encoding.

Agree!. It's an illusion that it's possible to hide the complexitiy of different string encodings from the programmer. Instead, such tries always lead to a disaster when the user tests his programs with simple strings which work and then encounters the problems late in the project when unusual characters are used. Better confront the programmer with it right from the start so he knows what to expect.

Jürgen Hestermann.

In our previous episode, Jonas Maebe said:

The solution of Tiburon is the same as Florian's original solution for the multi unicode string type TUnicodeString (that now is still UTF16 only): add an encoding field to ansistring, and alter ansistring declaration with an encoding type:

Type
 TUtf8String = ansistring (cp_UTF8);

This way you can explicitely flag anything internal as UTF-8, and communicate with the outside 1-byte world using the native codepage (which might be UTF-8 too, if desired)

The solution has Windows written all over it (including viewer UTF-8 as a codepage), but it has merits IMHO.

In our previous episode, Martin Schreiber said:

> The solution has Windows written all over it (including viewer UTF-8 as a > codepage), but it has merits IMHO. Are you sure about the encoding field for every string instance?

Yes.

It could be done by compiler magic and an encoding field in the typeinfo too?

No, since then the runtime routines must be overloaded again, for each compiler magic'ed type (all codepages +UTF-8, and another two).

A runtime routine must be able to detect what kind of string type it receives, or you need a runtime routine per type.

Are you sure about the encoding field for every string instance? It could be done by compiler magic and an encoding field in the typeinfo too?

Martin

Or the compiler adds the conversion to the function call because at compiletime the encoding of both types are known. Delphi converts always to an intermediate utf-16 AFAIK.

Martin

In our previous episode, Martin Schreiber said:

No, not always. It often does so for actual processing, but not when it passes to a procedure that simply passes the string on to the next.

And that is the problem in your case. You can't declare a procedure to accept a general ansistring, just to pass it on, since then the type info is lost. Having a general ansistring limits the conversion when passed to procedures that only accept a certain encoding, and to the (typically leaf-) procedures that do actual processing.

Moreover, the 4 bytes encoding are not a real problem. If it can save a conversion here and there, it is already better.

1 - I understand the mixed as
Unit1 have class A with string variables/functions (but it will compiled as AnsiString)
Unit2 have class B inherited from the A but this unit2 but now passing directive to compile it with the strings as WideString, it must not compiled correct? (bad english)

2 -
What if i have i loaded a utf file to TStringList then i want to process the strings as utf-8 and pass it to my functions(P:widestring) without convert from ansi to wide because already have unicode chars. A:String
B:AnsiString
C:WideString

B:=C;//here must convert from Wide To Ansi
B:=A;//as an options in compiling must not convert
B:=WideToAnsi(A);here must convert from Wide To Ansi

I think the answer is above done

In Lazarus we consider that the best solution is storing utf-8 data in ansistrings for a series of reason, but the fpc team didn't agree.