Hi,
Is there a way in current FPC to have unicode or wide resourcestrings? Thanks,

Martin

Op Fri, 29 Feb 2008, schreef Martin Schreiber:

Hi, Is there a way in current FPC to have unicode or wide resourcestrings? Thanks,

Resourcestrings are ansistrings, so the answer is no. This is indeed a shortcoming in a widestring environment, the only solution is probably to encode them UTF-8.

Daniël

This works with utf-8 system encoding only -> not in an usual Windows installation.
Are there plans to implement unicode resourcestrings?

Martin

Op Fri, 29 Feb 2008, schreef Martin Schreiber:

There is utf8decode/utf8encode.

Are there plans to implement unicode resourcestrings?

No, but this is just because you are the first one to ask. Could you make a proposal? Then we'll see how this can be implemented.

Daniël

Afaik XP+ supports UTF-8, and it could be it is also backported to 2000?

MSEgui applications should run on win95/98 and on every locale encoding the user wants to use.

Martin

The big advantage of resourcestrings is that they can be used like normal string constants in the source. If I need to call utf8encode for every used stringconstant I can use the MSEgui widestring stockcaption facility instead.

Ideally from my point of view would be if the resourcestrings are stored in utf-8 if the unit is compiled with -Fcutf8 and decoded by utf8decode for widestring assignment on runtime independent of the system encoding.

Martin

Op Fri, 29 Feb 2008, schreef Martin Schreiber:

This has been discussed before. Automatic conversion to/from ansistrings will always be to/from system encoding. If you want another encoding than the system encoding you will have to do a manual conversion.

For the compiler, a resourcestring has the same type as an ansistring, so no distinction can be made between resourcestrings/ansistrings.

Daniël

Another suggestion:

implementation
resourcestrings
 resstring1 = 'abc';
 resstring2 = '1234';
resourcewidestrings
 resstring3 = 'abc';
 resstring4 = '1234';
procedure test;
var
 wstr1,wstr2: widestring;
begin
 wstr1:= resstring3; //no conversion
 wstr2:= resstring1; //uses system encoding to convert
end;

Martin

There is already a bug report about it:
http://bugs.freepascal.org/view.php?id=6477

Jonas

I don't like inventing Delphi incompat syntax for it, specially since probably starting with D2008 resourcestrings will be unicode too.

So a directive would make more sense (and more importantly, that can be abstracted in e.g. a global include files as many OSS pascal projects do)

Actually you as the application developer can control how conversions between ansistrings and widestrings are handled by using getwidestringmanager/setwidestringmanager .

So for example you could use a byte value of $7F at the beggining of an ansistring as a marker that the string was in UTF-8 and implement a Ansi2WideMoveProc that checked for it and either interpreted it as UTF-8 if it saw the marker and passed it off to the old Ansi2WideMoveProc if not.

There is a non-ambiguous UTF-8 to UTF-16 mapping. It should be no problem at all, except for the extra programming effort.

That means that the program must be compiled in the same system encoding as the target runs in order to get working resourcestrings? That would resourcestrings for i18n purposes render useless apart from utf-8-only or latin-1-only. utf-8-only is not possible for older existing installations.

The compiler does not know that the source is a resource string? But the compiler could know it, the information is available? If the resourcestring kind of source is known, fpc_ResourceStr_To_WideStr could be called instead of fpc_AnsiStr_To_WideStr, the widestring manager needs an additional procedure Resource2WideMoveProc where the appropriate conversion takes place.

Martin

Op Sun, 2 Mar 2008, schreef Martin Schreiber:

In the traditional way of translating software (from the age when resourcestrings were invented) you encode your translation in the encoding of the target system. If you do a Dutch translation you could encoude your translation in ISO-8859-1 and if you do a Czech translation you could encoude the translation in ISO-8859-2.

A more modern approach is to encode all your translations in Unicode and convert them to the target encoding. I.e. UTF-8 to system encoding is what Lazarus does. There is utf8toansi function in the system unit for this purpose. But then you have to call utf82ansi on every resourcestring you access, similar to utf8decode if you use widestrings.

The compiler does not know that the source is a resource string? But the compiler could know it, the information is available?

Language wise, resourcestrings are ansistring constants. So whenever you use a resourcestring identifier in your code, the compiler considers this an expression with type ansistring. Therefore, regarding typing they are equal.

Regarding code generation, there is a difference between ansistrings and resourcestrings, since with a resourcestring load, the compiler must look into the resourcestring tables to find the ansistring constant.

Daniël

So it is theoretical possible to call "fpc_ResourceStr_To_WideStr" instead of fpc_AnsiStr_To_WideStr if the source is a resourcestring?

Op Sun, 2 Mar 2008, schreef Martin Schreiber:

These kind of routines are called by the code generator part that does a type conversion. As there is no type conversion, no such routine can be called. I'm not stating it is theoretically impossible, just stating that the normal type conversion mechanism cannot be used.

Daniël

Then the only possible way to use resourcestrings independent from the system encoding is to compile all resource units with -Fcutf8 and to write every assignment as "widestringvar:= Utf8Decode(resourceconstant);"? Would this method work?

Martin

Op Sun, 2 Mar 2008, schreef Martin Schreiber:

Yes, that method would work.

Daniël

Does not work for me, see attachment. Compiled with -Fcutf8 on win2000
with "ANSI" encoding.
"
const
 strco1 = 'abcäöüûôõ';
 strco2 = 'Русский';
 strco3 = 'ABCDEFG';
resourcestring
 strco1res = 'abcäöüûôõ';
 strco2res = 'Русский';
 strco3res = 'ABCDEFG';
implementation
uses
 main_mfm;
procedure tmainfo.test(const sender: TObject);
begin
 di1.value:= '"'+strco1+'"';
 di2.value:= '"'+strco2+'"';
 di3.value:= '"'+strco3+'"';
 di1res.value:= '"'+strco1res+'"';
 di2res.value:= '"'+strco2res+'"';
 di3res.value:= '"'+strco3res+'"';
 di1resutf8.value:= '"'+utf8decode(strco1res)+'"';
 di2resutf8.value:= '"'+utf8decode(strco2res)+'"';
 di3resutf8.value:= '"'+utf8decode(strco3res)+'"';
end;
"
What did I wrong?

Martin

I'am not sure how this could be made working just a remark: -Fcutf8 influences _only_ the interpretation of string constants when converting them to unicode.

Op Sun, 2 Mar 2008, schreef Florian Klaempfl:

Right. Try to compile the (utf-8 encoded) file without specifying a code page, this should result in the constants ending up in the resourcestring without processing.

Daniël

I thought as much. :-)

And now the widestring constants are broken, see attachment...

Martin

I wonder, do you care also to try BIDI text as well ?

Ido

Did you utf-8 decode the string constants?

Op Sun, 2 Mar 2008, schreef Martin Schreiber:

Ok, that was indeed to be expected.

Yes, I believe some compiler changes need to be made. Resourcestrings should have the encoding of the source files, if you encode your source file in utf-8, the resourcestrings will have to be encoded in utf-8. No idea what processing currently occurs. The compiler will see the string constants as widestring constants, then must do some conversion back to ansistring to get them into the resourcestring tables.

Daniël

The program was the same as before.

ik wrote

I wonder, do you care also to try BIDI text as well ? Ido

No, BIDI is out of scope of MSEgui up to now, supporting all European languages without a headache is a must.

Martin

Inside our projects there are lots of WIdeStrings handling.

For Delphi/Kylix we use the following dirty hack (somewhat reworked routine from Borland RTL):
This one is Widows implementation,
There is also linux implementation

function LoadResStringW(const pRec: PResStringRec): WideString;
var
  hInst:cardinal;
  Buffer: array [0..1023] of char;
begin
  if pRec = nil then Exit;
  if pRec^.Identifier < 64*1024 then
  begin
    hInst:=FindResourceHInstance(pRec^.Module^);
    // try wide
    if LoadStringW(hInst,pRec^.Identifier, @Buffer, SizeOf(Buffer))>0 then
      Result:=PWideChar(@Buffer)
    else // try narrow
    if LoadString(hInst,pRec^.Identifier, @Buffer, SizeOf(Buffer))>0 then
      Result:=PChar(@Buffer)
    else
      Result:='-lost-';
  end
  else
    Result := PWideChar(pRec^.Identifier);
end;

To get the wide string from resource string by its identifier, we go the following:

resourcestring
  sWideStringID='Some string';
var
    mw:WideString;
mw:=LoadresStringW(@sWideStringID)

Under FPC there is special implementation, but dirty hack does not work all the time, since there is hard to determine - where was the resource string
was implicitly decoded to current codepage. PRestringRec is boralnd specific, under FPC it is PAnsiString;

function LoadResStringW(const pRec: PResStringRec): WideString;inline;
begin
  if pRec=nil then
  Result:=''
  else
  begin
    {$ifdef LINUX}
    Result:=PAnsiString(pRec)^;
    {$else}
    {$ifdef WindowsUnicodeSupport}
    Result:=UTF8Decode(PAnsiString(pRec)^);
    {$else}
    Result:=PAnsiString(pRec)^;
    {$endif}
    {$endif}
  end;
end;
{$ENDIF} // end platform-compiler dependent string loader

Please help -
How to implement that dirty hack.

Best regards,
Anton Kavalenka