Hi,
I did not find resourcestrings mentioned here:
http://wiki.freepascal.org/User_Changes_3.0
http://wiki.freepascal.org/FPC_Unicode_support

There encoding is not mentioned here:
http://www.freepascal.org/docs-html/prog/progse39.html#x228-2410009.2

http://www.freepascal.org/docs-html/prog/progse40.html states:
"
The resource strings are stored in tables, one per unit, and one for the program, if it contains a resourcestring section as well. Each resourcestring is stored with its name, hash value, default value, and the current value, all as AnsiStrings.
"

Are there any changes? How resourcestrings are encoded?

Thanks, Martin

Good remark.

The only change I am aware of is that the compiler can (and by default does, I think) write the resource strings as a JSON file instead of the older .rst format. [Note to self: I should still document this.]

That being the case, I would think the most reasonable format is UTF-8, but this is speculation.

Michael.

Martin Schreiber wrote on Sun, 20 Sep 2015:

Are there any changes? How resourcestrings are encoded?

Resourcestrings are stored using the same code page as constant ansistrings in the source file in which they are defined (i.e., according to the rules for constant ansistrings described on http://wiki.freepascal.org/FPC_Unicode_support ).

Jonas

Thank you.

FPC fixes_3_0 with the program
"
program json;
{$codepage utf8}
{$ifdef FPC}{$mode objfpc}{$h+}{$endif}
{$ifdef mswindows}{$apptype console}{$endif}
uses
{$ifdef FPC}{$ifdef unix}cthreads,cwstring,{$endif}{$endif} sysutils;
resourcestring
a = 'aäÄ';
begin
end.
"
produces the file json.rsj
"
{"version":1,"strings":[
{"hash":7200948,"name":"json.a","value":"a\u00C3\u00A4\u00C3\u0084"} ]}
"
Is this intended? How can translation tools know that the Unicode points must be interpreted as utf-8 code units in this case?

Martin

I have no idea. I only looked at the compiler/cresstr.pas unit and saw that the codepage of resource strings stored in the object code is set to the source file code page. I don't even know how the .rst/.rsj files are used by the compiler or RTL (or external tools), and have never worked on this aspect of the compiler/RTL.

Jonas

Martin Schreiber wrote

Is this intended? How can translation tools know that the Unicode points must be interpreted as utf-8 code units in this case?

I would guess you found a bug or the implementation is incomplete - but then FPC 3.0 is in rc phase as far as I know.

I would have expected it to be UTF-8 encoded in the JSON file - after all JSON is web technology and web technology uses UTF-8 everywhere as the norm.

Regards,
- Graeme -

What did you expect to see ?

Michael.

Ouch.
I hope you understand my frustration. Now we have the overly complicated codepage aware ansistring which nobody uses, not even Lazarus uses utf8string but sets default codepages to utf-8 instead and Delphi removed it from the newer compilers already, but we still have no full Unicode resourcestring environment which was the only missing piece in order to make comfortable Unicode applications since the implementation of widestring/unicodestring many years ago.
Should I make a bug report?

Another question, it looks to me that -Fcutf8 does not work like {$codepage utf8} for string constants. Is this intend?

Martin

"
{"version":1,"strings":[
{"hash":7200948,"name":"json.a","value":"a\u00E4\u00C4"} ]}
"

Michael.

[...] Another question, it looks to me that -Fcutf8 does not work like {$codepage utf8} for string constants.

What do you mean?

Mattias

Well, I didn't implement either support for codepage-aware ansistrings (I just used it to fix the RTL file system interfaces) or for resourcestrings in the compiler, so it's pretty logical that I have no clue about this.

I hope you understand my frustration. Now we have the overly complicated codepage aware ansistring which nobody uses,

At least the FPC RTL makes a lot of use of it internally, and it is very practical there to deal with the requirements of different platforms. In fact, without this functionality it was simply impossible to support shortstring and ansistring interfaces in the RTL for file system operations in a robust way. You may not have cared if we got rid of all those interfaces since you only use unicodestring, but a lot of people would have.

not even Lazarus uses utf8string but sets default codepages to utf-8 instead and Delphi removed it from the newer compilers already,

Delphi also removed support for mutable unicodestrings, I believe. We generally try to support everything, rather than removing things. That's obviously more work.

but we still have no full Unicode resourcestring environment which was the only missing piece in order to make comfortable Unicode applications since the implementation of widestring/unicodestring many years ago.

There are several more, including lack of support for non-ansi encoded file names and most classes that (still) exist only in ansistring form. You can of course create replacements for all of that, but then again, you could also implement a replacement for resourcestrings in that case.

I'm not suggesting that this is what you should do, nor that FPC shouldn't support resourcestrings in arbitrary encodings or at least with a deducible encoding for external tools (since apparently external tools are supposed to be able to work with these generated files), but I don't think there's a fundamental difference between these cases. Standard Pascal IO routines are as much part of the language as resourcestrings are.

Should I make a bug report?

There is already a (very old) bug report for that.

Another question, it looks to me that -Fcutf8 does not work like {$codepage utf8} for string constants. Is this intend?

I think they should have the same effect.

Jonas

If I looked right, {$codepage utf8} sets cp_utf8 in resource strings, compiling a unit with -Fcutf8 does not.

Martin

I did not miss it. ;-)

Martin

Do you think it is possible without compiler magic other than parsing the sources?

Can you give a link please?

Another bug?

Martin

I meant a bug report for the unusual json format.

Martin

Please do.

Michael.

If you wanted to use unicode filesystem routines before, you could not use the Pascal language functionality for this and had to create your own alternative implementation. If you wanted to use unicode resourcestrings, you could not use the Pascal language functionality for this and had to create your own alternative (e.g. like how gettext works, I guess).

Both standard I/O and resourcestring rely on compiler (and RTL) support.

If they don't have the same effect, it's a bug, yes.

Jonas

I would have expected the same as the original.

"
{"version":1,"strings":[
{"hash":7200948,"name":"json.a","value":"aäÄ"}
]}
"

Regards,
- Graeme -

It is an advantage to have pure ASCII files for such data.

Martin

http://bugs.freepascal.org/view.php?id=28717

Martin

Martin Schreiber wrote

It is an advantage to have pure ASCII files for such data.

Escaped text seems optional, but the UTF-8 encoding seem recommended. Using non-escaped Unicode text is faster to parse and easier to read [by a human] - and from my understanding of the RFC specs, less error prone.

See also the important point about escaped text and UTF-16.

“JSON documents can be encoded in UTF-8, UTF-16 or UTF-32, the default encoding being UTF-8.[14] These encodings support the full Unicode character set, including those characters outside the Basic Multilingual Plane (U+10000 to U+10FFFF). However, if escaped those characters must be written using UTF-16 surrogate pairs, a detail missed by some JSON parsers. For example, to include the Emoji character U+1F602 😂 face with tears of joy in JSON:

{ "face": "😂" }
// or
{ "face": "\uD83D\uDE02" }
”
--- [https://en.wikipedia.org/wiki/JSON#Data_portability_issues]

Reference also Section 8.1 from RFC7159:
https://tools.ietf.org/html/rfc7159#section-8.1

Regards,
- Graeme -

http://bugs.freepascal.org/view.php?id=28718

Martin