I am trying to figure out how to do that, or what I do wrong. I found a page about $codepage, but it did not help
http://wiki.freepascal.org/LCL_Unicode_Support
I didnt find the fpc specific page, if exists (I suspect it does)

I am calling a function "function Foo(A:string)" {$mode objfpc}{$H+} I call it with a constant, that contains an german umlaut. Checked with a hex editor, the constant in the source file is utf8

- If I save the source (in utf8), without a utf8 BOM, then it works fine on windows.
- If I had a bom, then the string received by the function appears to be ascii (checked memory dump in debugger "oe" becomes d6
- if I add {$codepage utf8} it also becomes ascii

If I do *not* add that, it seems something gos wrong with the encoding on a PowerPC Mac. Unfortunately this is someone else's pc, and I have no more info.
If I add it things also go wrong, only different. Again no more info.

-----------

I know the provided info, is very little. If there is anything obvious then tell me.

Thanks

Martin wrote

I am trying to figure out how to do that, or what I do wrong. I found a page about $codepage, but it did not help http://wiki.freepascal.org/LCL_Unicode_Support I didnt find the fpc specific page, if exists (I suspect it does)

The page is this: http://wiki.freepascal.org/FPC_Unicode_support though it's rather outdated... :(

Otherwise I can not help you, but it has definitely something to do with the different code page handling in 2.6.0 compared to 2.7.1.

Regards,
Sven

Probably this is due to significant change in FPC 2.7 RTL

*String* type implies the encoding inside
under WIndows it is ANSI by default.

Try to write simple application that concatenates (s:=a+b) two strings with umlauted letters.
The resulting string loose the umlauts under Windows.

The only thing that help at the RTL level -
{$ifdef FPC}
SetMultiByteConversionCodePage(CP_UTF8);
{$endif}

This brings similar behaviour for RTL functions ether in Windows and UNIX but completely breaks file IO.
You wont be able to open file which names translates to more-than-one-byte per symbol.
because RTL IO is ANSI-specific under Windows.

Other approach - use the *UnicodeString*. Forget the *string* type.

regards,
Anton

Martin's question is related to 2.6.0 (see his mail's subject) not 2.7.1.

Regards,
Sven

There is not enough information to be able to give an answer. Source code, source code, source code.

Jonas

Problem is, the original issue does not happen on my hardware. It is about an issue with the test case in components/EditorMacroScript. But only happens on powerPC hardware.
I only have/had an extract of the results. From the looks ofwhat went wrong, and what was the output (calculating char positions) utf8 coding was/is a strong suspect (not confirmed though.

On my hardware it is normally all fine, but fails if I add the $codepage. I could spent a lot of work boiling that down to a sample. But given that I couldn't even find the docs what I really should expect, and therefore might be doing something wrong, I thought I go first looking for what should happen.

Add {$codepage utf8}
on top of
lazarus\components\macroscript\test\testscriptprocs.pas

and the behaviour changes so that the test will fail. (there are utf8 constants in the source, and it appears, that with the $codepage the called code does NOT get that utf8 string, but something else instead.

Martin wrote

On my hardware it is normally all fine, but fails if I add the $codepage. I could spent a lot of work boiling that down to a sample. But given that I couldn't even find the docs what I really should expect,

Without a {$codepage xxx} directive, string constants containing characters > #127 remain exactly as they appear in the source code.

With a {$codepage xxx} directive, string constants containing characters > #127 are converted into unicodestrings during the parsing (according to the specified code page), and converted back into ansistrings (using the "ansi" code page of that particular program run) at run time if they are assigned to ansistring/shortstrings or passed to routines expecting such parameters.

Note that the above is for 2.6.x (as the subject mentions).

Jonas

ok, leaves me with my original problem.

On said ppc, using the original file (no codepage directive). file should be identical (svn checkout) (yet on 2nd thought I can't be sure, that it wasn't open in an editor and saved with utf8 bom...).

So on that ppc something goes wrong. From the feedback I had, it looks exactly as if the encoding of the constant was changed. So what I was looking for was a way to 100% prevent that. Something that tells the compiler: Whatever encoding you find or expect or whatever encoding the output should be, do not touch strings. just take them byte by byte. Does that exist?

As mentioned in my previous reply: if you don't use the codepage directive, then the compiler won't change anything. If you assign the string constant to a unicodestring or pass it as such a parameter, it will of course still be converted from ansi to utf-16 at run time.

Jonas

How does it work in trunk?

Martin

The strings are stored as ansistrings with the appropriate code page.

Jonas

So
"
UnicodeStringVariable:= 'abcdäüö';
"
always will call a conversion function?

Martin

And how works
"
{$codepage 8859-1}
...
UnicodeStringVar:= 'abcd'#228#246#252#1092#1080#1089#1074;

"
?
Martin

The assignment node will insert a type conversion of the right hand side to unicodestring.

In 2.6.x, the right hand side will already be a unicodestring and nothing will happen.

In 2.7.x, the type conversion node will be simplified into a unicodestring constant because it is a typeconversion of a constant (just like int64(1) is also handled at compile time).

That string contains codepoints > #255 and hence is a unicodestring rather than a single byte string. No conversion at either compile or run time happens, and the codepage directive has no influence.

Jonas

Thanks, another question, or is the behavior already documented?

 {$codepage utf8}
 ...
UnicodeStringVar:= 'abcd'#228#252#246;

Does it store 'abcdäüö' in trunk?

Martin

Martin Schreiber wrote

Thanks, another question, or is the behavior already documented?

What you are asking about has always been the same. I don't know to what extent it is documented.

I have no idea how anything I wrote suggests that it wouldn't. As mentioned, the only difference is that string constants containing characters >#127 are no longer always converted to unicodestring constants at compile time. They are ansistring constants with the appropriate code page by default, and hence are only converted (at compile, since they are constants) to a different string type/code page when required.

Jonas

So #n or #nn or #nnn or #nnnn or #nnnnn always means Unicode codepoint and will be at compiletime converted to an 8bit character sequence depending on {$codepage} and stored in a cpstrnew with the codepage of {$codepage} if assigned to a cpstrnew variable?
And if the constant is assigned to a UnicodeString variable the Unicode codepoints are converted and stored to a utf-16 16bit character sequence at compiletime independent if they contain codepoints > 255? Has somebody a link to Embarcadero documentation about the matter? I assume FPC trunk does exactly the same as Delphi XE3 with strings?

Thanks for your patience, Martin

Does this really mean that, when the codes > #255 are removed, the remaining codes have a different meaning?

DoDi

I'm confused. Are these stupid questions? Or should we not touch the theme strings and FPC anymore? But how should we be able to build serious programs with FPC then?

Has somebody a link to Embarcadero documentation about the matter? I assume FPC trunk does exactly the same as Delphi XE3 with strings?

I found this:
http://docwiki.embarcadero.com/RADStudio/XE3/en/String_Types http://docwiki.embarcadero.com/RADStudio/XE3/en/Internal_Data_Formats http://docwiki.embarcadero.com/RADStudio/XE3/en/Fundamental_Syntactic_Elements

but I fear we can not use that information for development with Free Pascal because:
"
The string is represented internally as a Unicode string encoded as UTF-16. Characters in the Basic Multilingual Plane (BMP) take 2 bytes, and characters not in the BMP require 4 bytes.
"
and
"
A control string is a sequence of one or more control characters, each of which consists of the # symbol followed by an unsigned integer constant from 0 to 65,535 (decimal) or from $0 to $FFFF (hexadecimal) in UTF-16 encoding, and denotes the character corresponding to a specified code value. Each integer is represented internally by 2 bytes in the string. This is useful for representing control characters and multibyte characters. "
which seems to be different from Free Pascal.

Martin

Martin Schreiber wrote

Are these stupid questions?

No, but I seem to be unable to explain how it works since you keep asking about things I already tried to explain before, but I clearly failed to do properly. I can keep repeating myself, but I'm not sure whether that will help anyone.

For example, I said that basically nothing changed in 2.7.x compared to 2.6.x, except that certain string constants are no longer automatically converted to utf-16 at compile time, and then you ask "Or should we not touch the theme strings and FPC anymore?". Since basically nothing changed except for a few less blind auto-conversions at compile time, why should you no longer be able to touch anything anymore?

Let me repeat: your string constants will be parsed by the compiler into character sequences with exactly the same content in both 2.6.x and 2.7.x (and with content I mean that if they would be converted to the same code page in 2.6.x and in 2.7.x, you would end up with exactly the same binary data). Whether or not they contain character literals whose value is >#127 in the source code's code page, or explicit "#xx", "#xxx" etc expressions has no influence, nothing changed in the compiler in that account.

The *only* difference is that the compiler can now internally represent ansistrings with arbitrary code pages, and as a result the aforementioned character sequences may now be stored internally in the compiler in a different format, and also stored in the program in a different format if that can avoid conversions at run time. In particular, character sequences are no longer all converted immediately/by default/under all circumstances to UTF-16 in case characters >#127 need to be interpreted according to a particular code page (i.e., if a {$codepage xxx} directive is present).

The compiler will now only convert such character sequences to UTF-16, still at compile time (just like it was able to do in 2.6.x), if it is actually assigned to an UTF-16-encoded string, passed to an UTF-16 parameter etc. And the compiler will also convert it to another ansistring code page is case the character sequence appeared in e.g. a file with {$codepage utf-8} and is then assigned to a variable whose type is declared as "type ansistring(850)".

Jonas

Jonas Maebe wrote

For example, I said that basically nothing changed in 2.7.x compared to 2.6.x, except that certain string constants are no longer automatically converted to utf-16 at compile time, and then you ask "Or should we not touch the theme strings and FPC anymore?". Since basically nothing changed except for a few less blind auto-conversions at compile time, why should you no longer be able to touch anything anymore?

I think it was more meant in the context of the mailing list instead of a technical context. Like in "we had this topic a thousand times and maybe it's better we shut up about it now before we get moderated". Though I could not see where you, Jonas seemed upset about Martin's questions... (and in my opinion the answers cleared up many things - at least for me :) )

Regards,
Sven

On Saturday 05 January 2013 11:30:42 Jonas Maebe wrote: [...]

Thank you very much for the detailed explanation. What I could not found in all the answers (probably it is my ignorance of the English language), is, does #n mean a utf16 code unit as in Delphi XE3 or does it denote something other? You write:

Whether or not they contain character literals whose value is >#127 in the source code's code page, or explicit "#xx", "#xxx" etc expressions has no influence, nothing changed in the compiler in that account.

Assume {$codepage utf-8} how should we enter Russian character constants in #n form? How should we enter Russian character constants in #n form if {$codepage 8859-5} is defined?
And again, sorry for the impertinence, how do resource strings fit in the string handling scenario of Free Pascal trunk?

Martin

Correct. :-)

Martin

Martin Schreiber wrote

Thank you very much for the detailed explanation. What I could not found in all the answers (probably it is my ignorance of the English language), is, does #n mean a utf16 code unit as in Delphi XE3 or does it denote something other?

It was not in the explanation, because it is something that did not change between 2.6.x and 2.7.x. Whatever you use in 2.6.x will still work in exactly the same way in 2.7.x. The Delphi XE3 behaviour may be added to the {$mode delphiunicode} syntax mode, but has not yet been implemented and will never be applied to existing syntax modes.

Assume {$codepage utf-8} how should we enter Russian character constants in #n form?

Using whatever #xx#xx or #xx#xx#xx sequence represents the UTF-8 encoding of that character.

How should we enter Russian character constants in #n form if {$codepage 8859-5} is defined?

Using whatever #xx represents that character in code page 8859-5.

Alternatively, in both cases you can instead define a unicodestring/widestring constant instead of an ansistring/shortstring constant by embedding widechar constants in the character sequence. Such widechar constants are of the form #<number> with <number> a valid Pascal representation of an integer constant between 255 and 65535. Then you can use those widechars to represent the desired characters as UTF-16 code points. In that case, the entire string will however be parsed as a sequence of UTF-16 code points (because a string is either a sequence of ansichars, or a sequence of widechars; it can never be a mixture of the two), and hence also #1 or #128 appearing in a widestring will be parsed as widechar(#1) and widechar(#128) as opposed to being interpreted according to the current codepage setting.

And again, sorry for the impertinence, how do resource strings fit in the string handling scenario of Free Pascal trunk?

Unicode support for resourcestrings is still not available in FPC trunk. They can currently still only be used safely for ASCII content.

Jonas

Jonas Maebe wrote

> And again, sorry for the impertinence, how do resource strings fit in the > string handling scenario of Free Pascal trunk? Unicode support for resourcestrings is still not available in FPC trunk. They can currently still only be used safely for ASCII content.

What about UTF8 content?

Regards,
Sven

Then maybe I should just stop completely answering any questions about this, because apparently not answering completely enough to your liking gets interpreted as telling you to shut up or getting moderated. Just like Sven I don't understand where this interpretation comes from, and I strongly resent it. I didn't answer because I thought the information was all in my previous answers already, and if someone else felt they could clarify it better than I did, they were free to do so.

My time is also finite, and trying to get me to elaborate further by getting my on my high horse because I feel I'm being misrepresented, is something that will not work very well in the long term. It will much more likely result in silence than in more help.

Jonas

You can put anything in it and it may or may not work depending on the current system code page, but afaik the only thing that is guaranteed to work at this time is plain ASCII.

Jonas

05.01.13, 19:40, Jonas Maebe пишет:

You can put anything in it and it may or may not work depending on the current system code page, but afaik the only thing that is guaranteed to work at this time is plain ASCII.

ResourceStrings are stored as AnsiString type with 0 codepage (as I remember). Delphi now stores ResourceStrings as UnicodeString type. I think FPC will follow this in m_default_unicodestring modeswitch.

Best regards,
Paul Ishenin

So compiled with -Fcutf8
"
unicodestringvar:= 'Best'#228'tigung';
"
produces a different result on fixes_2_6 and trunk? I assume in trunk there will be a compile error? We use this form of character constants in MSEgui to have the sources in pure ASCII.

Martin

No. In both cases it results in a widestring with this content:

.short 66,101,115,116,228,116,105,103,117,110,103,0

I guess invalid utf-8 values are just copied through by the compiler. As mentioned: absolutely nothing whatsoever changed in how character sequences are interpreted by the compiler in 2.7.x. The explanation you quoted above (and which I deleted) applies to both 2.6.x and 2.7.x. I really don't know how I can say this in another way, and repeating it clearly doesn't help.

I think it's best if you compile trunk for yourself and test as many scenarios as you can, because I feel I cannot add anything further to the discussion, and I'm not interested in playing compile bot.

Jonas

Seeing that you have already invested lots of time in FPC, you could also ask yourself

'How can I help improve fpc so it remains the right tool for my purposes' ?

Or have you decided that cooperation with the FPC team is an impossibility ?

Michael.

No, no, you understand me wrong. I am merely cautious not to annoy the FPC team, please accept my apology, but I need to decide if FPC is still the right tool for my purposes.

Martin

Paul Ishenin wrote

ResourceStrings are stored as AnsiString type with 0 codepage (as I remember). Delphi now stores ResourceStrings as UnicodeString type. I think FPC will follow this in m_default_unicodestring modeswitch.

It would probably even be better to always do that. At least I don't see a downside, other than slightly larger binaries (and that's not an issue in this case as far as I'm concerned; maintaining two separate resourcestring systems/handlers is just not worth the trouble).

Jonas

But it means that for

Resourcestring
   AString = 'Something';
Var
   S : Ansistring;
begin
   S:=AString;
end.

Always a conversion will happen.

I do not think this is a good idea given that currently, String = Ansistring.

Michael.

It is not easy mainly because the mission goal is so broad. And a division of work probably would be the better solution. I make my job to build a highly productive open source development environment for Free Pascal and the FPC team makes a compiler which allows to build such a tool. But now we are off topic.

Martin

String will always be shortstring or ansistring in the syntax modes in which that is currently the case. And yes, it will involve a conversion in that case. Just like every single constant string assignment to an ansistring in 2.6.x in case the constant string contains non-ASCII characters and is part of a {$codepage xxx} file (because those strings are all stored as unicodestring in the program there).

Then again, it will also involve a conversion if the implementation using ansistrings is fixed to supported non-ASCII resourcestrings and the system codepage is different from the code page in which the resource string has been stored by the compiler. In fact, it will then cause two conversions on most systems (few systems can directly transcode from arbitrary code page X to arbitrary code page Y; most use UTF-16 as intermediate format, although some can probably also use UTF-8).

Yes, the exception is probably UTF-8 on Unix systems, but is that really worth it to complicate the compiler and RTL? Resourcestings are generally not used in performance-critical code, I'd assume. Always using UTF-8 is however also fine for me, btw. I just don't believe it is worth the trouble to support both unicodestring and ansistring resourcestrings.

Jonas

Then it was a misunderstanding again because I read "
Alternatively, in both cases you can instead define a unicodestring/widestring constant instead of an ansistring/shortstring constant by embedding widechar constants in the character sequence. Such widechar constants are of the form #<number> with <number> a valid Pascal representation of an integer constant between 255 and 65535.
"
and
"
Whether or not they contain character literals whose value is >#127 in the source code's code page, or explicit "#xx", "#xxx" etc expressions has no influence, nothing changed in the compiler in that account. "
and
"
I have no idea how anything I wrote suggests that it wouldn't. As mentioned, the only difference is that string constants containing characters >#127 are no longer always converted to unicodestring constants at compile time. "
--> >#255 >#127 and the question arose how can one define "widechar constants" for strings without a character value >255.

Martin

Martin

No, it was simply an omission in my explanation. As mentioned above: "I guess invalid utf-8 values are just copied through by the compiler". It's a special case, but the special case is the same in 2.6.x and 2.7.x (2.6.x converts the UTF-8 string to UTF-16 immediately in the scanner, while 2.7.x does it while processing the assignment; the actual conversion code that's used is however exactly same). The fact that everything remains 100% the same in all cases everywhere always between 2.6.x and 2.7.x has been mentioned at least 10 times in this thread, and that's what I keep trying to make clear. But I give up.

Jonas

Yes, the exception is probably UTF-8 on Unix systems, but is that really worth it to complicate the compiler and RTL? Resourcestings are generally not used in performance-critical code, I'd assume. Always using UTF-8 is however also fine for me,

I do vote for UTF-8

btw. I just don't believe it is worth the trouble to support both unicodestring and ansistring resourcestrings.

I agree.

As a digression from the main topic of this thread (and cringing in advance) I have to say that the FPC community should remember the saying "A picture is worth a thousand words".

It is obvious that Unicode is not a simple topic and among FPC/Lazarus developers/contributors,I suspect that few if any at all, have a detailed grasp of how it all hangs together in the current state of implementation. It brings to mind the parable of the 12 blind men and the elephant.

I think a diagram or graph of Unicode rules and their current state of implementation in FPC/Lazarus would go a long way to helping both developers and end users in this area. It is a topic which comes up regularly and it doesn't show signs of ever going to be properly resolved.

Sometimes I get the feeling there are areas where the developers should stop coding altogether, completely, and instead document the state of affairs as well as they can, and when that is done and no one is interested, take a long sabbatical and decide what to do when they return.

Just my thoughts.

Judging by all the code that I have written during 14 years, there would never be a single conversion necessary. This system would force them on me for every single use.

I do not think that the support of both ansi/unicode string resources is such a burden that it justifies that.

I admittedly have limited knowledge of compiler internals, but I cannot imagine that being able to store them in 2 formats (ansi and some form of unicode) is more than a matter of maintaining 1 flag per string, and writing a word instead of a byte.

All the other code, needed for conversions depending on codepage and whatnot settings, is necessary anyway.

Michael.

Where do you see a difference? The strings are stored in UTF-16, which is the same in every implementation, regardless of (possibly) different more verbose descriptions.

The new AnsiStrings are safe against misinterpretation, because they contain their encoding (codepage). Every char in an AnsiString now can be converted to one and only one Unicode char, when needed. This is not true for single AnsiChars, which still have no codepage information stored with them (in both Delphi and FPC). I strongly discourage the use of Char variables in all flavours (Char, AnsiChar, WideChar), because these are incapable of holding all possible Unicode characters. Only UnicodeChar or UCS4Char (if these exist) can hold all possible character codes, without possible codepage misinterpreation.

The discussion mostly covers the compilation of string *literals*, like 'äöü' or #123, for which every compiler tries to find the best interpretation and internal representation. FPC has a $codepage directive, which tells the compiler that *all* literals in this unit shall be treated as strings of that codepage. This is essential for files stored as Ansi, which have no information about the codepage of the contained single-byte characters. Files stored with UTF-8 encoding, and an UTF-8 BOM at their begin, are safe against misinterpretation.

When the compiler translates the source code string literals, it can store them either as Unicode (UTF-16) or as AnsiString of the given $codepage, depending on the *use* of the literal (type of the string variable in an assignment). This will reduce the number of implicit string conversions at runtime.

[Please correct me if I'm wrong]
DoDi

The FPC and Lazarus UTF details are not that difficult. The complexity comes from adding Delphi *, third party libraries and old FPC, Lazarus versions.

I think a diagram or graph of Unicode rules and their current state of implementation in FPC/Lazarus would go a long way to helping both developers and end users in this area. It is a topic which comes up regularly and it doesn't show signs of ever going to be properly resolved.

For Lazarus:
- works with fpc 2.6.x and 2.7.1
- LCL and most code expect ansistrings to hold UTF-8.
- pascal sources, lfm, po files are stored in UTF-8 without BOM. Special care has to be taken, when using widestrings/unicodestring.
- there are UTF-8 functions and classes (most in package lazutils).
- the IDE supports many encodings
- all this is documented via wiki and fpdoc
- no support for UTF-16 has been started

[...]

Mattias

Your summary sounds clear to me,
and the strategy selected looks to be well designed, I will bookmark this summary as the reference.

You forget also the code necessary to translate resourcestrings (at runtime). Currently the ResourceString related code inside rtl/objpas/objpas.pp only handles AnsiString and then this would need to be adjusted so that UnicodeString can also be handled. For example there will be the need for a "SetResourceStrings" overload with a UnicodeString based TResourceIterator.

Regards,
Sven

No, I had I though of that.
It will need to be changed anyhow, and fell under "is necessary anyway", since we'll need some kind of backwards-compatibility mechanism.

Michael.

IMO resource strings are for display purposes, so that UTF-8/16 encoding is expected by an OS API. AFAIR Win32 string resources are stored in UTF-16, so that assignments to an AnsiString already require a conversion. So IMO UTF-8 would be better, for now and in future.

DoDi

You are very much wrong.

To start with, resource strings are not stored as Win32 resources. Secondly, they are stored in the code as an ansistring.

The resource string of the above example is stored as:

.globl _$PROGRAM$_Ld2
_$PROGRAM$_Ld2:
.ascii "Something\000"
.balign 8
.short 0,1
.long 0
.quad -1,15
.globl _$PROGRAM$_Ld3

Thirdly: in my apps, no UTF-8/16 encoding is expected by the OS. If it were, I would have used widestrings instead of ansistring to begin with, and in that case I would not have made any remark...

So the conversion really would be 100% totally redundant.

Michael.

05.01.13, 23:54, Michael Van Canneyt пишет:

You are very much wrong. To start with, resource strings are not stored as Win32 resources.

I personally think that resources should be stored in their native formats where is possible. This will allow to change them using software designed for that environment. For example for windows there are many resource editors which can replace icons, bitmap and string resources too. It would be nice to have this ability also for binaries which FPC do. On OS X resources are also stored different from what FPC do currently - they are stored in application bundles as I know, so they can be edited by external programs too.

Best regards,
Paul Ishenin

And Jonas is worried about the overhead in the compiler by simple 1/2 byte format ? I doubt this will relieve his worries ;-)

The idea of FPC's resourcestrings implementation has always been to be independent of any OS features so it is a cross-platform solution. That's how it is implemented, and as far as I am concerned that's how it should stay as the default.

If we do what you think by default, then people making a cross-platform app will need to start using different technologies to translate their strings. I doubt that is a good solution.

Currently, people that want to use native win32/OSX resource strings always have the option of doing so, but no special language support for it exists.

Michael.

Not really. I was talking about Win32 resources, not about what FPC makes from resourcestring.

I don't know which OS you're using, but the WinAPI uses UTF-16 throughout. I suppose that other OS also use some Unicode string representation, for lossless representation of texts of all languages.

The dual W/A interface of Win32 is due to the stripped-down Win9x versions, which require Unicode extensions for supporting more than CP_ACP. But now we are in 2013, with Unicode being present everywhere.

So the conversion really would be 100% totally redundant.

It may look so to you...

Why then do you use resourcestring instead of ordinary string constants?

Another note and question, about multi-lingual resources. Windows resource scripts (.RC) allow for multi-lingual stringtables. In my recent research I learned that the resource compiler extracts the requested language from the script, and stores only these strings in the resource file (.RES) and application (.EXE, .DLL). That's why resourcestring was added to Delphi.

How does FPC support the same? (.PO files?)

DoDi

Point taken :-)

But I'm not sure about nowadays use of native resources. Even on Windows most programs nowadays don't use Windows resources for their menus, dialog boxes etc. any more. I've used the Delphi ResourceWorkshop for some time, to tweak some third party programs and even Windows itself.

This will be almost impossible with current software. Try e.g. to set the Windows menu color to yellow, what I did for a long time, and you'll find out that the Explorer and many other Windows tools don't honor that setting. Or you'll find that these system settings have been removed at all, replaced perhaps by themes?

So I'm not sure about the use of native resources, nowadays. How should a multi-platform application handle a string or graphical (icon...) resource, so that it can be designed on one platform, and be shown on all other platforms without modifications?

With graphical resources I'd use a single internal (FPC) format, which is converted by the widgetset for use on the target platform. String resources may require more adjustments than only a translation, to match the different semantics of other languages - independently from the target platform.

That's why I'd suggest UTF-8 encoding for resource strings, what doesn't affect program logic because AnsiString still can be used. The *encoded* AnsiStrings require that the coder knows about the best encoding of every string, when he wants to reduce the number of implicit string conversions. Using AnsiString(CP_ACP) may be a reasonable decision for use in a program with *very* limited usage (one country, one language, one target platform...), but FPC should support programs with a broader audience as well.

DoDi

The discussion is about unnecessary conversions in *FPC resourcestrings*, not about win32 resources.

Why you brought up the Windows resourcestrings was (and is) a mystery to me.

From your statement, I assumed that you probably thought FPC stores it's

resourcestrings as win32 resources. It does not.

I use both windows and Linux.

You are mistakenly assuming that I am using Windows GUI calls or so. There is no GUI.

Probably the only call that cares about codepage is FileCreate(), and that is not done using resource strings. For the rest, all is done using FileWrite() and sendto()/recvfrom(). Both do not care about encoding. They transfer bytes, that's it.

So I use ansistrings throughout.

And hence, resourcestrings being stored in unicode format would cause totally unnecessary conversions.

Michael.

Jonas Maebe wrote

Using whatever #xx#xx or #xx#xx#xx sequence represents the UTF-8 encoding of that character.

Sorry, I can't follow. Does #xx not just define a numerical representation of an 8 bit entity ?

The interpretation in any code might be done later by any code that digests the string.

Am I wrong ?

-Michael

Michael Van Canneyt wrote

String = Ansistring.

Which is the mother of all confusion, IMHO :-[ .

-Michael

Jy V wrote

I do vote for UTF-8

-1

Regarding that conversions in the RTL (or LCL) are a rather seldom runtime-task, GUI performance issues are not really necessary to be considered.

Viable issues seem to be Delphi compatibility, backward compatibility, usability, runtime-performance with time consuming complex string tasks (these seem to vote against UTF8, but for either static UTF 16 or (quasi-) dynamical (CE-alike) encoding; and memory usage and runtime-performance with time consuming simple string tasks (which vote for locale-based ANSI or UTF-8).

-Michael

Once upon a time, on 01/07/2013 12:39 PM to be precise, Michael Schnell said:

I *think* Jonas is trying to say that if you want the character `Ǿ` in a string you would either type
- 'Ǿ' or
- #$C7#$BE if you want to keep the source free of encoding specific characters

You as a programmer make up what you do with it afterwards, if you decide to write it to an UTF-8 terminal, you would get `Ǿ`, and if you write it to some other terminal you might see a character that matches $C7, followed by a character that matches $BE in the lookuptable of the encoding of the terminal. Look at it this way: the byte sequence ($C7, $BE) has got no meaning to the compiler whatsoever, it is a byte sequence. That's what matters to the compiler, what is in this sequence is for you to decide.

Correct me if I'm wrong.

.
.

...or
- #$01FE and then the whole string becomes a Unicode string which is either kept that way (if it is assigned to a UnicodeString constant), or it is converted to some 8-bit encoding at compile time (if it is assigned to an 8-bit constant/variable like ansistring)

(also just my understanding of what Jonas wrote)

Tomas

So the ambiguity with _filling_ a string with data in fact arises when _not_ using the #nn notation :-) . With #nn the effect (i.e. the resulting binary) is obvious.

-Michael

Tomas Hajny wrote

(also just my understanding of what Jonas wrote)

I feel you are wrong. The string does not know about the code it's content is to be interpreted in (other than with Delphi XE).

-Michael

Once upon a time, on 01/07/2013 02:17 PM to be precise, Michael Schnell said:

So the ambiguity with _filling_ a string with data in fact arises when _not_ using the #nn notation :-) . With #nn the effect (i.e. the resulting binary) is obvious.

Well, if there is literally the sequence $C7, $BE in your source code (that is, open up a hex editor and actually see the values there, as one byte each) that would also do the same, as the compiler will default to one byte strings I think. The only issue with this is that you also need to set your code editor to the encoding you want 'cause otherwise it will screw up the display and possible binary value of the character.

So, yes I would say the #nn notation is probably the safest to use, also handy if your character contains (or is) something that `cannot be there`, like a newline: #10 (or #13#10 under windows)

Also, if you use a literal utf-16 char in the code (so no #nnnn, but the actual character) I think the {$codepage utf16} directive might come in handy, as otherwise the compiler will interpret this series of bytes as sperate single bytes characters. This is however not an issue with the #nnnn notation, as there is no ambiguity with this interpretation.

Glad to hear this.

Sorry, your way of quoting makes it difficult for others to react.

I freely admit that I may be wrong, but I don't understand what you meant with your comment and thus I don't understand in what way you I am wrong in your view. The compiler obviously knows how the constant is used within the source code and thus it may proceed accordingly (i.e. either convert it to some 8-bit encoding at compile time if UTF-16 code constant appears in the source, or keep it in UTF-16 if assigned to a UnicodeString constant).

Tomas

Once upon a time, on 01/07/2013 05:05 PM to be precise, Tomas Hajny said:

Yep, the compiler does know how the constant is used and how it is defined (how else could it generate working code?), but I don't see how it could do something with it if it is assigned to another type of string (by type I mean `one-byte versus two-byte`). The compiler can't know for sure what you mean, it can do at least these things:
- Copy data without translating, so a one char two-byte string becomes a two char one-byte string; a three char one-byte string would become a three char two byte string; and then there is a pardox: should a three-char two-byte string become a six-char one-byte string? ==> this is probably not how it is done
- Translate the meanings of the characters of the string, but here the compiler needs to know in what encoding they are and in what encoding the string is wanted. (which it doesn't I believe; the $codepage directive is only used for the encoding of the characters in the unit intself) ==> I think this also isn't a a possibility
- Copy the data byte per byte, but then a one-byte string containing an uneven amount of chars needs padding + there are issues with endianness here ==> Not really an option no?
- Truncate every value of a two-byte string to convert it two a one byte string; the other way around would put each character of the one-byte string as one in the two-byte string ==> Solves the first paradox, but introduces loss of data

==> All the above options (except the translation, that is) ignore the escape charachter(s) of the string, so you wont get the data you want.

IMO I don't think it (typecasting a one-byte string to a two-byte string) can be done without human intervention. Look at it this way: typecasting a thread handle to an integer makes no sense either:
- They are both related (a thread handle is definitely a number, even if it is a pointer)
- But putting one in the other makes no sense at all: what does `comparing whether a thread id is less than zero` mean? on the other hand `comparing whether an integer is less than zero` has a distinct meaning.
- The sizes may be different (say an integer of 16 bit long and a thread handle of 64 bit long), how do you put one in the other? Sum the bytes together? Multiply them? Take the 16 bit CRC of the handle?

This is IMO the same with a one-byte char and a two byte char:
- They both represent letters/words/...
- But they are not the same and cannot be typecasted without extra knowlegde.

This last point is also valid for my example above: you could put all thread ids you know of in a lookup-table and put the index in that lookup-table in the 16-bit integer. Fixed. Same goes for our strings: if you know one is UTF-8 and you want to convert it to UTF-16 it can be done without error, but without this extra knowledge it can't give you decisive results.

Just a few points I think bear some potential to contemplate over a cup of $c0ffee ;-)

That's how I read it as well. In which case, is #A3 16-bit Unicode (representing the UK £ Sterling) or malformed UTF-8 (should be #c2#a3)?

Hi,
this is very off-topic, no serious responses expected... Everytime I read anything about strings here, this comes into my mind:

http://www.rigsofrods.com/entries/155-the-chaos-of-character-encodings

Strings seem to be a general problem for numeric machines :D

d.l.i.w

On Mon, Jan 7, 2013 at 6:05 PM, Mark Morgan Lloyd <

The way I understand it is that #A3 will be effected by $codepage directive of source file. So, if programmer correctly sets $codepage to match encoding used in editor (be it utf8 or some other encoding), compiler will also 'understand' that string correctly.

If programmer never uses UnicodeString, and always uses codepage which was used to write source code, everything will work fine - #A3 will stay whatever it is in specific encoding.

On the other hand, if there comes situation in which string containing #A3 needs to be converted to UnicodeString, compiler will either: a) convert it correctly to UnicodeString if encoding used is utf8, or b) call system-specific function to convert string to array of WideChar-s (in which case, correctness of the program depends on support for specific encoding on tharget system).

Correction:

You're right, Delphi treats control characters as UTF-16 codes, where FPC treats them as byte values (if less than 256).

I noticed the possible problem already, that the FPC interpretation of control characters is context sensitive. This leads to write-only code, because a change of the $codepage would require to change all control characters in that unit accordingly. This in addition to the removal or addition of control characters > 255, which also lead to a different interpretation of the remaining control characters *and* to a different internal representation.

DoDi

UNIcode = it needs several years of study at the UNIversity to use it.

-Michael;-)

I like the idea to try to represent visually (using a drawing) the implications of a technical choice for strings encoding, I guess it would be interesting to draw 2 pictures:
1. the technical areas covered for every type of string encoding (kinda similar to the graph you provided),
2. the volumes of data managed by the different encoding

this may provide some perspectives in the discussion in order for every developer to build their own personal opinion on the question of the choice for string encoding,

Jyv.