Hi,

I've just merged the cpstrrtl/unicode branch into trunk. Below you can find the commit message, which describes most changes, the added features and also a very important warning.

Jonas

o merged cpstrrtl branch (includes unicode branch). In general, this adds support for arbitrarily encoded ansistrings to many routines related to file system access (and some others).

  WARNING: while the parameters of many routines have been changed from
    "ansistring" to "rawbytestring" to avoid data loss due to conversions,
    this is not a panacea. If you pass a string concatenation to such a
    parameter and not all strings in this concatenation have the same
    code page, all strings and the result will be converted to
    DefaultSystemCodePage (= ansi code page by default). In particular,
    concatenating e.g. an Utf8String with a constant string and passing
    the result to a RawByteString parameter will convert the result into
    the DefaultSystemCodePage (unless the source code is compiler with
    {$modeswitch systemcodepage} or {$mode delphiunicode} *and* the ansi
    code page on the system you are compiling *on* happens to be UTF-8)
    You can define and use alternative routines that explicitly accept
    Utf8String parameters to avoid this pitfall. Internally, all of these
    routines ensure that they never trigger this condition and ensure that
    not unnecessary/unwanted code page conversions occur.
  + DefaultFileSystemCodePage variable that holds the code page used for
    communicating with the OS single byte file system APIs, and for the
    strings returned by those same APIs. Initialized with
   o the result of GetACP in the system unit of Windows platforms, except for
     WinCE which uses UTF-8 since its file system OS API calls already use
     the UTF-16 versions
   o CP_UTF8 on Unix platforms with FPCRTL_FILESYSTEM_UTF8 defined, and with
     DefaultSystemCodePage on other Unix platforms
   o DefaultSystemCodePage on Java/Android JVM targets
  + DefaultRTLFileSystemCodePage variable that holds the code page used to
    encode strings returned by RTL routines that return filenames obtained
    from OS API calls. By default the same as DefaultFileSystemCodePage on
    all platforms. Separate from DefaultFileSystemCodePage for clarity on
    platforms that may use either utf-16 or single byte OS API calls to
    send/receive file names (such as most Windows platforms)
  + new scpFileSystemSingleByte enum that can be passed to
    GetStandardCodePage() to get the default code page for OS single byte file
    system APIs, with implementations for Unix and Windows
  + SetMultiByteFileSystemCodePage() procedure to override the value of
    DefaultFileSystemCodePage
  + ToSingleByteFileSystemEncodedFileName() function to convert a string to to
    DefaultFileSystemCodePage (does *not* take care of OS-specific quirks like
    Darwin always returning file names in decomposed UTF-8)
  + support for CP_OEMCP
  * textrec/filerec now store the filename by default using widechar. It is
    possible to switch back to ansichars using the FPC_ANSI_TEXTFILEREC define.
    In that case, from now on the filename will always be stored in
    DefaultFileSystemEncoding
  * fixed potential buffer overflows and non-null-terminated file names in
    textrec/filerec
  * when concatenating ansistrings, do not map CP_NONE (rawbytestring) to
    CP_ACP (defaultsystemcodepage), because if all input strings have the
    same code page then the result should also have that code page if it's
    assigned to a rawbytestring rather than getting defaultsystemcodepage
  * do not consider empty strings to determine the code page of the result
    in fpc_AnsiStr_Concat_multi(), because that will cause a different
    result than when using a sequence of fpc_AnsiStr_Concat() calls (it
    ignores empty strings to determine the result code page) and it's also
    slower
  * do not consider the run time code page of the destination string in
    fpc_AnsiStr_Concat(_multi)() because Delphi does not do so either. This
    was introduced in r19118, probably to hide another bug + test
  * never change the code page of a non-empty string when calling setlength on
    it
  * handle the fact that GetEnvironmentStringsA returns the environment in the
    OEM instead of in the Ansi code page (mantis #22524, #15233)
  * don't truncate environment variable strings in GetEnvironmentString(),
    its result is now ansistring/unicodestring depending on whether the
    RTL was compiled with FPC_RTL_UNICODE
  * unix:
   o made the ansistring parameters of the fp*() file system routine overloads
     constant, changed them to rawbytestring and added
     DefaultFileSystemCodePage conversions
   o unicodestring support for POpen(), and DefaultFileSystemCodePage support
     for POpen(RawByteString)
  + DefaultFileSystemCodePage support for dynlibs unit
  + rawbytestring/unicodestring overloads for:
   o system: fexpand, lowercase, uppercase, getdir, mkdir, chdir, rmdir,
     assign, erase, rename
   o objpas: AssignFile,
   o sysutils: FileCreate, FileOpen, FileExists, DirectoryExists, FileSetDate,
     FileGetAttr, FileSetAttr, DeleteFile, RenameFile, FileSearch, ExeSearch,
     FindFirst, FindNext, FindClose, FileIsReadOnly, GetCurrentDir,
     SetCurrentDir, ChangeFileExt, ExtractFilePath, ExtractFileDrive,
     ExtractFileName, ExtractFileExt, ExtractFileDir, ExtractShortPathName,
     ExpandFileName, ExpandFileNameCase, ExpandUNCFileName,
     ExtractRelativepath, IncludeTrailingPathDelimiter,
     IncludeTrailingBackslash, ExcludeTrailingBackslash,
     ExcludeTrailingPathDelimiter, IncludeLeadingPathDelimiter,
     ExcludeLeadingPathDelimiter, IsPathDelimiter, DoDirSeparators,
     SetDirSeparators, GetDirs, ConcatPaths, GetEnvironmentVariable
    -- the default string type used by FindFirst/Next depends on whether the
      RTL was compiled with FPC_RTL_UNICODE. To force the RawByteString
      version pass a TRawByteSearchRec, for the UnicodeString version pass
      a TUnicodeSearchRec.
  + paramstr(longint):unicodestring available for {$modeswitch unicodestrings}
  + pwidechar versions in sysutils of strecopy, strend, strcat, strcomp,
    strlcomp, stricomp, strlcat, strrscan,strlower, strupper, strlicomp,
    strpos, WideStrAlloc, StrBufSize, StrDispose + tests

That conversion IMO is done by the every concatenation, apart from subroutine considerations.

Delphi has overloaded functions for RawByteString and AnsiString(0). FPC could add another Utf8String overload.

I'm not sure how efficient a RawByteString version ever can be. By default it has to convert the string into Unicode (Delphi: UTF-16), and the result back to CP_ACP. In these cases it looks more efficient to call the Unicode version immediately, and leave *eventual* further conversions to the compiler. Some routines may implement common processing of true SBCS, but I'm not sure how many these are.

Does there exist a FileSystemString type, for easy use in RTL and application code?

DoDi

Of course I need to jump in here ...

While I don't exactly know what you are up to, you might want to read some recent posts of mine where I pointed out that using a fully dynamically encoding auto-converting String type (which does not seem to exist in Delphi) for Objects (and other things) that should not predefine a certain encoding in their interface (making the thingy useful for other Strings to avoid unnecessary (auto-) conversion)

Of course I need to jump in here ...

While I don't exactly know what you are up to, you might want to read some recent posts of mine where I pointed out that using a fully dynamically encoding auto-converting String type (which does not seem to exist in Delphi) for Objects (and other things) that should not predefine a certain encoding in their interface (making the thingy useful for other Strings while avoiding unnecessary (auto-) conversion) will not impose any noticeable performance degradation, provided some appropriate compiler magic is implemented.

Of course I might be wrong with this analysis, but I _did_ some decent thinking on that.

-Michael

Of course I need to jump in here ...

Of course ;-)

I know your point.

Of course I might be wrong with this analysis, but I _did_ some decent thinking on that.

You defer the problem from the interface to the implementation. When a procedure has an string parameter of an unknown encoding, the implementation must check every time the *current* encoding of the parameter, and proceed as appropriate. I.e. *all* possible encodings must be handled (implemented) inside the procedure, what's almost impossible. That's why a concrete implementation only can check for a number of cases, which can be implemented in specialized code, and handles all remaining cases in the general (Unicode) way.

With already specialized (overloaded) versions of a routine there exists no reason for checks and different handling. What can be handled in existing specializations is determined at compile time, and all unhandled cases are passed immediately to the universal (Unicode) version.

DoDi

"Hans-Peter Diettrich" wrote

I'm not sure how efficient a RawByteString version ever can be. By

default it has to convert the string into Unicode (Delphi: UTF-16), and the result back to CP_ACP. In these cases it looks more efficient to call the Unicode version immediately, and leave *eventual* further conversions to the compiler. Some routines may implement common processing of true SBCS, but I'm not sure how many these are.

Not every RTL will use a 16-bit API. On Windows the RawByteString variant might be slower, but on Linux it will be faster as long as the string passed in will be encoded in the system encoding (mostly UTF-8).

Also this merge is only a preparation of the RTL to be potentially used with UnicodeString.

Regards,
Sven

Then CP_ACP on Linux will be UTF-8, and automatic conversion can use UTF-8 instead of UTF-16.

I consider both UTF-8 and UTF-16 "Unicode", so that it's debatable whether then also the UnicodeString type will become UTF-8, for even less conversions. But when the user decides that he wants UTF-16 UnicodeStrings, for simplified handling of BMP text...

DoDi

I think you mean "afaik" rather than "IMO". Anyway, the resulting code page of a concatenation is normally the code page of whatever you assign the string to (so if you assign to an utf8string, the resulting code page will be CP_UTF8). RawByteString is different in two ways: a) if all concatenated strings have the same code page, the result also gets that code page b) if there are different code pages involved, the result gets DefaultSystemCodePage

I think we could actually introduce a global variable in the system unit that changes the behaviour of b) to "the result will have a code page that can represent all characters from the concatenated strings", which by default is off. Turning it on should even break most Delphi code, since when a parameter or variable is RawByteString then the code should be able to deal with any possible single byte code page anyway.

Delphi has overloaded functions for RawByteString and AnsiString(0).

Does this really compile in Delphi?

type
  tcp866 = type ansistring(866);
  tutf8string = type ansistring(65001);
procedure test(const a: ansistring); overload;
begin
  writeln('ansistring');
end;
procedure test(const a: rawbytestring); overload;
begin
  writeln('rawbytestring');
end;
procedure test(const a: tutf8string); overload;
begin
  writeln('utf8string');
end;
var
  a: ansistring;
  b: tcp866;
begin
  test('abc');
  a:='ab';
  b:='c';
  test(a);
  test(b);
  test(a+b);
end.

Besides, utf8 *overloads* would be useless since even if the above compiles and has some sensible behaviour, such overloads would only be called if you pass in an utf8string. That already works correctly today. The problem is when passing in concatentations of strings.

I'm not sure how efficient a RawByteString version ever can be. By default it has to convert the string into Unicode (Delphi: UTF-16),

No, see Sven's answer.

and the result back to CP_ACP. In these cases it looks more efficient to call the Unicode version immediately, and leave *eventual* further conversions to the compiler. Some routines may implement common processing of true SBCS, but I'm not sure how many these are.

Even if you are on a platform with UTF-16 system interfaces, if you call the routine with a single byte string, then calling the rawbytestring version will always be more efficient than the unicodestring version because the code size will be smaller (the conversions are inside the wrapper rather than everywhere in your code where you call this routine). These routines do not convert more than necessary.

No, and you should be completely oblivious to it. Adding new magic code pages or string types to the existing mess would not be helpful. Use utf8string or unicodestring if you want to ensure that you don't have data loss. The DefaultFileSystemCodePage only exists to ensure that a) no data is lost between you giving a string to an RTL routine and calling a single byte OS API b) file names returned by single byte OS APIs are correctly handled/interpreted by the RTL and get their code page set correctly

Jonas

Jonas Maebe wrote

I think we could actually introduce a global variable in the system unit that changes the behaviour of b) to "the result will have a code page that can represent all characters from the concatenated strings", which by default is off. Turning it on should even

*not*

break most Delphi code, since when a parameter or variable is RawByteString then the code should be able to deal with any possible single byte code page anyway.

I don't talk about concrete code, so I cannot know anything.

Anyway, the resulting code page of a concatenation is normally the code page of whatever you assign the string to (so if you assign to an utf8string, the resulting code page will be CP_UTF8).

Maybe. I don't know how a concrete compiler handles string concatenations. When using subroutines for that purpose, the Result type specifies the target encoding, not the variable to which the result is assigned subsequently.

RawByteString is different in two ways: a) if all concatenated strings have the same code page, the result also gets that code page b) if there are different code pages involved, the result gets DefaultSystemCodePage

ACK. This means that the result may have to be converted once again, before assigning it to the final target.

That makes no difference when CP_ACP already is UTF-8.

The only break can be increased accuracy, when FPC returns an lossless UTF-8 string, where Delphi would return an (lossy) CP_ACP string. Until that point all internal operations are lossless, using either the unique encoding of an single string parameter, or Unicode (UTF-8/16).

> Delphi has overloaded functions for RawByteString and > AnsiString(0). Does this really compile in Delphi?

It compiles, of course. Unfortunately I cannot test the outcome in detail, due to the broken UTF-8 implementaion in my Delphi XE.

Besides, utf8 *overloads* would be useless since even if the above compiles and has some sensible behaviour, such overloads would only be called if you pass in an utf8string. That already works correctly today. The problem is when passing in concatentations of strings.

The overloads are required to prefer existing implementations, in favor of clumsy RawByteString or overkill UTF-16 strings.

You can safely remove the RawByteString versions, when the compiler-generated conversions into UnicodeString and back afterwards will do exactly the same.

But there exist special situations, essentially defeating the use of RawByteString (XE) has several broken
functions, which return wrong indices when passing in *other* than the explicitly handled CP_ACP strings. All functions, returning indices (lengths...) for the *original* string must *never* convert its encoding. I discussed this already a long time ago, when reporting my first experiences with Delphi XE.

Please tell me how a RawByteString can be handled *efficiently*, as soon as its encoding is *not* CP_ACP [or cpUTF8 if implemented explicitly].

IMO it can make a big difference, when all filename strings in a program (variables, StringLists...) can have the exact type as used in API calls.

Use utf8string or unicodestring if you want to ensure that you don't have data loss.

My point is the number of possible implicit conversions, not data loss.

So why do you only specify a codepage, without also declaring an accordingly encoded string type? Such a type would allow for automatic conversion (if ever required) of file/directory name arguments in RTL functions. Which string type would you use for such arguments?

DoDi

"In my opinion" means "I know the facts, and this is my opinion about those facts". "As far as I know" means "to the extent of my (possibly limited/incomplete/wrong) knowledge" or, more or less, "I think/believe".

Now you do. It's like that both in FPC and in Delphi.

I think that a "can't decide which overloaded function to call" error would be just as logical.

Unfortunately I cannot test the outcome in detail, due to the broken UTF-8 implementaion in my Delphi XE.

The example program I posted does contain any actual UTF-8 strings (only an utf8srting parameter, but if even that causes problems, you can replace it by any other custom ansistring type).

There is nothing particularly clumsy about the RawByteString implementations in our RTL.

But there exist special situations, essentially defeating the use of RawByteString (XE) has several broken functions, which return wrong indices when passing in *other* than the explicitly handled CP_ACP strings. All functions, returning indices (lengths...) for the *original* string must *never* convert its encoding. I discussed this already a long time ago, when reporting my first experiences with Delphi XE.

Please look at and/or test the FPC code, then present your findings about what is actually implemented for FPC.

Please tell me how a RawByteString can be handled *efficiently*, as soon as its encoding is *not* CP_ACP [or cpUTF8 if implemented explicitly].

By ignoring the encoding, because most routines don't care about the actual character values. Those that do (at least the ones mentioned in the commit message) only care about non-control characters in the ASCII range, which are identical in all encodings (except for EBDIC, which is not yet supported; I forgot to mention that) and hence those routines don't need special code either.

All of those routines return a "RawByteString". You can assign such a string to any ansistring type (including a plain ansistring) and the compiler will not insert any kind of code page conversion (and the string's code page will also remain the same, even if it is different from the declared code page of the string you assigned the value to). You can also pass those strings again as parameters to those routines without any data loss (or implicit conversion) because the routines have RawByteString parameters. If you have other string arguments you want to pass to those routines, their initial encoding will presumably depend on where those strings came from. The routines will convert them to the correct code page if necessary.

If you want to perform intermediate operations on such strings yourself rather than using standard RTL routines that take rawbytestring parameters (which all should preserve the data, otherwise file a bug report; and where possible they do so without conversions), use utf8string or unicodestring if you don't want to risk data loss. E.g., we also don't have special string types corresponding to the console input or output code pages, but when using readln(ansistringvar) the code page of ansistringvar will be set to the input code page of the console and it will contain data encoded in that code page regardless of its declared code page. And when using writeln(ansistringvar), the data will also be converted to the output code page of the console if it turns out to be different. If you perform intermediate operations on such data, use utf8string or unicodestring if you don't want data loss, or use RTL routines that take/return rawbytestring parameters.

Jonas

UnicodeString is not UTF-8.

In case of Linux + String=UnicodeString the call sequence will look like UnicodeString function => RawByteString function => OS function. In caes of Linux + String=AnsiString the call sequence will look like RawByteString function => OS function
In case of Windows + String=UnicodeString the call sequence will look like UnicodeString function => OS function
In case of Windows + String=AnsiString the call sequence will look like RawByteString function => UnicodeString function => OS function

Regards,
Sven

We do NOT want to force UnicodeString upon every target. The world not only consists of Windows! So for those cases where the RTL is very likely to be compiled with String=AnsiString (e.g. DOS, OS/2 or the embedded targets when reference counted strings are used) we want to reduce the conversion overhead by providing them the possiblity to call single byte versions (the RawByteString ones) directly.

Regards,
Sven

Hmm, that turns my understanding upside down.

Sorry, I didn't look at or step through the generated code. I still doubt that a concatenation function gets an TargetEncoding argument.

You are right, all calls are ambiguous except test(b).

Please look at and/or test the FPC code, then present your findings about what is actually implemented for FPC.

Later, perhaps, actually I'm busy with other projects.

This means that any AnsiString variable can contain any encoding, not only the declared one? What's the purpose of specially typed AnsiStrings then, and the difference vs. RawByteString?

DoDi

> I think that a "can't decide which overloaded function to call" error > would be just as logical. You are right, all calls are ambiguous except test(b).

Correction: only test(a) is accepted.

DoDi

The target string is passed as var-parameter and thus the codepage can be queried using StringCodePage. See fpc_ansistr_concat in rtl/inc/astrings.inc.

Regards,
Sven

Does this work also for empty string variables?

DoDi

StringCodePage returns the default code page (variable DefaultSystemCodePage) for empty strings (which are Nil).

Regards,
Sven

Hans-Peter Diettrich wrote

You defer the problem from the interface to the implementation. When a procedure has an string parameter of an unknown encoding, the implementation must check every time the *current* encoding of the parameter, and proceed as appropriate.

IMHO you are partly right. If a function needs to know the encoding of a string it handles, it of course needs to initiate a conversion or some ether encoding-specific action. With the appropriate compiler magic for this (not Delphi predefined) type conversion would happen automatically as soon as the user decides to assign the input string to a variable with any predefined fixed string encoding he chooses. In this case the overhead would be close to none regarding a conversion when calling the function.

But there are many cases where the called function does not need to know (and IMHO should not know) the encoding type of the string(s) it works on:

A1) A function that just calls some other function with the string it got as an argument. (Silly example: a function that sticks three things together to one.) (Supposedly some library functions like "copy()" might have build-in compiler magic to avoid conversion when used with - say - UTF8, anyway.)

A2) A function that does not interpret the content of a string but simply stores the String. Most prominent example being TStrings (and here TStringList as it's child). To store the string it does not need to know the encoding of the string but just it's byte-count, that is easy enough to be calculated

Or
B1) A (specially designed) function that is able to work on multiple encoding types without the necessity to completely convert the string into some predefined encoding.
An example would be a function that counts the printable characters (or more simple to do counts the number of digits) contained in a string.

I.e. *all* possible encodings must be handled (implemented) inside the procedure, what's almost impossible.

Not true at all (see above). This seems to be _the_ misconception with this issue. Especially TStrings/TStringList and friends would suffer severely when implemented in a Delphi-isch (thus Windowish) way with fixed predefined encoding for the user code interface.

Regarding the said (specially designed) function the implementing user of course needs to know which encoding he needs to support and thus to take care of when implementing.

-Michael

Sven Barth wrote

Not every RTL will use a 16-bit API.

These RTL/LCL functions / procedures / properties would be covered by the case "B1" in my post a little earlier in this thread..

-Michael

IMHO ( :-) :-) ) specially typed Strings are _very_ useful, as they allow the user to define the encoding he wants certain operations to take place in.

AFAIK ( :-) :-) ) RawByteString (in Delphi) don't ever trigger automatic conversion.

Now my opinion is that FPC (being much more prone to be used with multiple encoding types that the Windows-centric Delphi - which (as has been point out here) even is decently buggy when used for UTF-8 - _should_ _additionally_ feature a versatile fully dynamically encoded type (which does trigger auto-conversion _when_necessary_ ) and implement things ling TStrings and it's siblings using same.

-Michael

Sven Barth wrote

We do NOT want to force UnicodeString upon every target. The world not only consists of Windows!

+1 !

Of course a compiler switch to not use the "NewStrings" would be appropriate.

OTOH IMHO it should be possible to in fact use the "NewStrings" in Linux with a default encoding of UTF8.

Thus, a decently Delphi compatible definition of the encoding when defining Strings (not using the aliases provide) could be:

"($0000)" Default encoding (e.g. UTF16 when compiled for Windows and UTF8 when compiled for Liunx. The RTL OS-centric functions, and in Lazarus the LCL, internally would avoid many conversions when accessed with user code using the default encoding either by "($0000)" or "appropriately" defined strings. Identical to ("$mmmm") with $mmmm being the default encoding when compiling
"($nnnn)" Delphi compatible, auto converting
"($FFFF)" Delphi compatible raw byte string, not auto converting "($FFFF, 1)" Not Delphi compatible: identical to "($FFFF)" (after a ",", the element size is defined; without a "," the element size is set according to the character code)
"($FFFF, 2)" Not Delphi compatible: raw Word string, not auto converting "($FFFF, 4)" Not Delphi compatible: raw DWord string, not auto converting "($FFFF, 8)" Not Delphi compatible: raw QWord string, not auto converting "("$FFFE)" Not Delphi compatible: dynamically encoded String, auto converting when necessary.

The codes "($0000)" and "($FFFE)" are never stored within the string header nor are they known to the Library functions. They only trigger the appropriate compiler magic. The String headers always contain the actual encoding type which is fixed for "($0000)"-predefined Strings and dynamic for "($FFFE)"-predefined strings.

-Michael

Please get your terms right (I already wrote you this in July!). Those "NewStrings" are already used on every target that uses Ansistrings as Ansistrings *are* these "NewStrings". What Delphi 2009 changed was that String=UnicodeString by default instead of String=AnsiString (and at the same time they also introduced the "NewStrings" aka code page aware AnsiString). The UnicodeString type is however not code page aware and is always UTF-16.

Regards,
Sven

Sven Barth wrote

Please get your terms right

You are right I always are unsure about the correct terms here.

I feel like throwing up when I am supposed to use the Term ANSIString for things potentially being encoded as UTF-8, UTF-16 or such, which for me is the contrary of "ANSI".

That is why I avoid using this term altogether, which might be a source of ambiguity.

Sorry for this.
-Michael

Sven Barth wrote

Please get your terms right

Sorry for my ignorance, but what in detail is "cpstr" which is mentioned in the topic by the OP ?

(I seem to remember when the project started it was called something like "cpstrnew" which I interpreted as "new Delphi Strings".)

-Michael

In our previous episode, Michael Schnell said:

in the topic by the OP ? (I seem to remember when the project started it was called something like "cpstrnew" which I interpreted as "new Delphi Strings".)

cpstrnew is the new ansistring type and was merged long ago.

Last week cpstrrtl was merged which changes some functions in the RTL (opening and closing of files in system and sysutils) to be overloaded in rawbytestring and unicodestring instead of ansistring in the default encoding.

At the same time, on Windows, all API calls for switched to the -W functions.

Michael Schnell wrote

I feel like throwing up when I am supposed to use the Term ANSIString for things potentially being encoded as UTF-8, UTF-16 or such, which for me is the contrary of "ANSI".

i agree... ANSIstrings should be ANSI only... UTF8string and UTF16string should be available and used where necessary, IMHO... this would seemingly allow for better control as well as string conversion where necessary...

speaking of conversions, i would also like to see where UTF8 and similar strings can convert to (eg) CP437 where UTF characters like the trademark symbol are converted to their CP437 four character equivalent (eg) "(tm)"... registered and copyright symbols are similar... there are other sequences that can also be transliterated to multiple CP437 characters (eg) ae but these are language specific apparently...

i agree... ANSIstrings should be ANSI only... UTF8string and UTF16string should be available and used where necessary,

And the "NewStrings" that can (defined statically and - if compiler magic is done accordingly - dynamically) be used with any encoding, should have a decent name. That is why I call them "NewStrings", but this of course is a transient and not a decent denomination.

Do we indeed need to always use the sometimes silly that are suggested by Embarcadero ?

IMHO... this would seemingly allow for better control as well as string conversion where necessary...

After some initial concern and some researching, I feel that the Delphi-like way to implement this kind of strings ("quasi-dynamically" encoded strings, each containing definitions for code-word width (1, 2, 4 - and extensible to 8 - Bytes) and encoding id (ANSI Code page and special values for other schemes such as UTF) is a really good idea, and auto-conversion can be implemented in a rather user-friendly manner.

Only - as pointed out in the other mail -, the Delphi compiler magic is lacking support for fully dynamically encoded String which IMHO is doable without any change to the (complex and critical) library and without a noticeable performance degradation, making the use of these beasts a lot more flexible - which is much more important with fpc than with Delphi itself.

Especially providing TStrings and friend (or in fact any library function) with an interface featuring a predefined encoding, IMHO, is not a desirable way to go for a system that is supposed to support multiple OSes and environments.

-Michael

speaking of conversions, i would also like to see where UTF8 and similar strings can convert to (eg) CP437 where UTF characters like the trademark symbol are converted to their CP437 four character equivalent (eg) "(tm)"... registered and copyright symbols are similar... there are other sequences that can also be transliterated to multiple CP437 characters (eg) ae but these are language specific apparently...

My dream is that the compiler magic - when implemented in a way that supports auto-conversion for a fully dynamically encoded string type, additionally allows for arbitrary user-defined Strings type IDs (as well statically defined when creating the variable, when found in the encoding id field of a fully dynamically encoded variable). When seeing some of those, user-provided functions should be called instead of the (or by the) normal library call to allow for a user program implement it's own handling of propriety encoding schemes.

Obviously in many cases theses encoding schemes will be based on commonly used ones and the user functions will call library function on the way. Your "(tm)" example and "html entities" could nicely be handled in such a way without the fpc team needing to bother.

-Michael