Hi,

I really think the FPC team should make it blatantly clear that the usage of the aliased "String" type should not be used any more with FPC 3.x. It turn I think major alarm bells should start ringing when AnsiString type is used too.

Both are fraught with dangers of loosing data. I've long given up with the UTF-16 vs UTF-8 debate. FPC has now made an even bigger mess than Delphi 2009 did. I've now reached the point where I think - just switch the RTL to UTF-16 only so we can stop loosing data during string conversions! Java does it, C# does it, Delphi 2009 does it. It's time FPC stops trying to please everybody, and switch too.

My reasoning for the above. Data loss during string conversions!

1. The "String" type has too many meanings. I recommend you simply stop using it in your code. Instead, use the exact type you really mean or support in your application.

2. AnsiString in FPC 3.x is now code page enabled, and can have many different meanings.
There is no guarantee that assigning a UTF8String or UnicodeString to a AnsiString is safe. On some systems the AnsiString might default to a UTF-8 or UTF-16 encoding which is fine, but importantly, on other systems in might default to something else, and then you simply loose data during the conversion. What makes it even worse, is that the default encoding for AnsiString is determined at runtime, so the programmer is completely helpless in preventing this. This issue is also not specific to only certain operating systems. You can have a Linux or FreeBSD system that defaults to a non-Unicode type code page, yet the Linux system running right next to that one could have been set up to use a UTF-8 code page. Same for Windows platforms.

FPC should seriously start ringing those alarm bells, as many programs compiled with FPC 3.x are bound to hit these errors (BUGS). Valuable data could get corrupted.

To all programmers, start using UnicodeString or UTF8String types only, and everywhere!

Regards,
Graeme

Regards,
Graeme

The same is true in previous FPC versions.

2. AnsiString in FPC 3.x is now code page enabled, and can have many different meanings. There is no guarantee that assigning a UTF8String or UnicodeString to a AnsiString is safe.

The same is true in previous FPC versions.

The same is true in previous FPC versions.

so the programmer is completely helpless in preventing this.

Unlike in previous FPC versions, in FPC 3.x the programmer can specify the encoding that ansistring should use at run time via the SetMultiByteConversionCodePage().

This issue is also not specific to only certain operating systems. You can have a Linux or FreeBSD system that defaults to a non-Unicode type code page, yet the Linux system running right next to that one could have been set up to use a UTF-8 code page.

The same is true with previous FPC versions (since this is obviously unrelated to the compiler/RTL/language at all, and previous FPC versions also queried the OS to set the code page used for ansistrings if a widestring manager was used -- without a widestring manager, nothing changes either).

Jonas

True, and from earlier mailing list discussions, not many know about that. No fault of yours of course. The most important point is that by the time they find out they need to call
SetMultiByteConversionCodePage() they already lost valuable data. All the compiler does is raise "warnings" in the compiler output, yet happily compiles further and generates you a fatal executable. Those should really be errors!

I'm not in the fpc-devel mailing list, but do you mind sharing some quick information on what the FPC team are planning for the RTL? I remember years back there was a discussion about two RTL versions (AnsiString and Unicodestring). Personally I think that would be a disaster to maintain. In FPC 3.0.0 there seems a hybrid mess (yes I understand the RTL is not Unicode ready, FPC 3.0.0 only made the compiler Unicode ready).

My opinion: go the Delphi 2009 route and simply change the RTL to Unicodestring everywhere (or make a hybrid UTF8String & UnicodeString versions). If anybody wants the AnsiString or ShortString RTL's, stay with FPC 2.6.4 and maintain it yourself.

Regards,
Graeme

As mentioned in my previous mail, the same happened in previous FPC versions.

All the compiler does is raise "warnings" in the compiler output, yet happily compiles further and generates you a fatal executable. Those should really be errors!

You also want errors when assigning longints to bytes etc? (which does not even produce a warning) The previous compiler version did not even produce warnings for string conversions that could result in data loss, and yet you argue that the situation got worse with FPC 3.x.

I'm not in the fpc-devel mailing list, but do you mind sharing some quick information on what the FPC team are planning for the RTL?

There is no such quick information, because it's a very hairy topic and no agreement has been reached yet.

My opinion:

Having opinions available is not the issue. Getting people to completely implement all of their opinions so that we can compare all options in practice, with the chance that most of those people's work will be discarded afterwards (or have them implement it in a way that is usable no matter which option is chosen in the end), is —understandably— much more difficult.

Jonas

Jonas Maebe wrote

Having opinions available is not the issue. Getting people to completely implement all of their opinions so that we can compare all options in practice,

And we all know that will never happen in FPC. Neither does it happen [implementing every possible alternative to a problem] in any other software product or company I've ever come across. So if you think that is a solution for FPC, I fear you are mistaken.

Again, real-world solution have already been presented to you [the FPC team]. Java, C# and Delphi 2009+ and many others. Don't tell me they all got it wrong (with there much larger development teams and resources). Simply switch the RTL to UnicodeString (UTF-16) everywhere and be done. Yes you might take a popularity hit for a while with some, but there is NO way you can please everybody. Somebody will always have an issue. And like I said, if they don't like the new direction, fork the project and do your own thing, or stick with FPC 2.6.4 and maintain it yourself.

Regards,
Graeme

Graeme Geldenhuys wrote

On 2016-05-09 11:47, Jonas Maebe wrote:

Hello Graeme,

Regardless from other decisions, we all have already known that new UnicodeString implementations (or at least checks of full UnicodeString support without any changes apart from replacing 'string' with 'UnicodeString') are necessary for string handling in the RTL for quite some time. Nevertheless, not much of that has been created so far.

If such implementations are created, you can always decide if string/ansistring interfaces should continue to exist somehow (maybe just performing a conversion and calling the UnicodeString version where possible), or whether they should become the only ones available.

Tomas

Jonas Maebe wrote

Getting people to completely implement all of their opinions so that we can compare all options in practice, with the chance that most of those people's work will be

I already have some UnicodeString RTL updates while I worked on the tiOPF code. I want to submit those to Mantis, but where does RTL unit tests go (not compiler syntax tests, but RTL)? All my changes have accompanied unit tests. As far as I understand the <src>/tests/ directory is only for compiler and language syntax tests.

Regards,
Graeme

Tomas Hajny wrote

If such implementations are created, you can always decide if string/ansistring interfaces should continue to exist

Exactly what I've done. I've created UnicodeString only versions. I don't see the point of the others.

Regards,
Graeme

Graeme Geldenhuys wrote on Mon, 09 May 2016:

tests/test/units/unitname

Jonas

Jonas Maebe wrote

tests/test/units/unitname

What I've created are FPCUnit based unit tests. Looking a bit further, what I'm looking for seems to be in tests/test/units/fpcunit/

Thanks again.

Regards,
Graeme

Graeme Geldenhuys wrote on Mon, 09 May 2016:

If these are tests for non-core units (basically nothing that's in the RTL directory), that's fine. RTL tests should not depend on FPCUnit, because FPCUnit itself may depend on those RTL units functioning correctly, which would create a rather annoying catch-22 in case something breaks.

Jonas

Jonas Maebe wrote

If these are tests for non-core units (basically nothing that's in the RTL directory), that's fine. RTL tests should not depend on FPCUnit,

I unit tested the StrUtils.pp unit. It is part of the RTL, but seems not to live inside the “rtl” directory, but rather the “packages/rtl-objpas/” directory. So I presume it is okay to use FPCUnit then.

Regards,
Graeme

Graeme Geldenhuys wrote

the usage of the aliased "String" type should not be used any more with FPC 3.x.

You can't seriously suggest to dump all code that ever had been written in Pascal (Delphi Dialect).

-Michael

Graeme Geldenhuys wrote on Mon, 09 May 2016:

I unit tested the StrUtils.pp unit. It is part of the RTL, but seems not to live inside the “rtl” directory, but rather the “packages/rtl-objpas/” directory. So I presume it is okay to use FPCUnit then.

Yes, that should be fine.

Jonas

Am 09.05.2016 13:11 schrieb "Graeme Geldenhuys" <

In case you missed it we are supporting more targets than Delphi did back when they switched to UTF-16. Unlike C# and Java and Qt we also support MS-DOS and Win16 where memory restrictions are much more likely to arise. We take backwards compatibility rather serious and thus a decision like this where there isn't a technical reason behind it (unlike an upcoming RTTI change) is much less easy to make.

Regards,
Sven

Sven Barth wrote

MS-DOS and Win16 where memory restrictions are much more likely to arise.

Fair enough. Then use UTF-8 (instead of UTF-16) as the default (or only) encoding for the RTL. UTF-8 memory usage is on-par with AnsiString (FPC 2.6.4 and earlier) for MS-DOS and Win16 targets - or at least it should be (the first 255 characters of UTF-8 are single byte encoded).

I fully understand there are a lot of decisions to be made by the Core FPC Team. But these Unicode and RTL discussions have been going on for 3-5 years now, and still the team is undecided about which direction to take for the RTL. This doesn't bode a lot of confidence for us or any company that would like to invest and use Free Pascal exclusively. Bite the bullet and make a decision - and least then I and other contributors know where and what to contribute towards. This state of limbo is not good.

Regards,
Graeme

.
.

No, not quite - your statement does not apply for anything beyond the first 127 characters (the meaning of all characters starting with #128 depends on the selected codepage and thus encoded differently in UTF-8 in many cases).

Tomas

That's a big claim, and I suggest it could do wit a bit more precision.

What, /exactly/, are you saying can be lost, and under what circumstances?

Mark Morgan Lloyd wrote

What, /exactly/, are you saying can be lost, and under what circumstances?

You loose “data” due to codepage based AnsiString (aka the String type) not always supporting all code points of UTF8String or UnicodeString data.

eg:
I write a program that assigns a UnicodeString value to an AnsiString variable. My program uses compiler mode OBJFPC and {$H+}. I run that same executable on two different Linux systems.
NOTE: it's the same executable.

Linux Box #1:
The default codepage is UTF-8, thus String equals AnsiString(65001). No data is lost when converting from UnicodeString to String on this system. Essentially it’s a conversion of UTF-16 to UTF-8 - both support the full Unicode range.

Linux Box #1:
Here Linux has been setup with a default codepage of ISO-8859-1 (Latin 1). I have a UnicodeString variable which contains BMP and Planes 1-12 code points. The program assigns that to my String type variable, which is actually AnsiString(<latin1>). Only the first 255 characters of the 1.4 million Unicode code points will be converted. All the others will be replaced by a '?' symbol. A massive data loss, and that data could be critical.

What does FPC do about this? It only gives you a compiler warning when the application was compiled, but still generates the executable as normal.

I now fully understand why Delphi 2009 and later uses UnicodeString as the default type and their String = UnicodeString = UTF-16. It defaults to UTF-16 on all its supported platforms (granted, Delphi support a lot less platforms than FPC does). At least with Delphi it protects the developers which still uses the String type everywhere (remember String = UnicodeString there). Much safer than what FPC 3.x does now!

Now some would say, simply switch your compiler mode to DelphiUnicode. But I don't want to do that, because I like the stricter ObjFPC mode, and prefer ObjFPC's syntax.

Regards,
Graeme

So which of these are you complaining about:

a) AnsiString doesn't support codepoints > 0xff ?

b) AnsiString doesn't support codepoints > 0x7f ?

c) AnsiString might apply an inappropriate translation for a codepoint <= 0x7f ?

Graeme Geldenhuys wrote

Now some would say, simply switch your compiler mode to DelphiUnicode. But I don't want to do that, because I like the stricter ObjFPC mode, and prefer ObjFPC's syntax.

While still missing in the documentation, you can already do that with
{$modeswitch unicodestrings}.

And please stop this nonsense about how dangerous the String type in FPC 3.x supposedly is, after I've explained several times in this thread already how it's just as dangerous in FPC 2.6.x.

Jonas

Jonas Maebe wrote

Graeme Geldenhuys wrote:

Hello Graeme,

In addition to the comment raised by Jonas, note that in the example mentioned by you, the decision about possible data loss isn't cause by the compiler but rather by the user who sets his environment to use a character set which can only process a limited subset of all possible characters while he intends to use a bigger set. Similarly, a user processing a text file saved in a particular character set needs to understand that such a text file cannot host files not supported in the particular character set.

Let's have another look at what FPC gives you: In spite of platform limitations, FPC provides the programmers with means to support the full range of Unicode characters even on platforms using limited character sets natively (e.g. using fpwidestring Unicode string manager). However, such a decision comes with a price (e.g. bigger executables and memory requirements plus a possible slow-down with certain operations) and the programmer needs to decide whether he wants to pay the price, or not.

You can use UnicodeString in your own code since long time ago. You're welcome to help us with removing restrictions existing in our RTL today (like you did in your strutils patch provided today, thanks!). There are many places where UnicodeString and ansistring/shortstring interfaces may coexist happily. Admittedly, there are other places where this isn't possible and you're perfectly right that the FPC team needs to make some decision about those. My _personal_ feeling is that we probably will bite the bullet (as you suggested) for those in the end, although it will probably result in certain incompatibility for those particular routines / interfaces. However, I don't think that we need to sacrifice all backwards compatibility everywhere just because of that.

Tomas

Tomas Hajny wrote

about possible data loss isn't cause by the compiler but rather by the user who sets his environment to use a character set which can only process a limited subset of all possible characters while he intends to use a bigger set.

My issue is with String = AnsiString, where they could or couldn’t support the full Unicode range. And the decision is same executable (think my company releasing software to the public) could behave differently depending on the end-users environment. CodeGear/Embarcadero saw the dangers in that, and thus forced String = UnicodeString on all platforms.

The issue can occur anywhere, not just loading files from a hard disk. eg: Reading data from a database. The data might be stored in UTF-8 or UTF-8 and contain above char(255) data. Somewhere in TDataset (or any component for that matter), that Unicode data might be assigned to a String type. On your Linux system String = AnsiString(<utf8>), but on my system in could be String = AnsiString(<latin1>). As a end-user you will not get or see any warning - you’ll just end up with rubbish data.

Now from what you are saying (if I understood correctly), is that we now need to tell every end-user they need to change their system’s code-page settings just to run our software. Well, that will not fly. Plus, Qt, C#, Mono and Java applications run and behave just fine on such systems, and those languages support the full Unicode range.

G*d knows why Delphi introduced code-page aware AnsiStrings, when they had a functioning UnicodeString type. Even worse is why FPC decided to follow that horrible direction years later. There really should be a limit to “delphi compatibility”.

Everybody knows how terrible the Variant data type is. Any respectable programmer tries to avoid it at all costs. AnsiString (with its variable internal encoding) is now equally bad. Unfortunately there are millions and millions of lines of Object Pascal code where the String type is used.

Tomas, I can only hope a decision on the RTL’s future comes sooner rather than later. In the mean time I’ll keep plugging away at tiOPF with FPC 3.x and created patches for FPC’s RTL where I can.

Regards,
Graeme

Jonas Maebe wrote

While still missing in the documentation, you can already do that with {$modeswitch unicodestrings}.

Very good to know. Is that in FPC 3.0.x too, or only FPC Trunk?

Regards,
Graeme

Am 09.05.2016 23:38 schrieb "Graeme Geldenhuys" <

It's also in 3.0.0. In fact it's one of the two modeswitches that has mode DelphiUnicode differ from mode Delphi (the other is for some specific Delphi compatibility regarding the default code page if I remember correctly).

Regards,
Sven

Sven Barth wrote

> > Very good to know. Is that in FPC 3.0.x too, or only FPC Trunk? It's also in 3.0.0. In fact it's one of the two modeswitches that has mode

Excellent. That should also fix another issue with FPC 3.x in tiOPF's D2009+ branch.

  function SomeName(const AParam: AnsiString); overload;
  function SomeName(const AParam: String); overload

Earlier the compiler couldn't differentiate between the two overloads. Now it can.

Regards,
Graeme

2016-05-09 16:34 GMT-03:00 Jonas Maebe :

Just to clarify:

1 - So, is there a way to set the codepage to be used by the String type regardless of the platform?

2 - If question 1 is true, what are the implications for the RTL? We can expect some unexpected behavior when doing that?

3 - If question 1 is true, what are the recomended way to do that?

4 - What should we do when we find flaws in the documentation? Should we report to Michael Van Caneyt here on the list? (from what I understand, he is the maintainer of the documentation). Should we open a bug report on the bugtracker about documentation?

Best regards

Jonas Maebe wrote

While still missing in the documentation, you can already do that with {$modeswitch unicodestrings}.

If this avoids the issues, Graeme found, why is this not enabled as default, as well in the user code as in the RTL interface ?

-Michael

Mazo Winst wrote

1 - So, is there a way to set the codepage to be used by the String type regardless of the platform?

Yes, apparently via SetMultiByteConversionCodePage(). I still need to check the documentation to find out more details. I think this is what Lazarus is doing for the LCL+FPC3.x too.

2 - If question 1 is true, what are the implications for the RTL? We can expect some unexpected behavior when doing that?

No idea. As far as I understand pretty much nothing has been done to the RTL regarding text encodings, so I wouldn't expect too much.

3 - If question 1 is true, what are the recommended way to do that?

Another good question. I would guess it must be called as soon as possible in your application and can't change afterwards. But that's just a guess.

4 - What should we do when we find flaws in the documentation? Should we report to Michael Van Caneyt here on the list?

I would suggest to always report any issues on Mantis. That way they are not lost or forgotten. Also, I believe Documentation bugs reported in Mantis are automatically assigned to Michael.

Regards,
Graeme

Michael Schnell wrote

You can't seriously suggest to dump all code that ever had been written in Pascal (Delphi Dialect).

No, just search and replace all String types with AnsiString and force a specific encoding. Or search and replace all String types with UnicodeString and run thorough tests over the code.

Regards,
Graeme

Why do you say that ?

What would you expect to be done ?

That is correct. Usually I fix bugs in the documentation quite quickly. Documentation is recompiled only when a new version of the compiler is released, so changes may not be , starting with version 3.0.0, I have set up the beginning of versioning of the on-line documentation, much like it is in Postgres.

It allows me to start publishing documentation for trunk.

Michael.

By that I meant a lot in the RTL is still missing regarding support for various string types, overloaded methods, Delphi compatibility etc.

Just try and compile the tiOPF’s “tiopf3” branch with FPC 3.x and see the amount of compiler errors. Saying that, I am working my way through those errors and implementing the missing bits in the RTL.

Regards,
Graeme

Graeme Geldenhuys wrote on Tue, 10 May 2016:

That is a non sequitur. The original question was about what happens if you change default code page of ansistrings. The absence of a Delphi2009+-compatible RTL, or the need to add {$mode delphiunicode} to compile code written for Delphi2009+, is completely unrelated to that. Please stop conflating everything in order to what comes across as attempts to merely voice the same complaints over and over again in every new email you send.

As to the original question: most RTL routines don't care about what the default code page is of ansistrings. Of the ones that do, most have been adapted to deal with changing the default ansistring code page (http://wiki.freepascal.org/FPC_Unicode_support#RTL_changes ). There is one notable set of exceptions that is not adapted in 3.0 (and incompletely in current trunk): SysUtils.ExecuteProcess() and related the TProcess() class.

Jonas

In our previous episode, Jonas Maebe said:

ExecuteProcess is in trunk since late march.

It is fairly self contained and could be merged to 3.0.2 technically.

I planned to continue with TProcess, but that stumbled on classes/tstringlist.

A list would be appreciated.

Because honestly, I do not understand the problem.

At the moment the situation is quite simple and clear IMHO:

- We have codepage aware single-byte strings (ansistring)

- The compiler handles conversion between these codepages transparantly. Some variables determine the exact behaviour.

- We have unicode strings.
Again, the compiler handles conversion transparantly.

- Basic pascal file IO routines handle unicode filenames transparantly in either single our double byte characters.

The rules governing conversion are somewhat obtuse in some corner cases, but other than that...

I don't see any problems except 1, the classes unit: the base classes are still using string = ansistring (with all that this implies).

That is a known issue which must be dealt with.

Michael.

marcov wrote on Tue, 10 May 2016:

I mentioned that it is in trunk, but it's incomplete: it does not translate the command line arguments to the code page of the environment in which the child is started, which means that the child may wrongly interpret these parameters. And it's definitely not something for 3.0.2 as far as I'm concerned, even with such fixes.

Jonas

In our previous episode, Jonas Maebe said:

It assumes for windows utf16, and for Unix filesystem encoding (e.g. unix.pp:254).

If you have more info, please share.

Come on Graham, I'm waiting for your explicit statement here rather than this continual FUD that you seem to enjoy injecting.

Michael Schnell wrote on Tue, 10 May 2016:

It does not avoid "the issues". All that modeswitch does is make "String" an alias for "Unicodestring" in the declaring module.

Jonas

Jonas Maebe wrote

It does not avoid "the issues". All that modeswitch does is make "String" an alias for "Unicodestring" in the declaring module.

Am I correct in understanding that it only affects individual units. So using it in my application, it would not affect the meaning of String in the RTL for instance (String in the RTL still means AnsiString(<whatever_default_codepage>) )?

Regards,
Graeme

Graeme Geldenhuys wrote on Wed, 11 May 2016:

Indeed, just like any other {$mode xxx} and {$modeswitch xxx} directive. I don't think we have any directive or switch that can cross compilation unit boundaries.

Technically, it would also be very hard to make e.g. {$modeswitch unicodestring} affect other units that you use. After all, the source code for those other units may not be available, and they would have to be recompiled if the meaning of "string" changed in their source code.

Jonas

Several reasons:

1. Backwards compatibility.
2. We'd need to check all code. Much of the code assumes char = 1 byte. That includes all other code besides RTL: Database code, web code etc.
3. It increases memory use.
All of a sudden, all your programs will use lots more memory. Maybe not important on a server, but it is e.g. in embedded.
4. For Delphi, the choice was clear: Windows is using UTF16, so all APIs are better off using UT16. Under unix-like OSes, UTF8 is a better choice IMHO.

There is no simple answer...

Michael.

marcov wrote on Tue, 10 May 2016:

The filesystem encoding is fine to find the binary on Unix, but the parameters you specify to that binary will be interpreted by that same binary once it's running. It will not interpret those parameters according to the filesystem encoding, but according to whatever is determined to be the DefaultSystemCodePage *by that executed binary*.

Jonas

In our previous episode, Jonas Maebe said:

I do understand that, and it will be fine most of the time.

I'm not sure what you are suggesting as alternative. Saving the encoding at startup, so it can be reused in cases like this, in case somebody change defaultsystemcodepage?

Or using defaultsystemcodepage instead of filesystem encoding?

Jonas Maebe wrote

Indeed, just like any other {$mode xxx} and {$modeswitch xxx} directive.

Thank you. That’s what I thought. I just wanted to double check.

Regards,
Graeme

Why not simply make the exec calls use RawByteString ?

It is then the responsability of the programmer.
99,99% of cases it will get passed the correct system encoding, and if need be the programmer can change it.

We should not fall in the trap of overengineering.

Michael.

Michael Van Canneyt wrote on Wed, 11 May 2016:

We have to look up the code page to use in the current program's environment at the time fpexec*() is called (or, in case the environment is specified explicitly, in that environment)

Why not simply make the exec calls use RawByteString ?

They will use RawByteString in any case, because we don't want the caller's string arguments to be converted before the routine is called, for two reasons:
1) a plain "ansistring" argument could result in data loss
2) we will have to convert the data afterwards to the correct code page anyway, so adding a potential extra conversion by declaring the argument as e.g. utf8string would serve no purpose (in the worst case it would introduce an additional, useless, conversion; in the best case it would move the conversion from the callee to the caller side)

It is then the responsability of the programmer.

It is the responsibility of the programmer when they call the pchar version of those routines. The string versions of all other fp*() routines also convert the arguments to the appropriate code page required by their purpose.

We should not fall in the trap of overengineering.

We should not fall in the trap of going for an inconsistent, buggy-by-design solution because doing it correctly is not trivial.

Jonas

In our previous episode, Michael Van Canneyt said:

I don't like that. The 3.x idea is to get rid of manual conversions and hack-and-convert-it-as-you-go encoding management, not just rebadge the old practices to rawbytestring.

You may not like it, but there is simply no other choice:

Since you don't know what the receiving program does with the receives arguments, all attempts to guess it are erroneous by definition.

Only the programmer knows what the receiving program will do (or should), so he must take it into account. Hence rawbytestring.

Don't pamper the programmer so much. He needs to make correct decisions.

Michael.

Michael Van Canneyt wrote on Wed, 11 May 2016:

It will either interpret them according to the code page specified by the environment, or not perform any both
cases, converting the arguments to the code page of the environment is the right thing to do.

Only the programmer knows what the receiving program will do (or should), so he must take it into account. Hence rawbytestring.

If it is an FPC program, then if it is compiled with FPC 2.6.x it will not reinterpret the command line arguments at all, while when compiled with FPC 3.x it will reinterpret the command line arguments. The caller of such a program should not have to worry with which version of FPC it has been compiled.

Don't pamper the programmer so much. He needs to make correct decisions.

They have to correctly set or specify the environment for the program they are executing when they want to force a specific encoding for its command line arguments.

Jonas

And in the case it makes an assumption of the code page, regardless of
environment variables ?

(don't say that doesn't happen. It does, I know a programmer that does so)

Michael.

Michael Van Canneyt wrote on Wed, 11 May 2016:

The caller can work around such bugs by either
a) using the pchar version of fpexec, or
b) specifying the code page that this target program uses in the environment used to invoke it

Jonas

a) obviously
b) As said, the target program completely ignores the environment.

I was just trying to point out that while your solution is undoubtedly correct in the large majority of cases (let's assume 99,99%), it is not a rock-hard guarantee.

IMHO, in this large majority of cases, using RawByteString will be correct as well, since chances that the calling program uses a different codepage from the called one are very small. (in casu: UTF8)

I am not saying that attempting to convert to the code page used by the environment is wrong (except maybe in an small minority as pointed out), but I do think it is overengineering.

Michael.

Michael Van Canneyt wrote on Wed, 11 May 2016:

b) is exactly why you have to specify the code page that this *target program* uses in the environment when executing it, so that the invoking FPC program will convert the parameters to this code page.

I was just trying to point out that while your solution is undoubtedly correct in the large majority of cases (let's assume 99,99%), it is not a rock-hard guarantee.

I never claimed it was. I only said it is the only possible correct behaviour. It obviously cannot fix other broken programs, although as explained it is sufficiently flexible to deal with them.

Jonas

You are now assuming that this is possible. This may not be the case.

We clearly have different understandings of the words 'correct behaviour' then :-)

Michael.

Michael Van Canneyt wrote on Wed, 11 May 2016:

Why would it not be possible? Please be more concrete, because right now I feel like I'm arguing against my own imagination, which is not very useful.

Jonas

In our previous episode, Michael Van Canneyt said:

You are right, if that scenario was the common one. Note that Jonas hasn't named a scenario where it actually happens, so keeping it all manual for one basket case is IMHO not an option.

(and even then better served by a "no translation" boolean than forgetting about automatic conversions alltogether)

I honestly don't understand this argument: in 99,99% of all cases, the code will just work if you use RawByteString, since everything will be in the codepage of the environment anyway ?

Michael.

In our previous episode, Michael Van Canneyt said:

I honestly don't understand this argument: in 99,99% of all cases, the code will just work if you use RawByteString, since everything will be in the codepage of the environment anyway ?

Well, we already convert to target encoding on windows(utf16), and that has to stay anyway.