Hello,

SysUtils for Windows utilizes only A Windows APIs, while it could use W APIs in Windows NT. The attached patch starts correcting improving the situation.

Advantages of the patch:

* Allows for unicode file names in TStringList.LoadFromFile if you install paswstrings as the unicode manager.

Disadvantages of the patch:

* None that I can imagine.

Waiting for comments.

The related bug tracker item is:

http://mantis.freepascal.org/view.php?id=21114

As you know, we are very keen on backwards compatibility.

Are you sure that if a AnsiString filename which contains codepage-specific characters is used to open a file, it will still open the same file after this patch is applied ?

In particular, does the opening of a DOS-style filename (Program~1\blah) still work as it used to ?

Michael.

Felipe Monteiro de Carvalho wrote

The related bug tracker item is: http://mantis.freepascal.org/view.php?id=21114

IMO This should not be done that way (at all):
MS does it by respecting the PE flag for unicode and expects strings accordingly: If the PE says it's unicode, a default string should be unicode, (so apiW) if the header says ansi the default should be apiA. Just my three cents.
But that still requires changes, I agree with that part. This might be an issue for xplatform, although I believe nixes have the same behaviour.

As you know, we are very keen on backwards compatibility....

Ummm, I'll need to investigate that. But if it seams risky we could always add "and (DefaultSystemCodePage = CP_UTF8)" to the if, which would disable it for legacy code and enable it only for Unicode applications. If you use SetMultiByteConversionCodePage then you are clearly saying you desire Unicode support.

Thaddy wrote

IMO This should not be done that way (at all): MS does it by respecting the PE flag for unicode and expects strings accordingly: If the PE says it's unicode, a default string should be unicode, (so apiW) if the header says ansi the default should be apiA.

The Microsoft way is not the same as the Free Pascal way. We are not required to immitate them when implementing our routines.

The Microsoft way has nasty side effects:
1> makes it impossible to support Unicode and support Windows 9x at the same time, but I doubt that Free Pascal wants to drop Windows 9x support (even if it is obsolete).
2> Breaks the interface because the string type changed

No, this is not true, not at all, to the contrary, that's what the PE flag is for:

If you want to compile for any windows version, choose ansi mode and the OS "respects" and "expects" that.
If you want a unicode default (i.e. win2000 and higher) choose unicode mode. The library should be totally opaque when the programmer decides what he wants.
What you are doing is making changes in the generated code that shoukld be resolved at compile time.
This can be resolved at link time, but I expect it should be resolved at compile time.
That's why I agree with you that changes are necessary. (Although the PE flag is a linker instruction)

What we are required to do is respect the OS we compile for.

No, this is not true, not at all, to the contrary, that's what the PE flag is for:

If you want to compile for any windows version, choose ansi mode and the OS "respects" and "expects" that.
If you want a unicode default (i.e. win2000 and higher) choose unicode mode. The library should be totally opaque when the programmer decides what he wants.
What you are doing is making changes in the generated code that shoukld be resolved at compile time.
This can be resolved at link time, but I expect it should be resolved at compile time.
That's why I agree with you that changes are necessary. (Although the PE flag is a linker instruction)

What we are required to do is respect the OS we compile for.

Out of interest: Which flag are you talking about? Because I'm not aware of any such flag.

Regards,
Sven

There is no option to set anything unicode related in the linker according to this page:
http://msdn.microsoft.com/en-us/library/y0zzbyt4.aspx

So what are you talking about?

Regards,
Sven

In our previous episode, Felipe Monteiro de Carvalho said:

1> makes it impossible to support Unicode and support Windows 9x at the same time, but I doubt that Free Pascal wants to drop Windows 9x support (even if it is obsolete).

Afaik we already have. Recent versions require patches for win9x

It is known which versions are unicode. I am merely refering to the OS version flags. These are the ones which determine if the windows version is unicode or ansi.
There's no point in running an ansi compiled exe on a unicode platform. Once a programmer decides his/hers minimum requirements, the compiler should obey. The OS does (up to a certain point)
These decisions should be made by the compiler based on platform choice, not in rtl code. That should be optional (as of 2010/XE this is also D compatible)

In C on Windows the UNICODE symbol controls whether undecorated functions should use the A or W versions, and what the TCHAR symbol translates to. There's nothing stopping an application from calling the A or W versions explicitly, and it's entirely possible for an otherwise Unicode app to use the A versions for certain functions. That's handled by the compiler's pre-processor, not the linker and not the loader.

Delphi does the same thing, except the change was built into the language, rather than done using a conditional symbol, and the A or W decision was hard-coded rather than wrapping them in {$IFDEF UNICODE} blocks.

Craig Peterson wrote

In C on Windows the UNICODE symbol controls whether undecorated functions should use the A or W versions, and what the TCHAR symbol translates to.

This exactly the case and I do noy suggest this to be dropped. (This is not only C but also FP-C btw)

The case is that the default A/W should be determined by the platform and the exceptions by the programmer. Just like in any proper language be it C or FP-C <smile>

The check you are talking about in C++ is a compile time define that just switches the defines for e.g. CreateProcess to CreateProcessW or CreateProcessA (depending on whether UNICODE is defined or not when including the Windows headers).

There is no other "magic" in Windows that changes the , Sven

Otherwise you can hand-code it.
The A/W versions of the windows API have been in Delphi at least since D3, but I can check D2 in a minute.
Of course it's a compile time switch! Point made by me in the first place.

BTW you can even fool delphi and fpc to behave irratically (do what I want) by doing things like -uFPC -Mdelphi -dVER200 in fpc or defining -dVER200 when you have a D7 (VER150) .
Sometimes handy to know the compilers (be it C or FP-C) at least understand the programmers meaning.
(Just in case you wonder why I would fool the compiler to be over his age: parsing some badly behaving sourcecode)

Craig Peterson wrote

Delphi does the same thing, except the change was built into the language, rather than done using a conditional symbol, and the A or W decision was hard-coded rather than wrapping them in {$IFDEF UNICODE} blocks.

I am not sure about this, (I do not know if the latest production version has this) but I think you can (could) undefine unicode. I propose FPC follows a similar strategy.
We already did it for Kol, which has a (almost) totally opaque KOLstring/char type depending on switching on UNICODE_CTRLS or ANSI. (And we had that switch some time before FPC)

No, you said that it's a PE flag. That is a completely different thing than a preprocessor macro.

Regards,
Sven

06.02.2012 20:39, Felipe Monteiro de Carvalho пишет:

So, this is basically a first step of locking Windows RTL to use utf-8 by default and reducing chances it ever will call 'W' API without conversions?

Sergei

But in FPC you'll need to recompile the Windows RTL if you want to have UNICODE defined (thus having e.g. CreateProcess default to CreateProcessW instead of CreateProcessA). In C you just include the header with the define either defined or not.

Regards,
Sven

That is another point that worries me too. Windows NT + are not native UTF8 but UTF16.
That's why my suggestion to make the default string type fully opaque per platform requirements.
That can be done on the compiler level with little effort in rtl/fcl. (For KOL it took weeks, not months)

Of course, KOL is a framework, not a compiler.
I merely want to state it can be done in a proper way.

Nope, again no: this has to do with the RTL/FCL
Any KOL program (try it) compiles in either unicode (-dUNICODE_CTRLS --Mdelphi -uFPC -dVER150) or ansi (skip the unicode_ctrls) and just plain works as expected. Not only with a Delphi compiler, also with fpc 2.6.0 and higher.
This has nothing to do with the compiler itself. That behaves already like it should.

06.02.2012 23:45, Marco van de Voort пишет:

Just in case: I have written a small unicows.dll loader unit (under 10KBytes source code, in contrast with >300KByte Rob Kennedy's one) that can be used as a 'last goodbye' for win9x, allowing to switch most APIs to 'W'.
The only issue is that it needs to be executed as early as possible, ideally before system.pp initialization. This probably can be solved by redefining the entry point passed to linker, or by placing its code into yet another sysinit unit, which is selected for win9x.

If this is interesting I can provide more details.

Sergei

Sergei Gorelkin wrote

under 10KBytes

Any decent code under 10k is interesting <smile>

Thaddy wrote

UNICODE} blocks. I propose FPC follows a similar strategy. We already did it for Kol, which has a (almost) totally opaque KOLstring/char type depending on switching on

Meaning/implies: when you are the compiler guardian/owner you can simply enforce string to unicode or ansi or ebdic or whatever based on the platform/os defaults. Like most of FPC already does. And leave it to the programmer to deviate by using the A alternative api. We could not do that since we do not have an underlying compiler to our kol framework. So we chose KolString. FPC can define it opaque as a compiler switch with a default.

IMO a more radical solution is desireable, WRT Win9x. Did anybody test already, how FPC/Lazarus apps behave on such a system, which does not support themes etc., and does not even support Unicode without system updates?

I'd split the Win32 target into WinNT (W) and Win9x (A), so that the users can decide whether to support

As you know, we are very keen on backwards compatibility.

ACK.

Are you sure that if a AnsiString filename which contains codepage-specific characters is used to open a file, it will still open the same file after this patch is applied ?

Win32 filesystems store long filenames in Unicode (UTF-16), I'd not expect problems in addressing files using UnicodeStrings.

In particular, does the opening of a DOS-style filename (Program~1\blah) still work as it used to ?

I'm not sure, but aren't DOS filenames restricted to the OEM codepage? There exists no direct correspondence between long filenames and their 8.3 names, which may have any number after "~", depending on the contents of the directory. Such names never should occur hard-coded in an application.

DoDi

Sergei Gorelkin wrote

So, this is basically a first step of locking Windows RTL to use utf-8 by default

No, it will not use UTF-8 by default because that would break compatibility. It will use the system encoding by default and you can use SetMultiByteConversionCodePage to change it to use UTF-8.

And this is not about just about Windows, but all platforms of the Ansi RTL. Just that most other platforms are already using APIs which can handle Unicode so they don't need changes.
SetMultiByteConversionCodePage is not specific to the Windows RTL.

and reducing chances it ever will call 'W' API without conversions?

This is a very strange argument. You cannot oppose something by arguing that then it will be successful and people will not want to implement your way. There are no such barriers in a volunteer led project. If someone wants the UnicodeString RTL ... he can write it. I am not against another RTL existing which will use UnicodeString or which will use 1 different string in each platform or 1 different string for each weather condition. But I just don't want to use it. I want to use the Ansi RTL + SetMultiByteConversionCodePage to make it UTF-8 capable, so that's what I am writing.

So do you think that the Ansi RTL should not exist? Or you just think it should exist, but in disgrace, using obsolete APIs and with APIs which don't work properly to get people to give up on it? It has already been clearly stated that the Ansi RTL will not be deleted, and this RTL will not follow Delphi Unicode (because that would break compatibility). So if we will have an Ansi RTL, then people that use it intensively should be allowed to improve it. Make it respect SetMultiByteConversionCodePage fully and make it work reliably, make it support UTF-8 and use newer APIs when possible. Like my proposed patch does for improving the Unicode support of
TStringList.LoadFromFile / SaveToFile.

Let me intercede here:

The plans are already laid out.

The ANSI RTL will be exactly that: ANSI;
No unicode or UTF-8 hacks will be added to it.

The planned unicode RTL will be unicode, and by corollary will support UTF-8.

So any patches to "utf8-enable" the Ansi RTL will not be accepted.

You'll have to wait for the unicode RTL or start work on it yourself.

If you want, I can send you the details in private, and you can decide for yourself what to do.

Michael.

07.02.2012 1:49, Thaddy пишет:

On 6-2-2012 22:14, Sergei Gorelkin wrote: > under 10KBytes Any decent code under 10k is interesting <smile>

Well, here is it attached.

It was actually written for Delphi 7 with its "specific" linker. Therefore a big part of it is Delphi specific (works with FPC, too, but FPC-only version would be even easier and smaller). Delphi specifics also make it is incompatible with exe compressors, because it needs to read the original image.
Add this unit to the project uses clause, put unicows.dll on the PATH, 'W' APIs should now work :)

Sergei

In our previous episode, Felipe Monteiro de Carvalho said:

No, it will not use UTF-8 by default because that would break compatibility. It will use the system encoding by default and you can use SetMultiByteConversionCodePage to change it to use UTF-8.

Note that this might sound useful now ansistring without codepage is still the default string, but if the default string type changes, this is not really useful.

IOW it is a short term hack with a long time support burden IMHO.

That being said, I do think we will need to start migrating to the W versions in time.

But first a proper decision must be made about the default string type, and such things shouldn't be slipped by the back door.

In our previous episode, Hans-Peter Diettrich said:

it will still be on the level of winNT.

But I do think that a win9x vs winnt split is unavoidable in time. Specially since otherwise one gets in the current situation that win9x is occasionally fixed for a release, but in reality constantly broken in between.

Hello,

Here is a second version of the patch. I think it is better then my first one because it will not make a conversion for legacy applications (they don't need it anyway) and it will only improve the UTF-8 handling in the Ansi RTL which already partially exists in SetMultiByteConversionCodePage.

I tested that it works correctly with a LCL application like this:

procedure TForm1.Button1Click(Sender: TObject);
var
  lStrings: TStringList;
begin
  SetMultiByteConversionCodePage(CP_UTF8);
  lStrings := TStringList.Create;
  lStrings.SaveToFile(Edit1.Text);
  lStrings.Free;
end;

http://img404.imageshack.us/img404/5478/lazarusutf81.png

And the best of all! It is not mandatory! If you like the old behavior just write like this:

procedure TForm1.Button1Click(Sender: TObject);
var
  lStrings: TStringList;
begin
  //SetMultiByteConversionCodePage(CP_UTF8);
  lStrings := TStringList.Create;
  lStrings.SaveToFile(Edit1.Text);
  lStrings.Free;
end;

And it works exactly like before (which means it will generate a wrong filename in this case).

So here we are: 10 lines of code away from having TStringList finally work with full Unicode support and with full integration with Lazarus. How many lines of code till a perfect integration of the RTL with Lazarus, a perfect UTF-8 support in the RTL? Maybe 100? Hardly a terrible maintenance burdain.

Will we be able to close the division among us as get our 10 lines of code going and benefit hundreds of thousands of Lazarus users? Or are these 10 lines of code a major treat to the universe as we know it? Stay tunned for the next chapters of the Unicode wars =)

Felipe Monteiro de Carvalho wrote

Stay tunned for the next chapters of the Unicode wars =)

Here we agree... ;) *fetches popcorn*

Regards,
Sven

Regardless of other points raised in the discussion so far and their validity or not, there's one more thing not clear to me. Your patch suggest to use either *A or *W call depending on a condition evaluated at run-time. What's the benefit? If your intention was to keep the programs working with Win9x, this won't help because the program containing unresolved imports wouldn't start anyway. Is there any other reason?

Tomas

2012/2/8 Tomas Hajny :

If your intention was to keep the programs working with Win9x, this won't help because the program containing unresolved imports wouldn't start anyway. Is there any other reason?

Windows9x has all W functions, Microsoft added them as stubs to avoid this kind of problem. They all exist and they all simply don't do anything so a runtime check is appropriate. There will be no unresolved imports error. We do this in a similar fashion in Lazarus since maybe 5 years without problems and it always ran correctly in Windows 9x.

Marco van de Voort wrote

Note that this might sound useful now ansistring without codepage is still the default string, but if the default string type changes, this is not really useful.

You cannot change the default string type without killing the Ansi RTL and destroy backwards compatibility.

IOW it is a short term hack with a long time support burden IMHO.

I can maintain it. I have here the final solution to Lazarus problems 100 lines of code away, it is a trivial thing to maintain. As long as the Ansi RTL exists, of course.

Right. Other (post-9x) functions cause more trouble, because these prevent an application from 's why Win9x support
should either provide workarounds for all API functions, which are not natively available, or (much easier) should exclude all these declarations. One more reason for a dedicated Win9x RTL.

Microsoft added them as stubs to avoid this kind of problem. They all exist and they all simply don't do anything

They *do* the string conversion, necessary for calling the "native" functions internally. In Win9x the W functions convert strings into Ansi, on NT the A versions convert strings into WideString. So what's the real benefit of conditionally calling A/W functions, and depending on what?

There exist at least two cases, worth more consideration:

1) UTF-8 strings. These require a conversion, regardless of subsequent calls of the A or W function version. Here the knowledge of Windows version can prevent excess A/W conversions inside the OS implementation.

2) Filenames. Here I'm not sure, because even Win9x uses UTF-16 filenames on the disks, and thus the W versions should be preferred. OTOH the MSDN "Note that Unicode support on Windows Me/98/95 requires Microsoft Layer for Unicode." suggests that testing for *this* update is the only relevant check, in any A/W calling decision.

DoDi

Hans-Peter Diettrich wrote

They *do* the string conversion, necessary for calling the "native" functions internally. In Win9x the W functions convert strings into Ansi, on NT the A versions convert strings into WideString. So what's the real benefit of conditionally calling A/W functions, and depending on what?

I don't have Windows 9x to test so I was just playing it safe by making it work like before for people that don't want UTF-8 and by supporting at least ASCII for people using UTF-8 in Windows 9x, which is pretty much all that it would give us reliably anyway since other chars will break between different computers. The if is also efficient, because in Windows 9x it will avoid a double conversion. Ansi or UTF-8 -> UTF-16 -> Ansi

I was trying to fix my problem while causing no impact for anyone else. But sure, for me it is also fine to go W all the way if someone can test that it works in Windows 9x or if someone does the decision to kill support for it.

Felipe Monteiro de Carvalho wrote

Windows9x has all W functions,

No it does not. The Microsoft layer for Unicode is a redistributable DLL that provides W versions of functions and calls the A versions internally. Win9x does export a bunch of W functions so you can link to both versions and conditionally call the correct one, but if you actually try calling the W versions the vast majority will immediately fail with ERROR_CALL_NOT_IMPLEMENTED.

The platform SDK used to say which platforms supported the W versions, but that information hasn't been included in quite a while.

For simplicity's sake, assuming continued Win9x support is even desirable, the RTL should call the generic versions, and Windows.pas should have {$IFDEF UNICODE} blocks to decide between the A and W versions. The TNT Unicode controls did it the way you're suggesting, by deciding at runtime, so it is possible, but it would overly complicate the RTL code for an increasingly irrelevant platform.

Craig Peterson wrote

 The TNT Unicode controls did it the way you're suggesting, by deciding at runtime, so it is possible, but it would overly complicate the RTL code for an increasingly irrelevant platform.

We are big boys, I think that we can handle the imense complexity of 1 if clause in 10 functions.

Presumably lumping pre-W2K NT in with '9x. I'm thinking of http://mantis.freepascal.org/view.php?id=18803 which wasn't really resolved, and which was caused by the return code of a function not implemented in older OSes not being checked.

I do not think that you will break anything if you make W api or A api default based on platform. This can be switched in the winapi units. That will take little effort, as I explained: we did it already for KOL.