How is the twide3.pp test supposed to work?
If I run this in windows, the widestring is converted to an ansistring of length 5 with some wrong chars. Do I need to set a special code page and how can I do that?
Vincent
Mailing lists / fpc-devel / 2007-07
twide3.pp on windows
18 messages · 5 people ·
- Yury Sidorov 6
- Daniël Mantione 4
- Vincent Snijders 4
- Maxim Ganetsky 3
- Jonas Maebe 1
Reply by email Subscribe to this list Your subscription
Read-only archive. Use j and k to move between messages.
From: "Vincent Snijders"
Show quoted text 5 lines
How is the twide3.pp test supposed to work?
If I run this in windows, the widestring is converted to an
ansistring of length 5 with some wrong chars. Do I need to set a
special code page and how can I do that?
As I see, on Windows widestring is converted to ansi encoding not utf8. This test will work only if current locale is the same as 'łóżka' string. In other cases the string is converted to 'lozka'.
This test should be skipped on Windows or fixed.
Possible fix is to replace:
if (w'łóżka') thenby:
ww:='łóżka';
a:=ww;
if (wa) thenYury.
How is the twide3.pp test supposed to work? If I run this in windows, the widestring is…
Vincent Snijders wroteHow is the twide3.pp test supposed to work?
It assumes that the testsuite environment is configured to use utf8. I don't know how to make it encoding-independent, nor how to configure a Windows "console session" to always use utf8 (the testsuite scripts of the automated testsuite runs on Linux, Solaris and Mac OS X all explicitly set the LANG environment variable to ensure this).
Jonas
From: "Jonas Maebe"
Show quoted text 10 lines
On 16 Jul 2007, at 16:32, Vincent Snijders wrote:
> How is the twide3.pp test supposed to work?
It assumes that the testsuite environment is configured to use utf8.
I don't know how to make it encoding-independent, nor how to
configure a Windows "console session" to always use utf8 (the
testsuite scripts of the automated testsuite runs on Linux, Solaris
and Mac OS X all explicitly set the LANG environment variable to
ensure this).
Our Windows RTL use current ansi code page to perform widestring<->ansistring conversions. utf8 can not be used.
Yury.
Op Mon, 16 Jul 2007, schreef Yury Sidorov:
Show quoted text 13 lines
From: "Jonas Maebe"
> On 16 Jul 2007, at 16:32, Vincent Snijders wrote:
>
> > How is the twide3.pp test supposed to work?
>
> It assumes that the testsuite environment is configured to use utf8. I
> don't know how to make it encoding-independent, nor how to configure a
> Windows "console session" to always use utf8 (the testsuite scripts of
> the automated testsuite runs on Linux, Solaris and Mac OS X all
> explicitly set the LANG environment variable to ensure this).
Our Windows RTL use current ansi code page to perform widestring<->ansistring
conversions. utf8 can not be used.
How to set the current code page on Windows? The actual code page doesn't matter, but a fixed code page is needed for a deterministic test.
Daniël
From: "Daniël Mantione"
Show quoted text 23 lines
Op Mon, 16 Jul 2007, schreef Yury Sidorov:
> From: "Jonas Maebe"
> > On 16 Jul 2007, at 16:32, Vincent Snijders wrote:
> >
> > > How is the twide3.pp test supposed to work?
> >
> > It assumes that the testsuite environment is configured to use
> > utf8. I
> > don't know how to make it encoding-independent, nor how to
> > configure a
> > Windows "console session" to always use utf8 (the testsuite
> > scripts of
> > the automated testsuite runs on Linux, Solaris and Mac OS X all
> > explicitly set the LANG environment variable to ensure this).
>
> Our Windows RTL use current ansi code page to perform
> widestring<->ansistring
> conversions. utf8 can not be used.
How to set the current code page on Windows? The actual code page
doesn't
matter, but a fixed code page is needed for a deterministic test.
The code page (locale) is set in Control Panel for whole user session (or whole system).
It is not possible to set utf8 as locale.
We can create flag in System unit which control how widestring<->ansistring should be performed to enable utf8 on Windows.
Yury.
Op Mon, 16 Jul 2007, schreef Yury Sidorov:
The code page (locale) is set in Control Panel for whole user session (or whole system). It is not possible to set utf8 as locale.
I know. How do you set it to a specific locale for the current application?
We can create flag in System unit which control how widestring<->ansistring should be performed to enable utf8 on Windows.
No, you cannot (at least not with full functionality), since uppercasing and compare operations on UTF-8 ansistrings you need a Unicode library, which Windows does only provide in UTF-16 flavour.
Daniël
From: "Daniël Mantione"
Show quoted text 9 lines
Op Mon, 16 Jul 2007, schreef Yury Sidorov:
> The code page (locale) is set in Control Panel for whole user
> session (or
> whole system).
> It is not possible to set utf8 as locale.
I know. How do you set it to a specific locale for the current
application?
widestring to ansistring conversion is performed using WideCharToMultiByte. Currently it is done using ANSI code page specified in Control Panel.
There is no per application code page setting in Windows.
Show quoted text 9 lines
> We can create flag in System unit which control how
> widestring<->ansistring
> should be performed to enable utf8 on Windows.
No, you cannot (at least not with full functionality), since
uppercasing
and compare operations on UTF-8 ansistrings you need a Unicode
library,
which Windows does only provide in UTF-16 flavour.
WideCharToMultiByte can convert from widestring to utf8 if corresponding flag is specified. This flag is available in Windows 98/Me, Windows NT 4.0 and later.
MultiByteToWideChar do reverse conversion.
This functions already used in System unit for Windows.
Yury.
Op Mon, 16 Jul 2007, schreef Yury Sidorov:
Show quoted text 14 lines
From: "Daniël Mantione"
> Op Mon, 16 Jul 2007, schreef Yury Sidorov:
>
> > The code page (locale) is set in Control Panel for whole user
> > session (or
> > whole system).
> > It is not possible to set utf8 as locale.
>
> I know. How do you set it to a specific locale for the current
> application?
widestring to ansistring conversion is performed using WideCharToMultiByte.
Currently it is done using ANSI code page specified in Control Panel.
There is no per application code page setting in Windows.
There must be, because the CRT unit currently seems to do such vodoo (which is questionable behaviour by the way).
Show quoted text 14 lines
> > We can create flag in System unit which control how
> > widestring<->ansistring
> > should be performed to enable utf8 on Windows.
>
> No, you cannot (at least not with full functionality), since uppercasing
> and compare operations on UTF-8 ansistrings you need a Unicode library,
> which Windows does only provide in UTF-16 flavour.
WideCharToMultiByte can convert from widestring to utf8 if corresponding flag
is specified. This flag is available in Windows 98/Me, Windows NT 4.0 and
later.
MultiByteToWideChar do reverse conversion.
This functions already used in System unit for Windows.
Yes, but how would you implement ansiuppercase and friends when utf8 flag is set to true?
Daniël
From: "Daniël Mantione"
Show quoted text 21 lines
Op Mon, 16 Jul 2007, schreef Yury Sidorov:
> From: "Daniël Mantione"
> > Op Mon, 16 Jul 2007, schreef Yury Sidorov:
> >
> > > The code page (locale) is set in Control Panel for whole user
> > > session (or
> > > whole system).
> > > It is not possible to set utf8 as locale.
> >
> > I know. How do you set it to a specific locale for the current
> > application?
>
> widestring to ansistring conversion is performed using
> WideCharToMultiByte.
> Currently it is done using ANSI code page specified in Control
> Panel.
> There is no per application code page setting in Windows.
There must be, because the CRT unit currently seems to do such vodoo
(which is questionable behaviour by the way).
Yes. It is possible to specify code pages for console, gui controls, etc. But application must do it in runtime.
Show quoted text 22 lines
> > > We can create flag in System unit which control how
> > > widestring<->ansistring
> > > should be performed to enable utf8 on Windows.
> >
> > No, you cannot (at least not with full functionality), since
> > uppercasing
> > and compare operations on UTF-8 ansistrings you need a Unicode
> > library,
> > which Windows does only provide in UTF-16 flavour.
>
> WideCharToMultiByte can convert from widestring to utf8 if
> corresponding flag
> is specified. This flag is available in Windows 98/Me, Windows NT
> 4.0 and
> later.
> MultiByteToWideChar do reverse conversion.
>
> This functions already used in System unit for Windows.
Yes, but how would you implement ansiuppercase and friends when utf8
flag
is set to true?
Oh, yes it will not work on Windows, because utf8 is not supported here.
Then, twide3.pp should be skipped on Windows, since it is not possible to do it in locale independant way.
Yury.
Op Mon, 16 Jul 2007, schreef Yury Sidorov:
Show quoted text 5 lines
> There must be, because the CRT unit currently seems to do such vodoo
> (which is questionable behaviour by the way).
Yes. It is possible to specify code pages for console, gui controls, etc. But
application must do it in runtime.
Perfect, this is what twide3 needs.
Oh, yes it will not work on Windows, because utf8 is not supported here. Then, twide3.pp should be skipped on Windows, since it is not possible to do it in locale independant way.
Well, the test could be adapted to use some kind of code page. It could work on all platforms then.
Daniël
From: "Daniël Mantione"
Show quoted text 11 lines
Op Mon, 16 Jul 2007, schreef Yury Sidorov:
> > There must be, because the CRT unit currently seems to do such
> > vodoo
> > (which is questionable behaviour by the way).
>
> Yes. It is possible to specify code pages for console, gui
> controls, etc. But
> application must do it in runtime.
Perfect, this is what twide3 needs.
It is not possible to change code page used for wideansi translation per applicaton.
Show quoted text 10 lines
> Oh, yes it will not work on Windows, because utf8 is not supported
> here.
>
> Then, twide3.pp should be skipped on Windows, since it is not
> possible to do
> it in locale independant way.
Well, the test could be adapted to use some kind of code page. It
could
work on all platforms then.
This test will work on Windows only if utf8 support will be added to Windows rtl.
Yury.
Yury Sidorov wrote 29 lines
From: "Daniël Mantione"
> Op Mon, 16 Jul 2007, schreef Yury Sidorov:
>
> > > There must be, because the CRT unit currently seems to do such > >
> vodoo
> > > (which is questionable behaviour by the way).
> >
> > Yes. It is possible to specify code pages for console, gui >
> controls, etc. But
> > application must do it in runtime.
>
> Perfect, this is what twide3 needs.
It is not possible to change code page used for wideansi translation
per applicaton.
> > Oh, yes it will not work on Windows, because utf8 is not supported >
> here.
> >
> > Then, twide3.pp should be skipped on Windows, since it is not >
> possible to do
> > it in locale independant way.
>
> Well, the test could be adapted to use some kind of code page. It could
> work on all platforms then.
This test will work on Windows only if utf8 support will be added to
Windows rtl.
I changed my windows settings to Polish and it now uses codepage 1250 (before it was 1252), now it runs successfully.
Vincent
Vincent Snijders пишет:
Show quoted text 8 lines
Yury Sidorov schreef:
> From: "Daniël Mantione"
>> Op Mon, 16 Jul 2007, schreef Yury Sidorov:
>>
>> > > There must be, because the CRT unit currently seems to do such >
>> > vodoo
>> > > (which is questionable behaviour by the way).
>> >
<snip>
I changed my windows settings to Polish and it now uses codepage 1250 (before it was 1252), now it runs successfully.
I think the hack from CRT unit can be useful here too. Just do:
OldConsoleOutputCP:=GetConsoleOutputCP;
SetConsoleOutputCP(1250);at the beginning of test and
SetConsoleOutputCP(OldConsoleOutputCP);at the end.
OldConsoleOutputCP is of type Word.
Best regards,
Maxim Ganetsky
Maxim Ganetsky wrote 21 lines
Vincent Snijders пишет:
> Yury Sidorov schreef:
>> From: "Daniël Mantione"
>>> Op Mon, 16 Jul 2007, schreef Yury Sidorov:
>>>
>>> > > There must be, because the CRT unit currently seems to do such >
>>> > vodoo
>>> > > (which is questionable behaviour by the way).
>>> >
<snip>
> I changed my windows settings to Polish and it now uses codepage 1250
> (before it was 1252), now it runs successfully.
I think the hack from CRT unit can be useful here too.
Just do:
OldConsoleOutputCP:=GetConsoleOutputCP;
SetConsoleOutputCP(1250);
No, this does not work. GetACP still returns the system setting.
Vincent
Vincent Snijders пишет:
Show quoted text 25 lines
Maxim Ganetsky schreef:
> Vincent Snijders пишет:
>> Yury Sidorov schreef:
>>> From: "Daniël Mantione"
>>>> Op Mon, 16 Jul 2007, schreef Yury Sidorov:
>>>>
>>>> > > There must be, because the CRT unit currently seems to do such
>>>> > > vodoo
>>>> > > (which is questionable behaviour by the way).
>>>> >
>
> <snip>
>
>> I changed my windows settings to Polish and it now uses codepage 1250
>> (before it was 1252), now it runs successfully.
>
> I think the hack from CRT unit can be useful here too.
> Just do:
>
> OldConsoleOutputCP:=GetConsoleOutputCP;
> SetConsoleOutputCP(1250);
>
No, this does not work. GetACP still returns the system setting.
Where do you see GetACP? Instead of GetACP pass 1250.
Best regards,
Maxim Ganetsky
Maxim Ganetsky wrote 6 lines
Vincent Snijders пишет:
> No, this does not work. GetACP still returns the system setting.
>
Where do you see GetACP? Instead of GetACP pass 1250.
I don't see, it is what is used by
WideCharToMultiByte(CP_ACP, 0, source, len+1, nil, 0, nil, nil);in Win32Wide2AnsiMove.
Vincent
Vincent Snijders пишет:
Show quoted text 11 lines
Maxim Ganetsky schreef:
> Vincent Snijders пишет:
>> No, this does not work. GetACP still returns the system setting.
>>
>
> Where do you see GetACP? Instead of GetACP pass 1250.
>
I don't see, it is what is used by
WideCharToMultiByte(CP_ACP, 0, source, len+1, nil, 0, nil, nil);
in Win32Wide2AnsiMove.
Ah, now I see. Having looked at the code of the test I think that of course my suggestion won't work since it changes console codepage and the test doesn't work with it.
Best regards,
Maxim Ganetsky