-----------
Mime-Version: 1.0
X-Mailer: mPOP Web-Mail 2.19
X-Originating-IP: [194.150.126.26]
Date: Tue, 23 May 2006 18:27:07 +0400
Reply-To: =?koi8-r?Q?=F0=A3=D4=D2=20=EB=CF=D3=C1=D2=C5=D7=D3=CB=C9=CA?= Content-Type: text/plain; charset=koi8-r
Content-Transfer-Encoding: 8bit

The bug was closed during weekend.

This is somewhat lengthy and special feedback.

(It concerned the behavior of writeln() with symbols from the upper half of the 8 bit character table. Those symbols were output as question marks.) By the way, as I get it, some string conversions are performed only on the lower 128 symbols too (widestring->common string).

This bug is solved.

However (as Maxim Ganetsky, who invented the patch, noted) the windows console is a mess.

There still is a not very nice problem.

I describe it below for fpc 2.1.x, win32, winxpsp2, codepages cp-1251 and cp-866 (both are russian, i.e. cyrillic). I'm sure, there are other non-english character sets, suffering from this inconveniency. I don't think that unices are featureless in this aspect too.

In console mode (I use Far manager) real character set dep

However (as Maxim Ganetsky, who invented the patch, noted) the windows console is a mess.

It is indeed. Windows uses the ANSI code page for most GUI stuff but uses the OEM code page for the console (at least by default i belive there may be ways to change it). In MOST cases the ANSI and OEM code pages are significantly different!

{ansi|short}string <-> widestring conversion is not really possible to do right automatically. The compiler simply can't know what character set the {ansi|short}string is meant to represent! It defaults to ISO-8859-1 (aka direct code value mapping) which at least provides round trip compatibility but apps that care are strongly advised to either register thier own conversion functions or avoid automatic ansistring <-> widestring conversions.

> However (as Maxim Ganetsky, who invented the patch, noted) the > windows console is a mess.

pg> It is indeed. Windows uses the ANSI code page for most GUI stuff but uses pg> the OEM code page for the console (at least by default i belive there may be pg> ways to change it). In MOST cases the ANSI and OEM code pages are pg> significantly different!

And the patch used system functions to know, which codepage to use.

My message was just a report about uncovered features. ("crt garbles writeln() output" bug was higher priority: it concerned TP compatibility.)

pg> {ansi|short}string <-> widestring conversion is not really possible to do pg> right automatically. The compiler simply can't know what character set the pg> {ansi|short}string is meant to represent! It defaults to ISO-8859-1 (aka pg> direct code value mapping) which at least provides round trip compatibility pg> but apps that care are strongly advised to either register thier own pg> conversion functions or avoid automatic ansistring <-> widestring pg> conversions.

It's easy: (just like Maxim wrote)
function w2a(const widestring; var ansistring; target_encoding: integer or something): integer; --- for utf16
function w2a(const widestring; var ansistring; source_encoding: ...; target_encoding: ...): integer; --- for EBCDIC etc

(output for success/fail result and various info, like first unconvertible symbol, number of converted symbols etc; some errors may be reacted by exceptions: not enough length etc)

And, finally, typecast: widestring->ansistring, assuming utf16 and GetConsoleOutputCP() (or something else on other platforms/in non-console modes).

And similar procedures in the opposite direction.

It is matter of priorities: the mappings are long known. Unicode standard is quite open too. If I had much free time, I'd work on porting such a thing (from somewhere, there are many places) to FPC. But I'm not yet accustommed to my full-time job:(

oh yes writing the converters is easy, but what the compiler can't do is know what you meant to do when you write

widestringvar := ansistringvar;

because the ansistringvar could contain anything! it could be utf-8 from a network protocol, windows ansi from the windows gui, windows oem from the console etc.