How does the RTL support using unicode filenames (e.g. file names that cannot be represented by the ansi char set)?

For example the FileExists function takes a string which is encoded in the system char set. If the system char set is UTF8, like most linuxes and Mac OS X, then there is no problem. But on windows using a western european charset, I cannot check for existence of a file with cyrilic characters, even though I can enter them in the windows explorer and create such files.

Vincent

Vincent Snijders wrote

How does the RTL support using unicode filenames (e.g. file names that cannot be represented by the ansi char set)?

It doesn't.

Jonas

In MSEgui I use my own set of widestring based file utilities to overcome the problem. They are located in lib/common/msefileutils.pas and the systemspecific msesysintf.pas. The windows version currently converts the MSEgui widestring filenames to the system encoding before doing system calls, I plan to call the *W versions of the system routines instead if available (post version 1.8).

Martin

See also http://bugs.freepascal.org/view.php?id=7863

Jonas

How does the RTL support using unicode filenames (e.g. file names that cannot be represented by the ansi char set)?

As said on IRC, afaik decisions about this have been continuously postponed.

There are some problems:

- If the border condition that w9x must remain supported persists, this becomes a magnitude more work. A way to deal with this has to be found (two win32 FPC releeases, one advocated as D2..D2006 compat + w9x, one as NT + Unicode/Tiburon?)

- Also I have some doubts that using two different encodings is a good thing for a portable compiler, so a decision has to be made about that too (e.g. always support UTF-8 and allow overloading with utf16 if the platform defaults to it)

- Do we have a non com widestring on Windows?

- Tiburon compability. It doesn't make sense to roll our own slightly incompatible schema. At least we should have a look at it, before we decide if we support it (if it can folded in the multiplatform vision and it is sane from a native perspective)

What are the exact plans of Lazarus in this? Is there some wiki page with how Lazarus plans to tackle this with all multi-platform concerns?

Btw I saw that Cygwin is giving up 9x support in the next release.

Marco van de Voort wrote

What are the exact plans of Lazarus in this? Is there some wiki page with how Lazarus plans to tackle this with all multi-platform concerns?

All strings the LCL are UTF8, see
http://wiki.lazarus.freepascal.org/LCL_Unicode_Support

For windows this means all strings in the LCL are converted to ansi, if the OS is win9X, and to widestring, if the OS is NT or higher.

See in particular:
http://wiki.lazarus.freepascal.org/LCL_Unicode_Support#Dealing_with_directory_and_filenames

IMHO there should be a better solution than to convert file and directory to/from ansi. I think, this is the archiles heel of the Lazarus unicode support, that it depends on a RTL that has insufficient unicode support.

Vincent

Yes. And with the above question I meant more what you want long term, and the reasons that we can't see (widgetset related, db components related), not which hacks you employ now to workaround that :-)

If we want to make an informed decision, we have to put all requirements on the table, e.g.

- Base principle for me: requiring too much handcoding is not desirable. In an application (not RTL/FCL) I don't want to have to insert manual conversions for each string operation and/or passing. Some of this must be automated, it is the delphi way IMHO.

- Which encoding(s) to support (utf-8 and/or utf-16 mostly)
- The and/or in the question above, one primal encoding or two? If you have just one, you have to convert on some OS to access API+widgetset, and header translated for that OS must be redone with gluecode doing the transforms If you have two, each general purpose string routine must be doubly implemented. (or face conversion chaos)
- keep one windows release per windows target (win32,win64) or have two (ascii+w9x compat, unicode+NTonly) ?
This because if we significantly step unicode API use up, keeping runtime compatability with w9x will require a lot of hackish code. Split them up, and you just have a few unicode vs ansi includefiles, and way less glue code to make mistakes in.
- Do we (longterm) let UTF8 piggy back on ansistring, or do we have a distinct type for it, so that the compiler knows that type is utf-8 (and consequently that ansistring isn't) ? I've a feeling that this might be required, even if we decide on utf8 as universal encoding to avoid having to add too many checks and hand transformations.

2008/6/29 Vincent Snijders :

In fpGUI we use UTF-8 for everything. We have wrapper file access functions which replaces the RTL ones. The unit is called gfx_utils.pas

eg:

function fpgFileExists(const FileName: TfpgString): Boolean;
begin
  Result := FileExists(fpgToOSEncoding(FileName));
end;

fpgToOSEncoding() is then implemented in platform dependent include files (like FPC also does with many functions).

Linux & *BSD X11:
---------------------------
// yes we assume UTF-8. Only very old Linux versions don't use UTF-8,
but that is very rare now.
function fpgToOSEncoding(aString: TfpgString): string;
begin
  Result := aString;
end;
Windows GDI:
-------
function fpgToOSEncoding(aString: TfpgString): string;
begin
  Result := Utf8ToAnsi(aString);
end;

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

I see you are crippled in the same way as the LCL, because you only can handle ansi filenames correctly.

Vincent

Marco van de Voort wrote

- Which encoding(s) to support (utf-8 and/or utf-16 mostly)

In order to complement Graemes mail, MSEgui uses widestrings for everything.

Martin

2008/6/29 Vincent Snijders :

I see you are crippled in the same way as the LCL, because you only can handle ansi filenames correctly.

fpGUI was tested under Windows with Russian locale and filenames. Not tested by me, but my a co-developer (Vladimir). He reported that the file dialogs and other file related functions worked correctly. He actually implemented the locale file handing.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Marco van de Voort wrote

- If the border condition that w9x must remain supported persists, this becomes a magnitude more work. A way to deal with this has to be found (two win32 FPC releeases, one advocated as D2..D2006 compat + w9x, one as NT + Unicode/Tiburon?)

procedure AnyFileRoutineInWin32(AFileName: widestring);
begin
  if UnicodeEnabledOS then SomeWin32APIW()
  else AnsiToWideString(SomeWin32ApiA())
end;

Not very hard to keep 9x support.

Yes, it works ok, if all the characters used are part of the system locale, so for Russian locale, it contains the ascii characters and the cyrilic characters.
Try using a path, that contains characters from more than codepage, for example using cyrillic characters in a file name and french accented chars in the directory. That is when you need real unicode support.

Vincent

Graeme Geldenhuys wrote

fpGUI was tested under Windows with Russian locale and filenames. Not tested by me, but my a co-developer (Vladimir). He reported that the file dialogs and other file related functions worked correctly. He actually implemented the locale file handing.

What if you have a russian directory in a Windows with western latin locale?

What if you have a chinese directory in a Linux with non-utf-8 locale?

We also support a russian directory in a russian windows the problem only appears if the locale cannot represent the characters in the file.

If you want even more details you can initialize UnicodeEnabledOS by reading the operating system version and the operating system type NT/9x very easely.

If you create a file with a russian name on your hard disk, can you use fpgFileExists to check for its existence?

Vincent

2008/6/29 Vincent Snijders :

If you create a file with a russian name on your hard disk, can you use fpgFileExists to check for its existence?

My Windows doesn't contain a Russian locale, but I tried it with French, German etc names and it works. I asked Vladimir to create a zip file containing English and Russian directory and file names. I'll unzip that and give it another try as soon as he emails the file. I'll let you know what happens. :)

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

2008/6/29 Vincent Snijders :

If you create a file with a russian name on your hard disk, can you use fpgFileExists to check for its existence?

I don't know how to change the Windows language to Russian, but I do under Linux. So I did the following. I changed my Linux systems locale for an application to Russian. I had Russian translations of fpGUI Toolkit so used those. I copied the rsCancel resourcestring value (in Russian) to a Edit component. Copied that to clipboard, used the File Open dialog to 'create directory' and pasted the Russian word for Cancel in their. Now I had a Russian directory on the hard drive. I quit the program, changed back to English locale. Loaded the program, and it displayed the Russian directory correctly.

PS:
Even Linux's terminal didn't display the russian directory correcty. A whole bunch of '?????????' instead. fpGUI worked fine! ;-)

See attached screenshot. Application using English locale (en_ZA.UTF-8) as can be seen by the grid headers and displaying a Russian directory name.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

2008/6/29 Vincent Snijders :

If you create a file with a russian name on your hard disk, can you use fpgFileExists to check for its existence?

I did a test and sent a screenshot of the results. I don't know what's the limit of attachments in this mailing list. So let me know if the attachment didn't go through. The file was 22kb in size.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Graeme Geldenhuys wrote

I did a test and sent a screenshot of the results. I don't know what's the limit of attachments in this mailing list. So let me know if the attachment didn't go through. The file was 22kb in size.

Nothing arived here. Can't you just say if it worked or not?

I suspect that this may be difficult to assess for Graeme if he doesn't know Russian. ;-)

As a side note related to Graeme's e-mail, French and German should be obviously supported properly if using the "western" charsets, so no wonder that this worked.

Finally, please, note that only the latest Info-zip beta (!) versions of zip and unzip (specifically, zip 3.0 and unzip 6.0 betas) support Unicode paths, so zip archives may not be the best candidates for this test. Other zip and unzip implementations may or may not already support Unicode paths even in released versions, but Info-zip implementation is certainly the most common one on many platforms (all Unix/Linux for sure). I'm not sure about situation with other archive formats (I believe that Unicode is supposed to be supported with RAR, although I have no idea whether that applies to e.g. the Linux version of unrar too).

Tomas

Even if it doesn't contain Russian locale, you would be able to create such files in the windows explorer to create such file, for example by copy / pasting the file name while renaming it.

Then let your fpGui program check for its existence.

Vincent

2008/6/30 Felipe Monteiro de Carvalho :

Nothing arived here. Can't you just say if it worked or not?

Crap! See the email below, I removed the screenshot for now. Under Linux, I could see and query the directory and file, while my system as setup for English. Again, no idea how to do this test under Windows without Russian locale support.

---------- Forwarded message ----------

2008/6/30 Vincent Snijders :

Under Linux I simply change the LANG environment variable and place the fpgui_ru.po file in the directory, and it uses the Russian language. I use GetText like done in Lazarus. How to I change the language locale in Windows, so I can use the Russian translaction of fpGUI?

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Can you test to create a chinese filename in the explorer. For example:

北方话.txt

Then check if the file exists. This should work independent of the locale.
With Utf8ToAnsi this can not work on 1-byte locales.

Mattias

How did you change to russian locale?

See attached screenshot. Application using English locale (en_ZA.UTF-8) as can be seen by the grid headers and displaying a Russian directory name.

Under Linux you use Result := aString; - no conversion. Still UTF-8.

Linux is not the problem here. Windows is. You must use the W functions.

Mattias

2008/6/30 Mattias Gaertner :

> Even Linux's terminal didn't display the russian directory correcty. > A whole bunch of '?????????' instead. fpGUI worked fine! ;-) How did you change to russian locale?

By setting the LANG environment variable.
export LANG=ru.UTF-8

Linux is not the problem here. Windows is. You must use the W functions.

We do use the W functions in the GDI backend. As show in Felipe's post.

  if UnicodeEnabledOS then
    ...   // W functions
  else
    ....  // plain functions

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

But for filenames in fpgFileExists?

Vincent

2008/6/30 Mattias Gaertner :

OK, I'll copy and paste that filename from the email into explorer. Then see what happens. I'll reply with the results.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

2008/6/30 Mattias Gaertner :

Can you test to create a chinese filename in the explorer. For example: 北方话.txt

I can't create that file in Windows Explorer. I'm using Windows 2000 in a VMWare session. I opened explorer. Then copied the filename above to the clipboard from Linux (where Firefox and GMail runs). The then go to Window VMWare session. Right Click and create now text file. Past the chinese name, which then seems to escape the name. When I press enter to confirm the name, Windows gives me an error.

See the screenshot.
http://opensoft.homeip.net/~graemeg/test.png

PS:
I hope I got the Apache access permisions correct to my home folder. Let me know if there is a problem accessing the image, then I'll move it somewhere else on the web.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

2008/6/30 Vincent Snijders :

But for filenames in fpgFileExists?

Sorry, yes the previous code was for things like painting text.

Here is the code for fpgFileExists

// platform indepenent
function fpgFileExists(const FileName: TfpgString): Boolean;
begin
  Result := FileExists(fpgToOSEncoding(FileName));
end;
// GDI dependent
function fpgToOSEncoding(aString: TfpgString): string;
begin
  Result := Utf8ToAnsi(aString);
end;
function fpgFromOSEncoding(aString: string): TfpgString;
begin
  Result := AnsiToUtf8(aString);
end;

No idea if this is enough for all cases (I'm not a unicode guru), but Vladimir (russian developer) reported that it works for him.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Zitat von Graeme Geldenhuys :

ok

Not ok.
UTF8ToAnsi converts to the current windows 8bit code page. Each code page only supports a few languages, but not all. See:

http://www.microsoft.com/globaldev/reference/WinCP.mspx

As you can see, the 1250 is for whole central europe, so french, spanish, german, english - they all use the same codepage.
Russian needs another.
And afaik chinese is converted to DBCS. (I did not try)

function fpgFromOSEncoding(aString: string): TfpgString; begin Result := AnsiToUtf8(aString); end;

ok.
I wonder, what windows gives for characters not in the current code page.

No idea if this is enough for all cases (I'm not a unicode guru), but Vladimir (russian developer) reported that it works for him.

Of course it works for him - MS keeps things going. As long as a user stays in his code page it works.

Mattias

2008/6/30 Mattias Gärtner :

Umm, no idea really.

Ah, now I get it!! I'm a bit slow today. :-) Using characters outside of the current code page range needs special attention. Thanks for your explanation. I now understand the problem.

Makes me wonder how CodeGear is going to handle something like this. Have they released more information on their Unicode support?

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

They actually use unicode support, so NT only -W functions.

I cannot create the chinese filename either, but I could create a russian filename: пишет.txt

UTF8ToAnsi converts the russian characters to ?.

Vincent

Zitat von Graeme Geldenhuys :

That's not a valid lang value.

export LANG=ru_RU.UTF-8

But the important part is the UTF-8. If you don't change this, then the terminal will still use UTF-8 and the fonts will not change. If it shows '??' then the filename is not a valid UTF-8 string or your terminal shows unsupported characters as '??'. For example xterm shows rectangles for valid but unsupported characters.

And the same must be done for filenames.

Mattias

Zitat von Graeme Geldenhuys :

Maybe that's the way 'explorer' handles unsupported characters in the font. Windows has still to go a long way for real unicode.

Ok. Thanks for trying out. It's not that important. I think it's clear now, that windows need the W functions.

PS: I hope I got the Apache access permisions correct to my home folder. Let me know if there is a problem accessing the image, then I'll move it somewhere else on the web.

Works here.

Mattias

Zitat von Marco van de Voort :

Do you know what happen with their plans of creating a new string type (a reference counted 2-byte string) and changing all String and Char to the 2-byte types? Is this only for .net or for the win32 too?

Mattias

It will be released in August.

Is this only for .net or for the win32 too?

Both. (and afaik .NET already is), but they don't have an unix to keep track of, so we can't simply follow them blindly.

Zitat von Marco van de Voort :

If they are bold enough for this big incompatibility, then maybe we must be braver too.

> Is this only for .net or for the win32 too? Both. (and afaik .NET already is), but they don't have an unix to keep track of, so we can't simply follow them blindly.

True.
I guess, FPC will support the new string type and create some compiler directives/flags to control the default string/char type, won't it?

And it will use the widestring manager to auto convert these types, won't it?

So FPC will be able compile the new Delphi code, although the code will run slower and there will conversion errors on non ansi characters.

Mattias

What, also stop supporting anything but Windows?

No decision yet. It is not really easy.

So FPC will be able compile the new Delphi code, although the code will run slower and there will conversion errors on non ansi characters.

No commitment to remain Delphi compat either. Until we 1) know exactly what Delphi does, 2) if we can unify that with a workable multi platform vision.

As you said, it is delphi that breaks in a major way. Doesn't mean we'll follow that.

Given that most polls I've seen indicate that most people still use D7, I don't think we should be too eager to jump into the void ourselves...

Of course a decision will be forced on us anyway in the long run :-)

Michael.

Zitat von Marco van de Voort :

;)

I meant, break some compatibility too.

At the moment the RTL uses Ansi = current code page < unicode. And so any filename that uses indirectly these functions is limited to the current code page.

One solution is to create a new string type UTF8String. Similar to AnsiString/WideString auto conversions, there would be an auto conversion AnsiString/UTF8String, IFF there is an UTF-8 string manager. The RTL and FCL can use it for filenames and similar things. There should be some compiler flags to set the default string type to either ansistring or utf8string.
All lazarus related sources would set the default string type to utf8 and would load an UTF-8 string manager.

Drawbacks:
- If a UTF8 string manager is enabled, old code that uses ansistring would double recode filenames under windows. From ansistring to UTF8 to widestring. I guess in case of filenames this is hardly measurable.

- Mixing code compiled with ansistring and compiled with UTF8String will be slower due to the conversions. Especially when exchanging big lists of string like TStringList.Assign. Remedy: enable UTF8String and fix the few issues. This might be arbitrary difficult and/or unpleasant. But hey, CodeGear says that even switching from ansistring to widestring is easy. And of course you can still use 'ansistring' explicitly.

- Mixing code compiled with ansistring and compiled with UTF8String with dirty typecasts will create hidden conversion errors. Remedy: Same as above.

- If the most common RTL/FCL functions are *not* converted to UTF8String then all UTF8String programs will become very slow and might even get conversion errors, due to typecasts. So, if RTL/FCL don't use it, no one will.

Mattias

on XP it is possible. Initially it shows only square boxes since the font used in the explorer doesn't support chinese. But if I change the font (=all tahoma) to a code2000 font, it shows 北方话.txt

However creating a file ßΩ≈≠≤≥.txt or a folder ßΩ≈≠≤≥ is possible. Not all tools can read it. Afais, using A functions this translates to ß?|?==.txt

Yep

Marc