Hi there,
 
In the discussion about resourcestrings I read that the RTL now uses codepage-aware strings for FileIO. So I wonder what kind of codepages do you use for FileIO?

The Windows-documentation calls Filenames "opaque sequence of WCHARs". https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247%28v=vs.85%29.aspx

So e.g. converting a Filename from the Windows-API to UTF-8 can be lossy. Does the new FPC-FileApi work correctly if a Filename contains invalid UTF-16 sequences? 

Assigning a codepage to something that basically is just some raw sequence of bytes from an external source sounds dangerous to me.

Andreas Dorn wrote

Hi there,   In the discussion about resourcestrings I read that the RTL now uses codepage-aware strings for FileIO. So I wonder what kind of codepages do you use for FileIO?

It uses UTF16 on windows, not a codepage aware string.

So if you use widestring for all your filename strings, there will be no problem. No conversions will happen.

If you so desire, you can start with a single-page, codepage aware string, but the RTL will convert it to UTF16 on windows anyway. if your codepage aware string contains "invalid" bytes, then the conversion may fail.

Michael.

Andreas Dorn wrote on Fri, 25 Sep 2015:

In the discussion about resourcestrings I read that the RTL now uses codepage-aware strings for FileIO. So I wonder what kind of codepages do you use for FileIO?

On Windows: UTF-16.

If you use the RTL file APIs with unicodestrings on Windows, then no conversions should occur because
a) we use the UTF-16 Windows APIs
b) all file name helpers are available both with unicodestring and rawbytestring parameters, so the unicodestring ones should be used.

If you use the RTL file APIs with an ansistring variant, then which code page is used is described at
http://wiki.freepascal.org/FPC_Unicode_support#Code_page_settings

Maybe we should add support for detecting invalid UTF-16 sequences in returned file names from Windows APIs, and if there are any ask for and return the "short/safe name" instead (file~1.txt and the like). For data that you pass in yourself, there is no problem (either you pass in UTF-16 and it will be passed on unmodified, or you use another code page and then it's your responsibility if it contains invalid data -- which pretty much only can happen with UTF-8, and possibly some single byte code pages that have undefined bytes, if there are any).

Assigning a codepage to something that basically is just some raw sequence of bytes from an external source sounds dangerous to me.

It is.

Jonas

Michael Van Canneyt wrote

So if you use widestring for all your filename strings, there will be no

For better performance probably UnicodeString not WideString.

Martin

Michael Van Canneyt wrote

It uses UTF16 on windows, not a codepage aware string. So if you use widestring for all your filename strings, there will be no problem. No conversions will happen.

 
If I understand that correctly, it stores the filename in a string that has been tagged as valid UTF-16.

Do I then have to be careful about any automagic conversions? Of course anything that assumes that a conversion of an UTF-16 tagged string to an UTF-8 one is lossless has to be avoided.

Is it safe to pass the Filename to procedures from the RTL without risking corruption?

For me this is more a general problem when dealing with external data. Should I tag raw external data as UTF-16/UTF-8 and be super-careful, or should I tag it as some kind of "raw" string (which one?) and handle any conversions manually.

For me Filenames are more a type that has some kind of affinity to an encoding for display, but I'd rather not tag its content as valid UTF-16
- any magic internal conversion is potentially lossy.

All in all I think if everything works for filenames, everything else will follow...

(Now lets better not start about the encoding of Filenames on non-Windows OS... :-))

Andreas Dorn wrote on Fri, 25 Sep 2015:

If I understand that correctly, it stores the filename in a string that has been tagged as valid UTF-16.

There are no tags for valid, invalid or unchecked UTF-16. A unicodestring is basically a sequence of widechars. Some operations, such as converting to a different string type,
uppercasing/lowercasing, case-insensitive comparison etc may however fail in case it's not valid UTF-16.

Do I then have to be careful about any automagic conversions?

As long as you don't assign it to another string type besides unicodestring and widestring (this includes passing it as a parameter to a routine expecting a non-unicode/widestring parameter): no.

Is it safe to pass the Filename to procedures from the RTL without risking corruption?

It depends which RTL procedures.

For me this is more a general problem when dealing with external data. Should I tag raw external data as UTF-16/UTF-8 and be super-careful, or should I tag it as some kind of "raw" string (which one?) and handle any conversions manually.

There is no "raw string" tag. There are only strings and arrays of bytes/words. There are no RTL file APIs accepting arrays of byte/word parameters.

(Now lets better not start about the encoding of Filenames on non-Windows OS... :-))

Everything about that for the supported platforms is explained at http://wiki.freepascal.org/FPC_Unicode_support#DefaultFileSystemCodePage

Jonas

In our previous episode, Andreas Dorn said:

Is it safe to pass the Filename to procedures from the RTL without risking corruption?

In theory no, but since mostly filenames will be passed to functions that have the same assumptions, I assume the only problem is if you put such a filename in a store that has problems with it.

That being said, converting such strings to other codepages is not a good idea.

The logical thing is to implement a simple validation function and call that in critical points and throw an error if the input is inconsistent.

For me this is more a general problem when dealing with external data.

Always assume external data needs to be validated.

Should I tag raw external data as UTF-16/UTF-8 and be super-careful, or should I tag it as some kind of "raw" string (which one?) and handle any conversions manually.

I think this is only needed in special cases (like embedded devices that sometimes spout garbage)

For me Filenames are more a type that has some kind of affinity to an encoding for display, but I'd rather not tag its content as valid UTF-16 - any magic internal conversion is potentially lossy.

Any passing to a subsystem that barfs on wrong surrogates causes trouble too (e.g. msxml or so). The golden rule is to be liberal in what you accept (and correct/reject appropiately) and exact in what you generate, and always validate input.

So if you are really worried about this, validate when strings read from the filesystem enter the your program (e.g. result of findfirst), before passing them on.

All in all I think if everything works for filenames, everything else will follow...

Well, at least you can recycle the validation routines you made. A quick look at the wikipedia utf16 article might give some clues.

(Now lets better not start about the encoding of Filenames on non-Windows OS... :-))

BSD/Linux afaik has the same problem. The filesystem is binary, not textual. The textual aspect is only interpretation.

[...] BSD/Linux afaik has the same problem. The filesystem is binary, not textual. The textual aspect is only interpretation.

Yes. I see invalid UTF-8 file names on Linux systems often (cannot be converted to UTF-16).

Are invalid UTF-16 file names under Windows really a practical problem?
What program allows to create invalid UTF-16 file names? With "invalid UTF-16" I mean not-convertable-to-UTF-8. I don't mean invalid code points.

Mattias

Marco van de Voort wrote

> > (Now lets better not start about the encoding of Filenames on non-Windows > > OS... :-)) BSD/Linux afaik has the same problem. The filesystem is binary, not textual. The textual aspect is only interpretation.

Under FreeBSD and Solaris, when using ZFS, file and directory names are stored in the locale's character set. With ZFS you can also force UTF-8 and even set the Normalization algorithm that must be used for storing the text.

http://docs.oracle.com/cd/E23824_01/html/E26033/glmio.html

Regards,
- Graeme -

In our previous episode, Mattias Gaertner said:

Not likely. I think it is more something for the security side of things than normal operation. (path traversal issues when not guarded by permissions)