Is it (already) possible to use Unicodestrings for file paths? I was able to access long paths (with many hundreds of characters) in the windows API functions FindFirstFileW (with Win32_Find_DataW data type) but I failed to open such files with Assign(File)/reset. Are they converted to (one byte) Ansistring internally? Am I forced to use CreateFile or similar function to open such files?

Jürgen Hestermann wrote

Is it (already) possible to use Unicodestrings for file paths?

Only if your current ansi code page supports the characters used in the unicodestring. There are not unicodestring overloads yet in the RTL.

Jonas

In our previous episode, Jonas Maebe said:

> Is it (already) possible to use Unicodestrings for file paths? Only if your current ansi code page supports the characters used in the unicodestring. There are not unicodestring overloads yet in the RTL.

I think Joerg means something else. The Unicode winapi functions on Windows have support for longer paths.

He said:

I was able to access long paths (with many hundreds of characters) in the windows API functions FindFirstFileW (with Win32_Find_DataW data type) but I failed to open such files with Assign(File)/reset. Are they converted to (one byte) Ansistring internally?

So as far as I can see, he was in fact asking whether the regular RTL functions (such as assign/reset) already included unicodestring support. He already knew that it works fine if you use the Windows APIs directly.

Jonas

functions (such as assign/reset)

already included unicodestring support. He already knew that it works

fine if you use the Windows APIs directly.

Correct. Accessing long paths under Windows can only be done when using Unicode (widestring) paths (prepended with '\\?\') for Windows API functions that end with W.

It is easy to replace the RTL FindFirst/Next functions by the Windows API functions FindFirst(Next)FileW but when I want to open a found file I am not able to use Assign/Reset to open them (i.e. text files). I am just wondering how it will still be possible to open such long paths with the Windows API function CreateFile and use Readln and other file functions afterwards. Is it possible to somehow fill the file data structure with vaild data other than using Reset/Rewrite? I can get a file handle with CreateFile but when using readln I have to hand over a text file data structure...

Hi,

According to what I have read (and I always wonder how up to date documentation is with respect to the state of the compiler and libraries), all standard library functions (Which would include system.assign) are still using the "local" encoding. For Mac OS X, luckily, this means UTF8.

Thus, the only question would be whether you have to manually convert a UnicodeString to a UTF8String or not.

If you don't trust it, you can build an overload version of assign to convert the string and call the old one, but note that in Windows the local encoding will be different than UTF8 in many cases (hence the wide version of the calls will be better), and in Linux it may be UTF-8 or something else) - so you can't always convert UTF16 to UTF8, call the old functions, and expect it to work on all OSs.

Still, fixing Assign/Reset/Rewrite would be better than using Unix calls, because you could submit a patch to save everyone else trouble in the future.

This kind of stuff is a real pain all the time, and not even that hard to fix, I am not sure why the "issues" remain - but perhaps there aren't enough volunteers with enough time to work on it.

I have done a lot of Unicode work, so if you have other problems let me know, I may already have written code to handle whatever the issue is, and would be happy to share it.

Thank you,
Noah Silva

2012/5/10 Jürgen Hestermann

Thus, the only question would be whether you have to manually convert

a UnicodeString to a UTF8String or not.

No, that would not help. Under Windows you can access long paths *only* when using special functions (i.e. FindFirstFileW instead of FindFirstFileA). And these functions need unicode (widechar) paths strings with a prepended '\\?\'. There is no other way to access long paths under Windows. Simply converting Unicode to ANSI may let me use most (but not all) Unicode characters but the paths would always be restricted to MaxPath (about 260) characters. Only when using API functions ending with a W I can access longer paths (up to 32000 characters) and these string *must* be Unicode (widechar). So the RTL needs to be rewritten to allow Unicode paths handed over without conversion and then use Windows API functions with the W at the end.

Because this is not the case (yet), I was wondering, whether it is possible to use other Windows API functions like CreateFileW and then somehow connect them to RTL file data structures so that I can replace "assign" and "reset/rewrite" by Windows API functions and then be able to use "readln" on such (text) files.

Still, fixing Assign/Reset/Rewrite would be better than using Unix

calls, because you could submit a patch to save everyone else trouble in the future.

Yes, I would like to but neither time nor knowledge allows me to do that currently.

Hi,

Jürgen Hestermann wrote

Noah Silva schrieb: > Thus, the only question would be whether you have to manually convert a UnicodeString to a UTF8String or not. No, that would not help. Under Windows you can access long paths *only* when using special functions (i.e. FindFirstFileW instead of FindFirstFileA). And these functions need unicode (widechar) paths strings with a prepended '\\?\'. There is no other way to access long paths under Windows. Simply converting Unicode to ANSI may let me use most (but not all) Unicode characters but the paths would always be restricted to MaxPath (about 260) characters. Only when using API functions ending with a W I can access longer paths (up to 32000 characters) and these

Ah, I forgot about that little detail. Given that, it doesn't really make sense to use the *A versions at all in Windows (Unless you want to support very old Windows versions). Thus the approach for Windows should be the opposite of Unix, I suppose. UTF8 or ANSI gets converted to UTF16 and call the *W functions.

string *must* be Unicode (widechar). So the RTL needs to be rewritten to allow Unicode paths handed over without conversion and then use Windows API functions with the W at the end.

I'm surprised this isn't available already. The 260 char path limitation is pretty severe these days.

Because this is not the case (yet), I was wondering, whether it is possible to use other Windows API functions like CreateFileW and then somehow connect them to RTL file data structures so that I can replace "assign" and "reset/rewrite" by Windows API functions and then be able to use "readln" on such (text) files.

I'm sure it's possible somehow since those structures probably track the Windows file handle, but it wouldn't be very portable or future-proof.

> Still, fixing Assign/Reset/Rewrite would be better than using Unix calls, because you could submit a patch to save everyone else trouble in the future. Yes, I would like to but neither time nor knowledge allows me to do that currently.

I have the same time problem, but I will take a look at the source code, I don't think it should be very difficult to figure out.

Thank you,
Noah Silva

_______________________________________________ fpc-pascal maillist - http://lists.freepascal.org/mailman/listinfo/fpc-pascal

does TFileStream use the same code as assign ?

you could create a TFileStream instance and then use AssignStream to a regular file type and keep on using plain old read/write...

not tested here, but could be worth a try...

2012/5/12 印場 乃亜 :

Hello Jurgen,

I'm assuming that everyone is aware of the solution for this. Every unicode file name still has ANSI-dos name (aka ShortName). You could write a function that would return a short name based on unicode name of a file. And pass the short name to Assign File function.

thanks,
Dmitry

Hi Dmitry,

Do you mean the ShortName as the 8.3 name defined on this page: http://msdn.microsoft.com/en-us/library/windows/desktop/aa365247%28v=vs.85%29.aspx#file_and_directory_names ... it states
"When you create a long file name, Windows may also create a short 8.3 form of the name, called the 8.3 alias, and store it on disk also." So it isn't compulsory to store the 8.3 filename - which is confirmed later on by:
"8.3 aliasing cannot be disabled for specified volumes until Windows 7 and Windows Server 2008 R2."... implying it can be disabled in Win7 and 2K8R2

Looking at the GetShortPathName function
http://msdn.microsoft.com/en-us/library/windows/desktop/aa364989%28v=vs.85%29.aspx ... the text itself only says the function can fail - not when it is likely to failx.
Comment below does shed more light:
"GetShortPathName fails to create a short filename if the file doesn't have a short file name."..."You can turn this option off by using the "System Policy Editor" (Poledit.exe). Certain file systems also don't support creation of short names by default"

Or are you talking about a different way of getting a short name?

Thanks,
Reinier

That's the one I'm talking about.

thanks,
Dmitry

Currently the Windows RTL uses (nearly?) always the Ansi functions (exception might be WinCE as there the Ansi functions don't exist), so using TFileStream won't help here.

Regards,
Sven

2012/5/13 Sven Barth :

Currently the Windows RTL uses (nearly?) always the Ansi functions (exception might be WinCE as there the Ansi functions don't exist), so using TFileStream won't help here.

Does AssignStream() also accept any other THandleStream? Maybe you can just make your own TFileStream descendant with a slightly different constructor that is using the widestring file open API? Maybe you could then use such a customized stream with AssignStream()? (Untested)

Bernd

Given that, it doesn't really make sense to use the *A versions at all in Windows (Unless you want to support very old Windows versions). Thus the approach for Windows should be the opposite of Unix, I suppose. UTF8 or ANSI gets converted to UTF16 and call the *W functions.

That's true. In general the data type with the most information (UnicodeString
in this case) should be used as much as possible and only when functions require other (lower) data types a conversion should be done. Otherwise Information gets lost.

I have the same time problem, but I will take a look at the source code, I don't think it should be very difficult to figure out.

I had a look at the sources and found the following (it's a bit longer, so if you are not interested just stop reading ;-)):

1.) The 2 fundamental file types filerec and textrec (for standard files and text files) are defined in
fpc\fpc\rtl\inc\filerec.inc and
fpc\fpc\rtl\inc\textrec.inc
Both differ only by some buffer structures at the end but share the first part
(including file name). The file name is of type

array[0..255] of char;

and needs to be replaced by UnicodeString.
This requires other changes:

1.) The order of elements in the file data structures have to be changed.
The name has to become the first element as in:
-----------------------------------------------------------
      TextRec = Packed Record
                name      : UnicodeString;
                Handle    : THandle; // Handle has to be directly after
name, see Assign!
                Mode      : longint;
                bufsize   : SizeInt;
                _private  : SizeInt;
                bufpos,
                bufend    : SizeInt;
                bufptr    : ^textbuf;
                openfunc,
                inoutfunc,
                flushfunc,
                closefunc : pointer;
                UserData  : array[1..32] of byte;
                LineEnd   : TLineEndStr;
                buffer    : textbuf;
                {$ifdef FPC_HAS_CPSTRING}
                CodePage  : TSystemCodePage;
                {$endif}
                End;
-----------------------------------------------------------
Reason: See next item 2.)

2.) The Assign routine in \fpc\rtl\inc\file.inc overwrites the whole file data structure with zeros (using fillchar). The name has to be omitted from this action because it is now managed automatically. So Assign needs to be changed to:

----------------------------------------------------------- Procedure Assign(out f:File;const NewName:string); { Assign Name to file f so it can be used with the file routines }

Begin // Assign
FillChar(FileRec(f).Handle,SizeOf(FileRec)-Sizeof(FileRec(f).name),0);
FileRec(f).Handle := UnusedHandle;
FileRec(f).mode   := fmClosed;
FileRec(f).Name   := NewName;
End; // Assign
-----------------------------------------------------------

3.) The routine do_open in \fpc\rtl\win\sysfile.inc has to be changed at 3 lines to (see "// THIS LINE CHANGED!") ----------------------------------------------------------- procedure do_open(var f; p:UnicodeString; flags:longint); // THIS LINE CHANGED!
{ filerec and textrec have both handle and mode as the first items so they could use the same routine for opening/creating. when (flags and $100) the file will be append
when (flags and $1000) the file will be truncate/rewritten when (flags and $10000) there is no check for close (needed for textfiles) }
Const file_Share_Read = $00000001;

      file_Share_Write  = $00000002;
      file_Share_Delete = $00000004;

Var shflags,

      oflags,cd : longint;
      security  : TSecurityAttributes;

begin // do_open
DoDirSeparators(p);
{ close first if opened }
if ((flags and $10000)=0) then
   begin
   case filerec(f).mode of
      fminput,fmoutput,fminout :
         Do_Close(filerec(f).handle);
      fmclosed : ;
   else  {not assigned}
         inoutres:=102;
         exit;
   end; // of case
 end;
{ reset file handle }
filerec(f).handle := UnusedHandle;
{ convert filesharing }
shflags := 0;
if ((filemode and fmshareExclusive)=fmshareExclusive) then { no sharing }
else
   if (filemode=fmShareCompat) or ((filemode and
fmshareDenyWrite)=fmshareDenyWrite) then
      shflags := file_Share_Read
   else
      if ((filemode and fmshareDenyRead)=fmshareDenyRead) then
         shflags := file_Share_Write
      else
         if ((filemode and fmshareDenyNone)=fmshareDenyNone) then
            shflags := {$ifdef WINCE}
                       { WinCE does not know file_share_delete }
                       file_Share_Read or file_Share_Write;
                       {$else WINCE}
                       fmShareDenyNoneFlags;
                       {$endif WINCE}
{ convert filemode to filerec modes }
case (flags and 3) of
   0 : begin
       filerec(f).mode := fminput;
       oflags          := longint(GENERIC_READ);
       end;
   1 : begin
       filerec(f).mode := fmoutput;
       oflags          := longint(GENERIC_WRITE);
       end;
   2 : begin
       filerec(f).mode := fminout;
       oflags          := longint(GENERIC_WRITE or GENERIC_READ);
       end;
   end; // of case
{ create it ? }
if (flags and $1000)0 then
   cd:=CREATE_ALWAYS
{ or Append/Open ? }
else
   cd:=OPEN_EXISTING;
{ empty name is special }
if p='' then                           // THIS LINE CHANGED!
   begin
   case FileRec(f).mode of
      fminput :
         FileRec(f).Handle := StdInputHandle;
      fminout, { this is set by rewrite }
      fmoutput :
         FileRec(f).Handle := StdOutputHandle;
      fmappend :
         begin
         FileRec(f).Handle := StdOutputHandle;
         FileRec(f).mode   := fmoutput; {fool fmappend}
         end;
      end; // of case
   exit;
   end;
security.nLength              := Sizeof(TSecurityAttributes);
security.bInheritHandle       := true;
security.lpSecurityDescriptor := nil;
filerec(f).handle :=
CreateFileW(pwidechar(p),oflags,shflags,@security,cd,FILE_ATTRIBUTE_NORMAL,0);
// THIS LINE CHANGED!
{ append mode }
if ((flags and $100)0) and
   (filerec(f).handle0) and
   (filerec(f).handleUnusedHandle) then
   begin
   do_seekend(filerec(f).handle);
   filerec(f).mode:=fmoutput; {fool fmappend}
   end;
{ get errors }
{ handle -1 is returned sometimes !! (PM) }
if (filerec(f).handle=0) or (filerec(f).handle=UnusedHandle) then
   begin
   errno:=GetLastError;
   Errno2InoutRes;
   end;
end; // do_open
-----------------------------------------------------------

3.) Of course, there are lots of places where the file data structures filerec and textrec are used in the code. But in most cases only non-changed record elements are used (so no changes are required) and only seldomly the name is used/changed or the whole data structure is modified. So I think the above modifications require to check a lot of code but only a few modifications (hopefully). I would be willing to do these changes but I am not sure whether I have overlooked something obvious. These are very fundamental changes which affect all operating systems so it may have unexpected side affects.
I also need to get familiar with SVN.

Any comments?

While I can not comment on the changes you listed (though they sound more or less valid) I can comment on those you didn't list: If you change do_open to support Unicode you should do this for the other functions (e.g. write, read) as well so that only the *W functions are used for them. Also you need to keep in mind that do_open is used by platform independant code so you would need to change the declarations/handling for all other platforms as well. Also you need to change the declaration of Assign as this expects a Short-/AnsiString.

Also I don't know whether the Unix users would be happy if we'd push UnicodeString upon them or whether they'd prefer AnsiString(CP_UTF8).

In the end maybe it should be AnsiString(CP_UTF8) for Unix systems and UnicodeString for Windows systems and AnsiString(CP_SCP) (the current string) for the other systems, so especially the platform independant code needs to be adjusted to support all three cases without problems (which might not need that much work in the end...).

I hope other core developers comment on this as well, as I'm not so firm regarding the consequences of such changes...

Regards,
Sven

In our previous episode, Sven Barth said:

While I can not comment on the changes you listed (though they sound more or less valid)

I'm not sure if the *rec records can handle pointers, one probably can't use reference types in them, because the standard filedescriptions are threadvars.

Think what happens if those are simply "cloned".

Probably they will be rawbytestring, so both can be passed to it, and the per platform code must then change it to whatever encoding (ansi,utf8,utf16) the API expects.

I'm not sure that is true even without the above remark.

I would say for *nix it just remains "ansistring", the default 1-byte encoding. This fixes the case where e.g. embedded Linux doesn't expect utf8, which could otherwise have undesirable effects.

Note that the default encoding can be the same as utf8. (and probably will be on desktop unix)

I hope other core developers comment on this as well, as I'm not so firm regarding the consequences of such changes...

There are some interesting discussions on various lists you might want to read.... :-)

If you change do_open to support Unicode you should do this for the

other functions (e.g. write,

read) as well so that only the *W functions are used for them.

I think read/write only work on the file handle and don't use the file name. So no changes are needed here (I think).
But of course there are other functions that need changing, for example

Assign
Reset
Rewrite
FindFirst(File)
Rename
some others.

Also you need to keep in mind that do_open is used by platform

independant code

so you would need to change the declarations/handling for all other

platforms as well.

Yes. It can be a can of worms. ;-)
Although, as said: Only very few functions realy act on the file name. Most use the file handle only (which will not change).

Also you need to change the declaration of Assign as this expects a

Short-/AnsiString.

I think it should be declared as "UnicodeString" or "String" so that other string types that
are handed over are converted automatically.

Also I don't know whether the Unix users would be happy if we'd push

UnicodeString upon them or whether they'd prefer AnsiString(CP_UTF8).

Yes, this is a general question:
What kind of string type should be used within Free Pascal mainly? I am not sure how this is all affected by
http://wiki.freepascal.org/FPC_Unicode_support#Roadmap_of_RTL_Unicode_support_with_UnicodeString

In the end maybe it should be AnsiString(CP_UTF8) for Unix systems

and UnicodeString

for Windows systems and AnsiString(CP_SCP) (the current string) for

the other systems,

so especially the platform independant code needs to be adjusted to

support all three

cases without problems (which might not need that much work in the

end...).

As far as I know currently all platforms share the same filerec/textrec definitions (which is
an array of bytes now). If that is changed then it would be changed for all platforms.
But it can be changed to generic string type too. I am not so sure what this actually
means when the type differs from one platform to the other. For example, I have to use a "pointer
to widestring" for CreateFileW for Windows. So I need to convert to a null terminated widestring
from the existing string type and this conversion must work under all circumstances.

On Sat, May 19, 2012 13:12, JĂźrgen Hestermann wrote: .
.

I don't know what's the order used by Delphi (assuming that file and text as basic Pascal constructs are still supported there), but it's certainly better not to change the order of the attributes at the beginning; changing the call to FillChar is no problem. Also, it may be better keeping the current field for 8-bit name and add the UnicodeString as a new field instead in order not to require encoding between 8-bit charsets and UnicodeString on platforms not having UnicodeString support.

Tomas

I'm not sure if the *rec records can handle pointers, one probably

can't use

reference types in them, because the standard filedescriptions are threadvars. Think what happens if those are simply "cloned".

Of course, all "cloning" needs to be changed too so that the name pointer is not cloned (but assigned in a standard way).

The alternative would be to reserve a 64k buffer for the file name in the *rec structures. I am not sure whether this is intended and useful. And in 10 or 20 years this limit is breached again and another change is necessary. Double work IMO.

Why shouldn't reference types work in threadvard records? As long as the reference count mechanism does it's work correctly it should work.

In the end there will be only one way to find out: try it (even if it's only locally).

Right... forgot about that one.

I know these discussions, but somehow the results/decisions (if there were any) got lost in the noise...

Regards,
Sven

Wait... now I understand what you meant with "cloning"... this might indeed be a problem, I haven't tested something like that yet...

Regards,
Sven

Regarding the filename, yes, but while we're at it we could also change functions like WriteFile to WriteFileW (etc.)...

I don't know how up to date and valid that page is... the latest changes in 2012 were only cosmetically and the ones before that were in 2009 way before the merge of cpstrnew.

Well... you can define a "RTLString" type which is an alias to another string type depending on the platform and which is then used in the records. Or, as Marco suggested, we use RawByteString for this.

Regards,
Sven

I just noticed: FillChar will be no problem here. It sets the name field (which is an implicit pointer) to Nil which is equal to "empty string". So no need to change the FillChar calls for FileRec (the only exception might be if you FillChar an existing file passed by a "var", but then a "finalize(f)" might be sufficient before calling FillChar).

Regards,
Sven

In our previous episode, Sven Barth said:

Afaik because the ref count mechanism doesn't. The cloning process might be done by the OS.

So you have a record with that points to an ansistring with ref count 1, and then you create twenty threads, and you have 20 records that point to an ansistring with refcount 1.

For the same reason, the content of the buffer is duplicated.

If e.g. you have 'blabla' in a threadvar filerec buffer, create a few threads and flush them, you will have several times 'blabla' written

But Pierre and Sergei have done so much on TLS, that I don't know the current situation.

This is the part where there is still consensus. It gets harder when you get to classes, or global variables or functions that write/return strings.

After writing the mail you answered to this occured to me as well which is why I wrote another mail directly after that ;)

But nevertheless thank you for explaining.

Regards,
Sven

Threadvars are never cloned. They are always initialized with 0 for new threads. We have special hooks that are called initialize the default system threadvars whenever a new thread is started.

Jonas

> Although, as said: Only very few functions realy act on the file name. > Most use the file handle only (which will not change). Regarding the filename, yes, but while we're at it we could also

change functions like WriteFile to WriteFileW (etc.)...

There is no WriteFileW. Only Windows API functions that act on file *names* have a W version.

You are right. Then forget that ;)

Regards,
Sven

In "fpc\rtl\inc\file.inc" (and fpc\fpc\rtl\inc\text.inc) the declaration of "Assign"
uses the generic "String" type for the file name. I think this can be AnsiString too, so it may be longer than 255 characters. But the assignment to the internal *rec element "name" (which is "array[0..255] of char") is done with

Move(s[1],TextRec(t).Name,Length(s));

That would be a bug IMO when S is longer than 255 characters. "Move" does no range check.

No, unit System (which uses these include files) is never compiled with $H+.

Tomas

Ok. If that can be guaranteed.
But I am always wondering why in such situations the type is not set to ShortString.
Why use generic String type if all code relies on the fact that it may never change
from ShortString to some other string type? It's just asking for trouble and also
makes the code less clear.

$H+.

Ok. If that can be guaranteed. But I am always wondering why in such situations the type is not set to

ShortString.

Why use generic String type if all code relies on the fact that it may

never change

from ShortString to some other string type? It's just asking for trouble

and also

makes the code less clear.

Estimated guess: the code is from a time where "String = AnsiString" wasn't possible.

Regards,
Sven

That would be explanation.
But then it must be *very* old code which shows that a revise is overdue.

Jürgen Hestermann wrote on Tue, 22 May 2012:

Sven Barth schrieb: > > Estimated guess: the code is from a time where "String = > AnsiString" wasn't possible.

That's indeed probably true.

That would be explanation. But then it must be *very* old code which shows that a revise is overdue.

That's a non sequitur:
a) even though the original code was written a long time ago, it has been changed many times since then
b) if it isn't broken, don't fix it. Example: changing all occurrences of "string" into "shortstring" will break the system unit. The reason is that a "var a: string" is an "openstring" parameter (with a hidden "high" parameter that tells you the size of the variable that was passed in), while "var a: shortstring" is not. So at the very least, you'd have to change var/out parameters into openstring rather than into shortstring.

Adding support for new functionality (such as unicode file names) would obviously require revising all of that code, but that's because the context changes rather than because it's old.

Jonas

That's a non sequitur: a) even though the original code was written a long time ago, it has

been changed many times since then

b) if it isn't broken, don't fix it. Example: changing all

occurrences of "string" into "shortstring" will break the system unit.

The reason is that a "var a: string" is an "openstring" parameter

(with a hidden "high" parameter that tells you the size of

the variable that was passed in), while "var a: shortstring" is not.

So at the very least, you'd have to change var/out

parameters into openstring rather than into shortstring.

As far as I know you can pass ShortString of any size to a ShortString of maximum length (255). So I don't see how "var a : ShortString" would lead to errors. But of course, you need to be careful.

Nevertheless, I would rewrite such code. Even at the danger of breaking something else (which would hopefully appear directly) I would consider clear code so important that I would not rely on some compiler switch (that may change some day) to make it run. Especially because *if* the compiler switch is changed it would not strike you immediately. It would run a long time until someone (by accident) has a path longer than 255 characters. And then it would not raise an exception (which would point you to the cause) but it would overwrite other data and then crash at a completely unrelated instruction (or just lead to wrong results).