Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-pascal / 2013-09

DoDirSeparators and special filenames on Windows

42 messages · 10 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Bart

Hi,

This is a folowup off a discussion in LazarusDev mailinglist. It concerns "sanitizing" filenames on the Windows platform.

The Windows API for handling filenames in most cases happily accepts filenames that have '/' in them. It treats '/' as a pathdelimiter. It also is very forgiving in using double pathdelimiters. So, filenames like 'C:\//\\foo/\bar' are treated as 'C:\foo\bar'

So far so good.
There is an exception though: filenames that start with '\\?\' are considered to be literal and treated as such by the WinAPI.

So if you try to open '\\?\C:\foo/bar' it will fail, because it treats the '/' as being part of the filename.
Note that '//?/C:/foo/bar' in contrast happily opens 'C:\foo\bar' (nice touch from M$).

Note that prepending filenames with '\\?\' is necessary to open files with names (fully qualified path) longer tham MAXPATH characters. So in real life your progrmam may need to use this filenaming scheme.

Some of the fpc filehandling functions call DoDirSeparators on the filename. (ExpandFilename is one of them).
DoDirSeparators transforms '\\?\C:/foo/bar' into '\\?\C:\foo\bar'

This can lead to strange results where e.g FileExists(Filename) returns FALSE, and FileExists(ExpandFilename(Filename)) returns TRUE, given that FileName = '\\?\C:/foo/bar' and a file 'C:\foo\bar' exists.

My question is: should DoDirSeparators handle this case and leave the filename untouched if it starts with '\\?\', or should we leave it up to the programmer not calling functions like ExpandFilename if a filename starts with '\\?\'

Note: if you want to test what windows does with '/' , '/\' , '\\?\' or even '//?\' don't rely on typing this into Explorer or a console. These do not understand this. You must test the WinApi itself.

Bart

Reply by email

Jürgen Hestermann
to Bart
Bart wrote
Note that '//?/C:/foo/bar' in contrast happily opens 'C:\foo\bar' (nice touch from M$).

Why is this a "nice touch"? IMO it is crap. This kind of inconsistencies in MS programs lead to false assumptions and mislead the user (programmer). If characters are forbitten then they should be forbidden and not *sometimes* be allowed and *sometimes* be converted to something different. Being consequent in this respect would have saved a lot of headaches for generations of programmers.

My question is: should DoDirSeparators handle this case and leave the filename untouched if it starts with '\\?\', or should we leave it up to the programmer not calling functions like ExpandFilename if a filename starts with '\\?\'

In general I would prefer if this '\\?\' prefix is hidden away from the names should be just that: File names. On windows *every* API function that expects file names should be prepended with '\\?\' internally while for Linux this should be omitted. But the programmer should not need to know about this. Internal functions that use WinAPI calls should add the prefix theirself in all cases(!) (if supported) and omit it for Linux. For all situations where the file name is moved/converted from one function to another (when no OS-API functions is involved) it should just leave the name as it is.

I have written my own functions which do this because it is a nightmare to find out which kind of string encoding is expected/required by what function within FPC/Lazarus. The documentation just says "string" but never says whether it is UTF8, UTF16 or whatever. So when I want to be sure what happens I have to reinvent the wheel and write my own OS independend functions which in my case *always* expect UTF8 (which is my standard encoding in all situations) and convert to UTF16 and add the prefix '\\?\' when feeding it to a WinAPI function:

function WinAPIPathName(const Pfad : UTF8String) : UTF8String;
begin
Result := Pfad;
if (length(Result)<=3) or
    (Result[1]='\') and
    (Result[2]='\') and
    (Result[3]='?') and
    (Result[4]='\') then exit;
if (length(Result)>=2) and
    (Result[1]='\') and
    (Result[2]='\') then      // UNC-path, i.e. "\\server\share"
    begin
    Result := '\\?\UN'+Result;
    Result[7] := 'C';
    end
else
    Result := '\\?\'+Result;
end;

which I then call like this:

H  := FindFirstFileW(pwidechar(UTF8Decode(WinAPIPathName(Pathname))),FW);

This way I don't need to bother about path lengths and unicode issues. I just use UTF8 in my programs. And when I need other encodings I know that I have to convert them before calling functions that need UTF8.

What I would expect as a programming interface when writing programs for multiple operating systems would be functions like RenameFileUTF8 and RenameFileUTF16 where I know that I have to use UTF8 (Ansi-)Strings or UTF16 (Ansi-)Strings as parameter without the need to prepend '\\?\' because it is done (if needed) by the internal OS-dependend function.

Reply by email

Sven Barth
to Jürgen Hestermann
"Jürgen Hestermann" wrote 6 lines
> My question is: should DoDirSeparators handle this case and leave the > filename untouched if it starts with '\\?\', or should we leave it up > to the programmer not calling functions like ExpandFilename if a > filename starts with '\\?\' In general I would prefer if this '\\?\' prefix is hidden away from the

names should be just that: File names. On windows
*every* API function that expects file names should be prepended with '\\?\' internally while for Linux this should be omitted. But the programmer should not need to know about this. Internal functions that use WinAPI calls should add the prefix theirself in all cases(!) (if supported) and omit it for Linux. For all situations where the file name is moved/converted from one function to another (when no OS-API functions is involved) it should just leave the name as it is.

I have written my own functions which do this because it is a nightmare

to find out which kind of string encoding is expected/required by what function within FPC/Lazarus. The documentation just says "string" but never says whether it is UTF8, UTF16 or whatever. So when I want to be sure what happens I have to reinvent the wheel and write my own OS independend functions which in my case *always* expect UTF8 (which is my standard encoding in all situations) and convert to UTF16 and add the prefix '\\?\' when feeding it to a WinAPI function:

Show quoted text 25 lines
function WinAPIPathName(const Pfad : UTF8String) : UTF8String; begin Result := Pfad; if (length(Result)<=3) or (Result[1]='\') and (Result[2]='\') and (Result[3]='?') and (Result[4]='\') then exit; if (length(Result)>=2) and (Result[1]='\') and (Result[2]='\') then // UNC-path, i.e. "\\server\share" begin Result := '\\?\UN'+Result; Result[7] := 'C'; end else Result := '\\?\'+Result; end; which I then call like this: H := FindFirstFileW(pwidechar(UTF8Decode(WinAPIPathName(Pathname))),FW); This way I don't need to bother about path lengths and unicode issues. I

just use UTF8 in my programs. And when I need other encodings I know that I have to convert them before calling functions that need UTF8.

The "\\?\" paths don't support relative paths and also not all I/O-APIs support this convention.

In my opinion a programmer needs to know the quirks of the platforms he/she wants to target.

Regards,
Sven

Reply by email

Jürgen Hestermann
to Sven Barth
Sven Barth wrote
The "\\?\" paths don't support relative paths

Then relative paths need to be expanded internally before use. If I have a deeply nested directory structure with paths longer than 255 characters then relative paths should work the same as when having shorter paths, otherwise it would be a ridiculous restriction. If that's what MS has designed (which would be just as I would expect it from their crap interface) then Lazarus/FPC should overcome this (if possible) by expanding all relative paths.

and also not all I/O-APIs support this convention.

For functions that do not support it it should simply not be used. Yet another argument to do it in the background so that the programmer can concentrate on the path name and does not need to think about when to use which naming convention.

In my opinion a programmer needs to know the quirks of the platforms he/she wants to target.

It is always better to know something. Therefore it should be carefully documented. But this is not an argument against it. It would just clean up the mess that MS has left for their users/programmers and unify the Lazarus/FPC file handling interface for all platforms.

Reply by email

Bart
to Jürgen Hestermann: Why is this a "nice touch"? IMO it is crap. This kind of inconsistencies in MS programs…
Jürgen Hestermann wrote 6 lines
Am 2013-09-07 12:52, schrieb Bart: > Note that '//?/C:/foo/bar' in contrast happily opens 'C:\foo\bar' > (nice touch from M$). Why is this a "nice touch"? IMO it is crap.

You missed the intended sarcasme sign. ;-)

Show quoted text 5 lines
In general I would prefer if this '\\?\' prefix is hidden away from the names should be just that: File names. On windows *every* API function that expects file names should be prepended with '\\?\' internally

That would be wrong.
Prepending wiht '\\?\ means that everything should be taken literally, so trying to open/read etc. through WinApi will fail if filename contains double pathedlims, dots, '/' etc, because these are illegal characters in a filename on Windows.

This is especially a problem with Lazarus (the IDE itself), where config files and include directives may contain '/', or are defined as relative paths.

You would also not be able to open files with a relative path: FileExists('foo\bar.txt') wil then internally be translated to "Ask the WinApi if '\\?\foo\bar.txt' exists", and this will fail, since the '\\?\' scheme needs a fully qualified path.
(And in a threaded program you may not want to call ExpandFilename(), since on Windows this is not threadsafe).

Bart

Reply by email

Jürgen Hestermann
to Bart
Bart wrote
That would be wrong. Prepending wiht '\\?\ means that everything should be taken literally, so trying to open/read etc. through WinApi will fail if filename contains double pathedlims,

Of course! Why the hell should there be double pathdelims? And why (silently) accept them as if they were one?

dots,

as written: Relative paths should be expanded. Otherwise you can never use them for paths longer than 255 characters.

'/' etc, because these are illegal

characters in a filename on Windows.

Illegal characters should be taken just like that: Illegal characters. Why should they be accepted/converted? Who adds illegal characters and expects them to be silently converted to something legal? Just the opposite. I have a directory which ends on a space (don't know how it was created). Although this is "illegal", many programs including explorer could handle this because the underlying file system NTFS allows it. But when you now silently change such illegal situations then the file/path cannot be found anymore.

You would also not be able to open files with a relative path:

See above.

(And in a threaded program you may not want to call ExpandFilename(), since on Windows this is not threadsafe).

Then you only have the choice between relative paths working for long paths or threadsafe programs.

Reply by email

Bart
to Jürgen Hestermann
Jürgen Hestermann wrote
Of course! Why the hell should there be double pathdelims? And why (silently) accept them as if they were one?

There should not be, but (history: DOS and) WinApi accept them, so the are NOT illegal in normal filenames.

Show quoted text 6 lines
'/' etc, because these are illegal > characters in a filename on Windows. Illegal characters should be taken just like that: Illegal characters. Why should they be accepted/converted? Who adds illegal characters and expects them to be silently converted to something legal? Just the opposite.

They ('/','.') are only illegal in the '\\?\ scheme, they are perfectly legal (but are converted by WinApi) in "normal" filenames.

Show quoted text 7 lines
> You would also not be able to open files with a relative path: > (And in a threaded program you may not want to call ExpandFilename(), > since on Windows this is not threadsafe). Then you only have the choice between relative paths working for long paths or threadsafe programs.

Well then, say goodbye to the Lazarus IDE, because we want and need both. Mutatis mutandis for many real world programs out there, I guess.

Bart

Reply by email

Mattias Gaertner
to Bart: You missed the intended sarcasme sign. ;-) That would be wrong. Prepending wiht '\\?\…
[...] This is especially a problem with Lazarus (the IDE itself), where config files and include directives may contain '/', or are defined as relative paths.

Filenames from other platforms have to be converted by the application (here: Lazarus IDE) before passing them to RTL file functions. That's why the IDE changes path delims of such values before using them. If not, then you found a bug.

Mattias

Reply by email

Jürgen Hestermann
to Bart: There should not be, but (history: DOS and) WinApi accept them, so the are NOT illegal in…
Bart wrote 5 lines
On 9/7/13, Jürgen Hestermann wrote: > Of course! Why the hell should there be double pathdelims? And why > (silently) accept them as if they were one? There should not be, but (history: DOS and) WinApi accept them, so the are NOT illegal in normal filenames.

In my many decades of MS experience I never have seen double path delimiters anywhere. How do you generate them? I would not know how to do it. And I doubt that all WinAPI function except such (and silently drop one delimiter in the background).

Show quoted text 5 lines
> Illegal characters should be taken just like that: Illegal characters. Why > should they be accepted/converted? Who adds illegal characters and expects > them to be silently converted to something legal? Just the opposite. They ('/','.') are only illegal in the '\\?\ scheme, they are perfectly legal (but are converted by WinApi) in "normal" filenames.

That's not true. An extract from http://msdn.microsoft.com/en-us/library/aa365247.aspx says:

Use any character in the current code page for a name, including Unicode characters and characters in the extended character set (128–255), except for the following: .....
The following reserved characters:
< (less than)

(greater than)

: (colon)
" (double quote)
/ (forward slash)
\ (backslash)
| (vertical bar or pipe)
? (question mark)
* (asterisk)
.....

Regarding relative paths: It would be easy to make an exception if the path starts with dots followed by a backslash (on Windows) in which case no '\\?\' prefix would be added. Still for full paths it *should* be added (if it does not yet exist). This way a path would be usable with and without the prefix even if it is longer than MAXPATH.

Reply by email

Tomas Hajny
to Bart: You missed the intended sarcasme sign. ;-) That would be wrong. Prepending wiht '\\?\…

On Sat, September 7, 2013 18:49, Bart wrote:
.
.

Show quoted text 6 lines
You would also not be able to open files with a relative path: FileExists('foo\bar.txt') wil then internally be translated to "Ask the WinApi if '\\?\foo\bar.txt' exists", and this will fail, since the '\\?\' scheme needs a fully qualified path. (And in a threaded program you may not want to call ExpandFilename(), since on Windows this is not threadsafe).

Why do you believe that ExpandFilename is not threadsafe (on Windows or any other platform)? If you refer to Sysutils.ExpandFilename, its implementation is common and not specific for different platforms. Obviously, there is special treatment implemented for specific characteristics of path names and the set of these characteristics differs among the supported platforms, but that should not make implementation for one platform any special. The only platform specific part inside ExpandFilename is a call to GetDir function retrieving the current directory. If that is not 'threadsafe' by itself (on Windows), then your statement is correct, but I hope that this is not the case. If you mean that some other thread may change the current directory during execution of ExpandFilename - well, that may be the case, but that is not special for Windows and it should be the responsibility of the programmer either not to change the current directory in other threads, or not to allow relative paths (aka referring to a moving target). If you mean something else, please elaborate.

Tomas

Reply by email

Bart
to Tomas Hajny
Tomas Hajny wrote
Why do you believe that ExpandFilename is not threadsafe (on Windows or any other platform)?

..

The only platform specific part inside ExpandFilename is a call to GetDir function retrieving the current directory. If that is not 'threadsafe' by itself (on Windows), then your statement is correct,

And that is exactly the point.
GetDir calls GetCurentDirectory and MS says (see:
http://msdn.microsoft.com/en-us/library/windows/desktop/aa364934%28v=vs.85%29.aspx) that multithreaded applications should not call GetCurrentDirectory.

that some other thread may change the current directory during execution of ExpandFilename - well, that may be the case, but that is not special for Windows and it should be the responsibility of the programmer either not to change the current directory in other threads,

But GetDir just does that: it changes the current directory if it tries to find the current directory on a drive that is not the current drive.
It then calls SetCurrentDirectory with only the drive letter, and calls GetCurrentDirectory again.

Why would I forbid my users to specify relative paths? How would you feel if, as Jürgen suggests we prepend anything with \\?\, you now cannot do a simple:
C:\Devel\Foo>fpc bar.pas -Fu..\MyLib
But you must do
C:\Devel\Foo>fpc C:\Devel\foo\bar.pas -FuC:\Devel\MyLib, because the compiler is mutithreaded (this may be untrue, but just for the sake of the argument), so expanding the filename is unsafe?

For the record: I am not stating that this (handling '\\?\') _must_ be "fixed" in fpc, I am perfectly happy with "you should handle that on the users side", which would mean NOT calling DoDirSeparators (and any function that uses that) on such a filename.
It would be nice however if a "formal" point of view from the fpc core team was given on this subject.

Bart

Reply by email

Jürgen Hestermann
to Bart
Bart wrote
And that is exactly the point. GetDir calls GetCurentDirectory and MS says (see: http://msdn.microsoft.com/en-us/library/windows/desktop/aa364934%28v=vs.85%29.aspx) that multithreaded applications should not call GetCurrentDirectory.

It also says there that multithreaded applications should not use relative paths. It seems that WinAPI functions which allow relative paths suffer from the same problem: They have to call GetCurrentDirectory too and are therefore not thread-safe either. That means that you can use relative paths only for single-threaded programs. Then it would not matter whether GetCurrentDirectory is called from FPC (while expanding) or through the WinAPI.

So I see no harm in predending '\\?\' and expand relative paths if needed. Programmers who do not think/know about the special handling of long paths under Windows (maybe because they started with Linux) would still get a program that works with long paths too.
I think this is expecially important because of the multi platform approach which IMO means that the programmer should *not* need to think about OS specialities (as far as possible). There are enough of them left which the programmer has to take care about, no need to add those who are avoidable.

Reply by email

Tomas Hajny
to Bart: .. And that is exactly the point. GetDir calls GetCurentDirectory and MS says (see:…
Bart wrote 14 lines
On 9/8/13, Tomas Hajny wrote: > Why do you believe that ExpandFilename is not threadsafe (on Windows or > any other platform)? .. > The only platform specific part inside > ExpandFilename is a call to GetDir function retrieving the current > directory. If that is not 'threadsafe' by itself (on Windows), then your > statement is correct, And that is exactly the point. GetDir calls GetCurentDirectory and MS says (see: http://msdn.microsoft.com/en-us/library/windows/desktop/aa364934%28v=vs.85%29.aspx) that multithreaded applications should not call GetCurrentDirectory.

OK, I'll have a look at the MSDN link once I sit at some machine with a bigger display than my phone.

Show quoted text 11 lines
> that some other thread may change the current directory during execution > of ExpandFilename - well, that may be the case, but that is not special > for Windows and it should be the responsibility of the programmer either > not to change the current directory in other threads, But GetDir just does that: it changes the current directory if it tries to find the current directory on a drive that is not the current drive. It then calls SetCurrentDirectory with only the drive letter, and calls GetCurrentDirectory again.

Good, thanks for the pointer. We may indeed want to add some safeguards to the Windows GetDir implementation (or possibly check some alternative solution for finding out the current directory on other drives).

Show quoted text 8 lines
Why would I forbid my users to specify relative paths? How would you feel if, as Jürgen suggests we prepend anything with \\?\, you now cannot do a simple: C:\Devel\Foo>fpc bar.pas -Fu..\MyLib But you must do C:\Devel\Foo>fpc C:\Devel\foo\bar.pas -FuC:\Devel\MyLib, because the compiler is mutithreaded (this may be untrue, but just for the sake of the argument), so expanding the filename is unsafe?

Even if the compiler becomes multithreaded in the future (not the case now), it should probably still handle the command line parameters within one space and perform the translation to full paths there. After all, it means that not only ExpandFilename is not threadsafe, but basically any opening of a file (and other I/O operations working with file names) wouldn't be threadsafe either. We will have to solve it somehow.

Show quoted text 6 lines
For the record: I am not stating that this (handling '\\?\') _must_ be "fixed" in fpc, I am perfectly happy with "you should handle that on the users side", which would mean NOT calling DoDirSeparators (and any function that uses that) on such a filename. It would be nice however if a "formal" point of view from the fpc core team was given on this subject.

OK, that is a fair point. Let's wait for the opinion of others.

Tomas

Reply by email

Bart
to Jürgen Hestermann: It also says there that multithreaded applications should not use relative paths. It…
Jürgen Hestermann wrote 8 lines
It also says there that multithreaded applications should not use relative paths. It seems that WinAPI functions which allow relative paths suffer from the same problem: They have to call GetCurrentDirectory too and are therefore not thread-safe either. That means that you can use relative paths only for single-threaded programs.

Yes, it is a conundrum.
You cannot be threadsafe and use filenames that are not fully qualified. And yet, I think many programs do, simply because the programmer did not realize this (me included).

So I see no harm in predending '\\?\' and expand relative paths if needed.

If needed, yes maybe, but if done silently it should then sanitize the filename (change '/' into '\' , remove double PathDelims) for backwards compatibiity, and the path should be expanded. (It should, if apllied, probably be configurable @runtime.)

Programmers who do not think/know about the special handling of long paths under Windows (maybe because they started with Linux) would still get a program that works with long paths too.

Hmm, long paths fail in Linux ATM (2.6.2) because all IO operations use ShortStrings.

I think this is expecially important because of the multi platform approach which IMO means that the programmer should *not* need to think about OS specialities (as far as possible).

Good point.

Another point is that (at least in Delphi mode) all this must remain Delphi compatible.

Bart

Reply by email

Bart
to Tomas Hajny: OK, I'll have a look at the MSDN link once I sit at some machine with a bigger display…
Tomas Hajny wrote
OK, I'll have a look at the MSDN link once I sit at some machine with a bigger display than my phone.

;-)

Good, thanks for the pointer. We may indeed want to add some safeguards to the Windows GetDir implementation (or possibly check some alternative solution for finding out the current directory on other drives).

It may very well not be possible.
The ugly truth ma be that we have to learn to live with it (it's been a long time since DOS).

After all, it means that not only ExpandFilename is not threadsafe, but basically any opening of a file (and other I/O operations working with file names) wouldn't be threadsafe either. We will have to solve it somehow.

Removing Thread support would solve it definately ;-))

It may be a good idea to document this somewhere (in the apprpriate sections about File IO as well as MultiThreading)? I mus admit, that at this point I was too lazy to look for that.3

OK, that is a fair point. Let's wait for the opinion of others.

Should be interesting.

Bart

Reply by email

Sven Barth
to Bart: .. And that is exactly the point. GetDir calls GetCurentDirectory and MS says (see:…
Bart wrote 14 lines
On 9/8/13, Tomas Hajny wrote: > Why do you believe that ExpandFilename is not threadsafe (on Windows or > any other platform)? .. > The only platform specific part inside > ExpandFilename is a call to GetDir function retrieving the current > directory. If that is not 'threadsafe' by itself (on Windows), then your > statement is correct, And that is exactly the point. GetDir calls GetCurentDirectory and MS says (see: http://msdn.microsoft.com/en-us/library/windows/desktop/aa364934%28v=vs.85%29.aspx) that multithreaded applications should not call GetCurrentDirectory.

The problem is not GetCurrentDirectory itself (this function and SetCurrentDirectory use the process's PEB lock to access the current directory which is stored inside the PEB as well), but that different threads could call SetCurrentDirectory with different values and thus a thread who expected the current directory to be a specific one might resolve a relative path (by using another WinAPI function) using a current directory set by a different thread.

So this can not be solved by using a lock inside ExpandFilename and might happen on other systems as well as long as the current directory is not stored per thread.

Regards,
Sven

Reply by email

Sven Barth
to Tomas Hajny: OK, I'll have a look at the MSDN link once I sit at some machine with a bigger display…
Tomas Hajny wrote 13 lines
>> of ExpandFilename - well, that may be the case, but that is not special >> for Windows and it should be the responsibility of the programmer either >> not to change the current directory in other threads, > > But GetDir just does that: it changes the current directory if it > tries to find the current directory on a drive that is not the current > drive. > It then calls SetCurrentDirectory with only the drive letter, and > calls GetCurrentDirectory again. Good, thanks for the pointer. We may indeed want to add some safeguards to the Windows GetDir implementation (or possibly check some alternative solution for finding out the current directory on other drives).

Would be a solution to a problem that does not exist in the way that it's discussed here. See my mail to Bart.

Regards,
Sven

Reply by email

Sven Barth
to Jürgen Hestermann: In my many decades of MS experience I never have seen double path delimiters anywhere.…
Jürgen Hestermann wrote 11 lines
Am 2013-09-07 21:46, schrieb Bart: > On 9/7/13, Jürgen Hestermann wrote: >> Of course! Why the hell should there be double pathdelims? And why >> (silently) accept them as if they were one? > There should not be, but (history: DOS and) WinApi accept them, so the > are NOT illegal in normal filenames. In my many decades of MS experience I never have seen double path delimiters anywhere. How do you generate them? I would not know how to do it. And I doubt that all WinAPI function except such (and silently drop one delimiter in the background).

You generate them by letting the user input a path and passing it to a function as a normal Win32 style path "X:\..." instead of a "\\?\X:..." path. The Windows function that converts the Win32 style path to a NT path will also reduce duplicated path delimiters. So every WinAPI function that accepts a Win32 style path (so basically *all* WinAPI functions) will also accept duplicate path path delimiters. Windows Explorer does not allow them, because it uses the long path names while on the other hand e.g. PowerShell and Cmd do indeed allow them.

Regards,
Sven

Reply by email

Mattias Gaertner
to Sven Barth: The problem is not GetCurrentDirectory itself (this function and SetCurrentDirectory use…
Show quoted text 22 lines
On 08.09.2013 14:17, Bart wrote: > On 9/8/13, Tomas Hajny wrote: > >> Why do you believe that ExpandFilename is not threadsafe (on Windows or >> any other platform)? [...] > And that is exactly the point. > GetDir calls GetCurentDirectory and MS says (see: > http://msdn.microsoft.com/en-us/library/windows/desktop/aa364934%28v=vs.85%29.aspx) > that multithreaded applications should not call GetCurrentDirectory. The problem is not GetCurrentDirectory itself (this function and SetCurrentDirectory use the process's PEB lock to access the current directory which is stored inside the PEB as well), but that different threads could call SetCurrentDirectory with different values and thus a thread who expected the current directory to be a specific one might resolve a relative path (by using another WinAPI function) using a current directory set by a different thread. So this can not be solved by using a lock inside ExpandFilename and might happen on other systems as well as long as the current directory is not stored per thread.

Actually the problem is that Windows has multiple current directories, but you can only read the one of the active drive.

For example:
Thread 1: ExpandFilename('C:test.pas');
Thread 2: ExpandFilename('B:test.pas');

In order to get the current directory of C: (B:) you have to switch the current drive and switch back.

Mattias

Reply by email

Tomas Hajny
to Sven Barth: The problem is not GetCurrentDirectory itself (this function and SetCurrentDirectory use…
Sven Barth wrote 24 lines
On 08.09.2013 14:17, Bart wrote: > On 9/8/13, Tomas Hajny wrote: > >> Why do you believe that ExpandFilename is not threadsafe (on Windows or >> any other platform)? > .. >> The only platform specific part inside >> ExpandFilename is a call to GetDir function retrieving the current >> directory. If that is not 'threadsafe' by itself (on Windows), then >> your >> statement is correct, > > And that is exactly the point. > GetDir calls GetCurentDirectory and MS says (see: > http://msdn.microsoft.com/en-us/library/windows/desktop/aa364934%28v=vs.85%29.aspx) > that multithreaded applications should not call GetCurrentDirectory. The problem is not GetCurrentDirectory itself (this function and SetCurrentDirectory use the process's PEB lock to access the current directory which is stored inside the PEB as well), but that different threads could call SetCurrentDirectory with different values and thus a thread who expected the current directory to be a specific one might resolve a relative path (by using another WinAPI function) using a current directory set by a different thread.

I believe that there are three different things:

1) We have a sequence in GetDir implementation for Windows targets which changes the current directory. While this is obviously necessary due to limitations of the respective Win32/64 API for retrieving current directory combined with characteristics of the supported path names, it may not be expected by the programmers who may have no call to SetCurrentDirectory within their code and thus expect no issues with thread safety. I don't know if it is possible to use a critical section or something like that to avoid switching to another thread during this context, but it might be useful to think about it in my opinion.

2) Restrictions of Win32/64 API (relative paths not supported for long paths) which will require a decision how to deal with this fact in FPC RTL
- basically, either find a "safe" way for expanding the relative paths to absolute ones before calling the Win32/64 API with the long paths, or decide not to support these long paths in the cross-platform RTL functions (i.e. leave it up to the programmers to use the Win32/64 functions themselves when requiring support for such long paths in their programs).

3) Programmers changing the current directory in their own code possibly leading to unexpected behaviour (unexpected by the respective programmers). I certainly wouldn't try to solve this one since that for sure cannot be solved on our side.

So this can not be solved by using a lock inside ExpandFilename and might happen on other systems as well as long as the current directory is not stored per thread.

Out of the 3 points above, the third one may obviously happen on any platform (and it should be the programmer's responsibility).

The second one is obviously specific to Win32/64 API.

The first one might exist for other platforms too, but I'm not aware of any particular one apart from Win32/64 (the pattern would be: support drives and/or volumes, allow paths to refer to current directory specific for that drive/volume, obviously also support multithreading, however do not allow querying the current directory on a specific drive/volume - e.g. OS/2 fits the first three conditions but not the last one, Unix platforms have no drives, Novell doesn't maintain current directory across different volumes and thus doesn't allow relative paths there as far as I know and DOS fits only the first two conditions).

Tomas

Reply by email

Tomas Hajny
to Bart: ;-) It may very well not be possible. The ugly truth ma be that we have to learn to live…
Bart wrote
On 9/8/13, Tomas Hajny wrote:

.
.

Show quoted text 8 lines
> Good, thanks for the pointer. We may indeed want to add some safeguards > to > the Windows GetDir implementation (or possibly check some alternative > solution for finding out the current directory on other drives). It may very well not be possible. The ugly truth ma be that we have to learn to live with it (it's been a long time since DOS).

This has nothing to do with time since DOS - someone designing the Win32/64 apparently either decided not to cover all the Win32/64 features (including those which were originally probably indeed added for DOS compatibility like drive letters and current directories maintained in parallel for different drives) within the provided API set, or simply didn't realize that some features are not supported well by the provided API.

Show quoted text 6 lines
> After all, it > means that not only ExpandFilename is not threadsafe, but basically any > opening of a file (and other I/O operations working with file names) > wouldn't be threadsafe either. We will have to solve it somehow. Removing Thread support would solve it definately ;-))

Indeed, and removing Win32 and Win64 support would solve it too. ;-)))

It may be a good idea to document this somewhere (in the apprpriate sections about File IO as well as MultiThreading)? I mus admit, that at this point I was too lazy to look for that.3

Yes, documenting it is the minimum which should be done.

Tomas

Reply by email

Sven Barth
to Mattias Gaertner: On Sun, 08 Sep 2013 19:32:23 +0200 Actually the problem is that Windows has multiple…
Mattias Gaertner wrote 26 lines
On Sun, 08 Sep 2013 19:32:23 +0200 Sven Barth wrote: > On 08.09.2013 14:17, Bart wrote: >> On 9/8/13, Tomas Hajny wrote: >> >>> Why do you believe that ExpandFilename is not threadsafe (on Windows or >>> any other platform)? > [...] >> And that is exactly the point. >> GetDir calls GetCurentDirectory and MS says (see: >> http://msdn.microsoft.com/en-us/library/windows/desktop/aa364934%28v=vs.85%29.aspx) >> that multithreaded applications should not call GetCurrentDirectory. > The problem is not GetCurrentDirectory itself (this function and > SetCurrentDirectory use the process's PEB lock to access the current > directory which is stored inside the PEB as well), but that different > threads could call SetCurrentDirectory with different values and thus a > thread who expected the current directory to be a specific one might > resolve a relative path (by using another WinAPI function) using a > current directory set by a different thread. > > So this can not be solved by using a lock inside ExpandFilename and > might happen on other systems as well as long as the current directory > is not stored per thread. Actually the problem is that Windows has multiple current directories, but you can only read the one of the active drive.

This. Is. Wrong.

MSDN remark for Set-/GetCurrentDirectory: Each process has a single current directory that consists of two parts: ...

The current directory per drive is only for the command interpreter and programs that use the crt library (not to be confused with our crt unit).

Regards,
Sven

Reply by email

Sven Barth
to Tomas Hajny: I believe that there are three different things: 1) We have a sequence in GetDir…
Tomas Hajny wrote 34 lines
On Sun, September 8, 2013 19:32, Sven Barth wrote: > On 08.09.2013 14:17, Bart wrote: >> On 9/8/13, Tomas Hajny wrote: >> >>> Why do you believe that ExpandFilename is not threadsafe (on Windows or >>> any other platform)? >> .. >>> The only platform specific part inside >>> ExpandFilename is a call to GetDir function retrieving the current >>> directory. If that is not 'threadsafe' by itself (on Windows), then >>> your >>> statement is correct, >> And that is exactly the point. >> GetDir calls GetCurentDirectory and MS says (see: >> http://msdn.microsoft.com/en-us/library/windows/desktop/aa364934%28v=vs.85%29.aspx) >> that multithreaded applications should not call GetCurrentDirectory. > The problem is not GetCurrentDirectory itself (this function and > SetCurrentDirectory use the process's PEB lock to access the current > directory which is stored inside the PEB as well), but that different > threads could call SetCurrentDirectory with different values and thus a > thread who expected the current directory to be a specific one might > resolve a relative path (by using another WinAPI function) using a > current directory set by a different thread. I believe that there are three different things: 1) We have a sequence in GetDir implementation for Windows targets which changes the current directory. While this is obviously necessary due to limitations of the respective Win32/64 API for retrieving current directory combined with characteristics of the supported path names, it may not be expected by the programmers who may have no call to SetCurrentDirectory within their code and thus expect no issues with thread safety. I don't know if it is possible to use a critical section or something like that to avoid switching to another thread during this context, but it might be useful to think about it in my opinion.

On Windows this is completely useless. A process has only *one* current directory. The current-directory-per-drive ones are only implemented inside the command line interpreter and other programs that use the crt library (not to be confused with the crt unit). So the calling of SetCurrentDirectory inside ExpandFilename is not only not threadsafe, but doesn't result in something useful either!

Show quoted text 7 lines
2) Restrictions of Win32/64 API (relative paths not supported for long paths) which will require a decision how to deal with this fact in FPC RTL - basically, either find a "safe" way for expanding the relative paths to absolute ones before calling the Win32/64 API with the long paths, or decide not to support these long paths in the cross-platform RTL functions (i.e. leave it up to the programmers to use the Win32/64 functions themselves when requiring support for such long paths in their programs).

Feedback of others would be good here to decide how to proceed...

3) Programmers changing the current directory in their own code possibly leading to unexpected behaviour (unexpected by the respective programmers). I certainly wouldn't try to solve this one since that for sure cannot be solved on our side.

Agreed.

Regards,
Sven

Reply by email

Sven Barth
to Sven Barth: This. Is. Wrong. MSDN remark for Set-/GetCurrentDirectory: Each process has a single…
Sven Barth wrote 37 lines
Am 08.09.2013 22:13, schrieb Mattias Gaertner: > On Sun, 08 Sep 2013 19:32:23 +0200 > Sven Barth wrote: > >> On 08.09.2013 14:17, Bart wrote: >>> On 9/8/13, Tomas Hajny wrote: >>> >>>> Why do you believe that ExpandFilename is not threadsafe (on >>>> Windows or >>>> any other platform)? >> [...] >>> And that is exactly the point. >>> GetDir calls GetCurentDirectory and MS says (see: >>> http://msdn.microsoft.com/en-us/library/windows/desktop/aa364934%28v=vs.85%29.aspx) >>> >>> that multithreaded applications should not call GetCurrentDirectory. >> The problem is not GetCurrentDirectory itself (this function and >> SetCurrentDirectory use the process's PEB lock to access the current >> directory which is stored inside the PEB as well), but that different >> threads could call SetCurrentDirectory with different values and thus a >> thread who expected the current directory to be a specific one might >> resolve a relative path (by using another WinAPI function) using a >> current directory set by a different thread. >> >> So this can not be solved by using a lock inside ExpandFilename and >> might happen on other systems as well as long as the current directory >> is not stored per thread. > Actually the problem is that Windows has multiple current directories, > but you can only read the one of the active drive. This. Is. Wrong. MSDN remark for Set-/GetCurrentDirectory: Each process has a single current directory that consists of two parts: ... The current directory per drive is only for the command interpreter and programs that use the crt library (not to be confused with our crt unit).

Correction, there seems to be a bit more magic going on... I'll report back.

Regards,
Sven

Reply by email

Sven Barth
to Sven Barth: On Windows this is completely useless. A process has only *one* current directory. The…
Sven Barth wrote 44 lines
Am 08.09.2013 22:32, schrieb Tomas Hajny: > On Sun, September 8, 2013 19:32, Sven Barth wrote: >> On 08.09.2013 14:17, Bart wrote: >>> On 9/8/13, Tomas Hajny wrote: >>> >>>> Why do you believe that ExpandFilename is not threadsafe (on >>>> Windows or >>>> any other platform)? >>> .. >>>> The only platform specific part inside >>>> ExpandFilename is a call to GetDir function retrieving the current >>>> directory. If that is not 'threadsafe' by itself (on Windows), then >>>> your >>>> statement is correct, >>> And that is exactly the point. >>> GetDir calls GetCurentDirectory and MS says (see: >>> http://msdn.microsoft.com/en-us/library/windows/desktop/aa364934%28v=vs.85%29.aspx) >>> >>> that multithreaded applications should not call GetCurrentDirectory. >> The problem is not GetCurrentDirectory itself (this function and >> SetCurrentDirectory use the process's PEB lock to access the current >> directory which is stored inside the PEB as well), but that different >> threads could call SetCurrentDirectory with different values and thus a >> thread who expected the current directory to be a specific one might >> resolve a relative path (by using another WinAPI function) using a >> current directory set by a different thread. > I believe that there are three different things: > > 1) We have a sequence in GetDir implementation for Windows targets which > changes the current directory. While this is obviously necessary due to > limitations of the respective Win32/64 API for retrieving current > directory combined with characteristics of the supported path names, it > may not be expected by the programmers who may have no call to > SetCurrentDirectory within their code and thus expect no issues with > thread safety. I don't know if it is possible to use a critical > section or > something like that to avoid switching to another thread during this > context, but it might be useful to think about it in my opinion. On Windows this is completely useless. A process has only *one* current directory. The current-directory-per-drive ones are only implemented inside the command line interpreter and other programs that use the crt library (not to be confused with the crt unit). So the calling of SetCurrentDirectory inside ExpandFilename is not only not threadsafe, but doesn't result in something useful either!

Correction, there seems to be a bit more magic going on... I'll report back.

Regards,
Sven

Reply by email

Sven Barth
to Sven Barth
Sven Barth wrote 49 lines
Am 09.09.2013 13:31, schrieb Sven Barth: > Am 08.09.2013 22:32, schrieb Tomas Hajny: >> On Sun, September 8, 2013 19:32, Sven Barth wrote: >>> On 08.09.2013 14:17, Bart wrote: >>>> On 9/8/13, Tomas Hajny wrote: >>>> >>>>> Why do you believe that ExpandFilename is not threadsafe (on >>>>> Windows or >>>>> any other platform)? >>>> .. >>>>> The only platform specific part inside >>>>> ExpandFilename is a call to GetDir function retrieving the current >>>>> directory. If that is not 'threadsafe' by itself (on Windows), then >>>>> your >>>>> statement is correct, >>>> And that is exactly the point. >>>> GetDir calls GetCurentDirectory and MS says (see: >>>> http://msdn.microsoft.com/en-us/library/windows/desktop/aa364934%28v=vs.85%29.aspx) >>>> >>>> that multithreaded applications should not call GetCurrentDirectory. >>> The problem is not GetCurrentDirectory itself (this function and >>> SetCurrentDirectory use the process's PEB lock to access the current >>> directory which is stored inside the PEB as well), but that different >>> threads could call SetCurrentDirectory with different values and >>> thus a >>> thread who expected the current directory to be a specific one might >>> resolve a relative path (by using another WinAPI function) using a >>> current directory set by a different thread. >> I believe that there are three different things: >> >> 1) We have a sequence in GetDir implementation for Windows targets >> which >> changes the current directory. While this is obviously necessary due to >> limitations of the respective Win32/64 API for retrieving current >> directory combined with characteristics of the supported path names, it >> may not be expected by the programmers who may have no call to >> SetCurrentDirectory within their code and thus expect no issues with >> thread safety. I don't know if it is possible to use a critical >> section or >> something like that to avoid switching to another thread during this >> context, but it might be useful to think about it in my opinion. > On Windows this is completely useless. A process has only *one* > current directory. The current-directory-per-drive ones are only > implemented inside the command line interpreter and other programs > that use the crt library (not to be confused with the crt unit). So > the calling of SetCurrentDirectory inside ExpandFilename is not only > not threadsafe, but doesn't result in something useful either! Correction, there seems to be a bit more magic going on... I'll report back.

Ok, while a process does indeed only have a single current directory the kernel32 DLL is setting up environment variables to reference the current directories of other drives and resolves them when SetCurrentDirectory is called.

My suggestion: Don't use the SetCurrentDirectory stuff inside GetDir anymore, but instead use GetFullPathName with the drive as parameter (see
http://msdn.microsoft.com/en-us/library/windows/desktop/aa364963%28v=vs.85%29.aspx ). It's still not thread safe in sense of your (3) (but that is the programmer's problem anyway), but the GetDir function itself would be thread safe.
Just in case: GetFullPathName is supported in Win9x and since Windows NT 3.0.

Regards,
Sven

Reply by email

Sven Barth
to Mattias Gaertner
Mattias Gaertner wrote 8 lines
On Mon, 09 Sep 2013 13:25:31 +0200 Sven Barth wrote: > [...] >> Actually the problem is that Windows has multiple current directories, >> but you can only read the one of the active drive. > This. Is. Wrong. If the GetFullPathName works, I'm glad that I was wrong. :)

Sorry ^^ The "magic" happened inside SetCurrentDirectory and not inside GetCurrentDirectory which is why I missed it.

Regards,
Sven

Reply by email

Jürgen Hestermann
to Sven Barth: You generate them by letting the user input a path and passing it to a function as a…
Sven Barth wrote
On 08.09.2013 10:44, Jürgen Hestermann wrote: > In my many decades of MS experience I never have seen double path > delimiters anywhere. How do you generate them? I would not know how to do it. Windows Explorer does not allow them, because it uses the long path names while on the other hand e.g. PowerShell and Cmd do indeed allow them.

That does not explain how double delimeters creep into paths. Because it seems hard to generate them I can't believe that they commonly exist.

Reply by email

Sven Barth
to Jürgen Hestermann
Jürgen Hestermann wrote 13 lines
Am 2013-09-08 19:56, schrieb Sven Barth: > On 08.09.2013 10:44, Jürgen Hestermann wrote: >> In my many decades of MS experience I never have seen double path >> delimiters anywhere. How do you generate them? I would not know how >> to do it. > Windows Explorer does not allow them, because it uses the long path > names while on the other hand e.g. PowerShell and Cmd do indeed allow > them. That does not explain how double delimeters creep into paths. Because it seems hard to generate them I can't believe that they commonly exist.

When editing a path inside an edit box I (and I also know others) manage here and then to miss a "\" so that the resulting path as duplicate path delimiters...

Regards,
Sven

Reply by email

Tomas Hajny
to Sven Barth
Sven Barth wrote 18 lines
On 09.09.2013 18:31, Jürgen Hestermann wrote: > > Am 2013-09-08 19:56, schrieb Sven Barth: >> On 08.09.2013 10:44, Jürgen Hestermann wrote: >>> In my many decades of MS experience I never have seen double path >>> delimiters anywhere. How do you generate them? I would not know how >>> to do it. >> Windows Explorer does not allow them, because it uses the long path >> names while on the other hand e.g. PowerShell and Cmd do indeed allow >> them. > > That does not explain how double delimeters creep into paths. > Because it seems hard to generate them I can't believe that they > commonly exist. When editing a path inside an edit box I (and I also know others) manage here and then to miss a "\" so that the resulting path as duplicate path delimiters...

...and the issue is that at least some Windows API functions happily accept such paths, i.e. programs using such API functions accept them too. If FPC RTL manages to "translate" a path accepted by Windows (and other programs not compiled to FPC too) to something which is not accepted by Windows (API) any longer, then we may consider fixing such "translation".

Tomas

Reply by email

Tomas Hajny
to Sven Barth: Ok, while a process does indeed only have a single current directory the kernel32 DLL is…
Sven Barth wrote
Am 09.09.2013 13:47, schrieb Sven Barth: > Am 09.09.2013 13:31, schrieb Sven Barth: >> Am 08.09.2013 22:32, schrieb Tomas Hajny:

.
.

Show quoted text 35 lines
>>> 1) We have a sequence in GetDir implementation for Windows targets >>> which >>> changes the current directory. While this is obviously necessary due >>> to >>> limitations of the respective Win32/64 API for retrieving current >>> directory combined with characteristics of the supported path names, >>> it >>> may not be expected by the programmers who may have no call to >>> SetCurrentDirectory within their code and thus expect no issues with >>> thread safety. I don't know if it is possible to use a critical >>> section or >>> something like that to avoid switching to another thread during this >>> context, but it might be useful to think about it in my opinion. >> On Windows this is completely useless. A process has only *one* >> current directory. The current-directory-per-drive ones are only >> implemented inside the command line interpreter and other programs >> that use the crt library (not to be confused with the crt unit). So >> the calling of SetCurrentDirectory inside ExpandFilename is not only >> not threadsafe, but doesn't result in something useful either! > Correction, there seems to be a bit more magic going on... I'll report > back. Ok, while a process does indeed only have a single current directory the kernel32 DLL is setting up environment variables to reference the current directories of other drives and resolves them when SetCurrentDirectory is called. My suggestion: Don't use the SetCurrentDirectory stuff inside GetDir anymore, but instead use GetFullPathName with the drive as parameter (see http://msdn.microsoft.com/en-us/library/windows/desktop/aa364963%28v=vs.85%29.aspx ). It's still not thread safe in sense of your (3) (but that is the programmer's problem anyway), but the GetDir function itself would be thread safe. Just in case: GetFullPathName is supported in Win9x and since Windows NT 3.0.

Yes, this is a good idea.

Actually, my testing (performed on WinXP; see attachment - C:\WINDOWS\SYSTEM32 and D:\TEMP are assumed to exist) shows that calling SetCurrentDirectory has no impact to the remembered values of current directories on other (non-default) drives and only the directories valid on startup are used (I used the *A variants of Win32 API calls, but I hope that this shouldn't make a difference). This raises a question if we shouldn't update the environment variable ourselves within ChDir, because as of now the behaviour is different compared to other platforms (e.g. the first test program compiled for DOS and Windows gives different results when run on the same Windows machine).

If we decide _not_ to change the ChDir behaviour (i.e. not to set the environment variable), we also need to decide whether to prefer GetCurrentDirectory or GetFullPathName result if the requested drive matches the current one, because the results would differ in cases like this:

 ChDir ('C:\WINDOWS\SYSTEM32');
 GetDir (3, S);

If the current directory for C: is e.g. C:\ on startup, GetFullPathname returns C:\, whereas GetCurrentDirectory returns C:\WINDOWS\SYSTEM32.

Tomas

Reply by email

Jürgen Hestermann
to Tomas Hajny: ...and the issue is that at least some Windows API functions happily accept such paths,…
Tomas Hajny wrote 5 lines
...and the issue is that at least some Windows API functions happily accept such paths, i.e. programs using such API functions accept them too. If FPC RTL manages to "translate" a path accepted by Windows (and other programs not compiled to FPC too) to something which is not accepted by Windows (API) any longer, then we may consider fixing such "translation".

IMO, when a path contains two consecutive path delimiters then something is wrong. It could also be that someone wanted to type a letter between both delimiters so dropping one of them is trying to guess what the user wanted which leads to errors in most cases. Suddenly a file or directory is accessed that was not meant by the user. A path should be correct and no more or less smart second guessing should be done. Otherwise you can also start a "nearest match" search that tries to find another directory in case the specified one is not found. That's something I would expect in C but not in Pascal. And even if the WinAPI allows such flaws I don't think that Pascal programmers like this too.

Reply by email

Unknown author
Jürgen Hestermann wrote 11 lines
Am 12013090922:28, schrieb Tomas Hajny: > ...and the issue is that at least some Windows API functions happily accept such paths, > i.e. programs using such API functions accept them too. > If FPC RTL manages to "translate" a path accepted by Windows > (and other programs not compiled to FPC too) to something which is not accepted by > Windows (API) any longer, then we may consider fixing such "translation". IMO, when a path contains two consecutive path delimiters then something is wrong.

sadly, i've seen this multiple delimiters thing a lot in recent years... it seems that in many cases, they are simply read as one delimiter...

i've seen this in a lot of web URLs when hiliting and/or clicking on URLs as well as in numerous applications on both *nix and winwhatever... some of this has even been seen in ported applications to OS2...

Reply by email

Mattias Gaertner
to Unknown author
Show quoted text 16 lines
On Tuesday, September 10, 2013 11:35 AM, Jürgen Hestermann wrote: > Am 12013090922:28, schrieb Tomas Hajny: > > ...and the issue is that at least some Windows API functions happily accept > such paths, > > i.e. programs using such API functions accept them too. > > If FPC RTL manages to "translate" a path accepted by Windows > > (and other programs not compiled to FPC too) to something which is not > accepted by > > Windows (API) any longer, then we may consider fixing such "translation". > > IMO, when a path contains two consecutive path delimiters then something is > wrong. sadly, i've seen this multiple delimiters thing a lot in recent years... it seems that in many cases, they are simply read as one delimiter... i've seen this in a lot of web URLs when hiliting and/or clicking on URLs as well as in numerous applications on both *nix and winwhatever... some of this has even been seen in ported applications to OS2...

Many webservers are running on Linux, where double path delims are pretty normal. You see them pretty often. A common cause are lines like this:

File=$Directory/filename.ext

You don't need to check if $Directory has already a '/' at the end, you simply concatenate. Reason is that Linux does not allow empty file names, so a '//' is never ambiguous and can safely be treated as a single '/'.

Mattias

Reply by email

Lukasz Sokol
to Jürgen Hestermann: IMO, when a path contains two consecutive path delimiters then something is wrong. It…
Jürgen Hestermann wrote 13 lines
Am 2013-09-09 22:28, schrieb Tomas Hajny: > ...and the issue is that at least some Windows API functions happily accept such paths, > i.e. programs using such API functions accept them too. > If FPC RTL manages to "translate" a path accepted by Windows > (and other programs not compiled to FPC too) to something which is not accepted by > Windows (API) any longer, then we may consider fixing such "translation". IMO, when a path contains two consecutive path delimiters then something is wrong. It could also be that someone wanted to type a letter between both delimiters so dropping one of them is trying to guess what the user wanted which leads to errors in most cases. Suddenly a file or directory is accessed that was not meant by the user. A path should be correct and no more or less smart second guessing should be done.

Only plausible 'smart guess' here is that '\/' or '/\' or '\\' in the path be converted to '\' simply, for the '\\?\'mode (on windows of course, Linux doesn't care about '//'s in path)

Otherwise you can also start a "nearest match" search that tries to find another directory in case the specified one is not found. That's something I would expect in C but not in Pascal. And even if the WinAPI allows such flaws I don't think that Pascal programmers like this too.

Nearest match search takes cpu, time and resources... GetLastOSError (and its format message companion) would tell us that path doesn't exist, will it not?

L.

Reply by email

Jürgen Hestermann
to Mattias Gaertner: On Tue, 10 Sep 2013 20:30:16 -0400 Many webservers are running on Linux, where double…
Mattias Gaertner wrote 8 lines
Many webservers are running on Linux, where double path delims are pretty normal. You see them pretty often. A common cause are lines like this: File=$Directory/filename.ext You don't need to check if $Directory has already a '/' at the end, you simply concatenate. Reason is that Linux does not allow empty file names, so a '//' is never ambiguous and can safely be treated as a single '/'.

Well, Windows doesn't allow empty file names either. And double delimiters *are* ambiguous:
Has a (one letter) file name been forgotten or was an additional delimiter typed (or appended by bad programmed routines)? The OS always seems to assume the latter but is it right?

And another question is: What appeared first?
The lazy programmers who did not care about correct path names or the OS that allowed such things and so provoked the laziness.

I am realy astonished that nobody seems to find anything wrong with this.

Reply by email

Reimar Grabowski
to Jürgen Hestermann
And double delimiters *are* ambiguous: Has a (one letter) file name been forgotten or was an additional delimiter typed (or appended by bad programmed routines)?

Ambiguity is not defined by how the symbol came into being. You just look at the symbol as is. So Mattias statement *is* correct.

I am realy astonished that nobody seems to find anything wrong with this.

mkdir hans
mkdir hans////////pete
cd hans
ls -la
total 12
drwxrwxr-x 3 user user 4096 Sep 11 19:13 .
drwxrwxrwt 9 root root 4096 Sep 11 19:13 ..
drwxrwxr-x 2 user user 4096 Sep 11 19:13 pete

Nothing wrong. All fine and dandy.

IEEE Std 1003.1, 2004 Edition (http://pubs.opengroup.org/onlinepubs/009695399/) says:

3.266 Pathname

A character string that is used to identify a file. In the context of IEEE Std 1003.1-2001, a pathname consists of, at most, {PATH_MAX} bytes, including the terminating null byte. It has an optional beginning slash, followed by zero or more filenames separated by slashes. A pathname may optionally contain one or more trailing slashes. Multiple successive slashes are considered to be the same as one slash.

R.

Disclaimer: I know that Linux is not Unix and not Posix.

Reply by email

Bernd Oppolzer
to Reimar Grabowski

On Windows, too:

C:\>cd \backup\\\///\\\sich

C:\backup\sich>

Only the first backslash has to be (only one) backslash;
no slash allowed.

Kind regards

Bernd

Reimar Grabowski wrote 31 lines
On Wed, 11 Sep 2013 17:37:36 +0200 Jürgen Hestermann wrote: > And double delimiters *are* ambiguous: > Has a (one letter) file name been forgotten or was an additional > delimiter typed (or appended by bad programmed routines)? Ambiguity is not defined by how the symbol came into being. You just look at the symbol as is. So Mattias statement *is* correct. > I am realy astonished that nobody seems to find anything wrong with this. mkdir hans mkdir hans////////pete cd hans ls -la total 12 drwxrwxr-x 3 user user 4096 Sep 11 19:13 . drwxrwxrwt 9 root root 4096 Sep 11 19:13 .. drwxrwxr-x 2 user user 4096 Sep 11 19:13 pete Nothing wrong. All fine and dandy. IEEE Std 1003.1, 2004 Edition (http://pubs.opengroup.org/onlinepubs/009695399/) says: 3.266 Pathname A character string that is used to identify a file. In the context of IEEE Std 1003.1-2001, a pathname consists of, at most, {PATH_MAX} bytes, including the terminating null byte. It has an optional beginning slash, followed by zero or more filenames separated by slashes. A pathname may optionally contain one or more trailing slashes. Multiple successive slashes are considered to be the same as one slash. R. Disclaimer: I know that Linux is not Unix and not Posix.

Reply by email

Jürgen Hestermann
to Reimar Grabowski: On Wed, 11 Sep 2013 17:37:36 +0200 Ambiguity is not defined by how the symbol came into…
Reimar Grabowski wrote
IEEE Std 1003.1, 2004 Edition (http://pubs.opengroup.org/onlinepubs/009695399/) says: 3.266 Pathname A character string that is used to identify a file. In the context of IEEE Std 1003.1-2001, a pathname consists of, at most, {PATH_MAX} bytes, including the terminating null byte. It has an optional beginning slash, followed by zero or more filenames separated by slashes. A pathname may optionally contain one or more trailing slashes. Multiple successive slashes are considered to be the same as one slash.

Ahh, that's the reason:
It's (another) sloppiness in Unix systems and has creeped into Windows from there. I have never seen such things on Windows systems.

Reply by email

Reimar Grabowski
to Jürgen Hestermann
Ahh, that's the reason: It's (another) sloppiness in Unix systems and has creeped into Windows from there.

Unfortunately M$ hasn't implemented it correctly (like always) and enriched the experience with their common inconsistencies. Therefor you have the problems this thread is about only on windows systems and Unix-like systems don't have any .

Reply by email

Henry Vermaak
to Reimar Grabowski
Reimar Grabowski wrote 9 lines
On Thu, 12 Sep 2013 07:10:19 +0200 Jürgen Hestermann wrote: > Ahh, that's the reason: It's (another) sloppiness in Unix systems > and has creeped into Windows from there. Unfortunately M$ hasn't implemented it correctly (like always) and enriched the experience with their common inconsistencies. Therefor you have the problems this thread is about only on windows systems and Unix-like systems don't have any problems at all.

"Those who don't understand Unix are condemned to reinvent it, poorly."

SCNR :)

Reply by email