In rev. #40849 I replaced GetAllFilesMask with AllFilesMask everywhere. GetAllFilesMask is defined in LCL, AllFilesMask is defined in FPC's libs. It was meant to be a cleanup and removal of duplicate code. It seems that it was wrong. AllFilesMask is '*' on Windows, too. GetAllFilesMask returned *.* on Windows.

There is a bug somewhere, IMO it is in FPC. AllFilesMask is related to file operations.
It should not be different between LCL and other code using FPC.

'*:*' is typically used on Windows and FindFirst/FindNext return also files without suffix when it is used.
I just fixed TFileSearcher to match the logic (issue #24306).

Juha

See here:
http://lazarus-ccr.sourceforge.net/docs/rtl/system/allfilesmask.html

Mattias

Mattias Gaertner wrote

See here: http://lazarus-ccr.sourceforge.net/docs/rtl/system/allfilesmask.html

It says:
... On windows and dos based systems, this will be '*.*', while for unix systems, this will be '*'.

There is a bug indeed! Earlier today I booted Windows to check its value there and it is '*'.
I will report it.

Juha

Before I report it, can somebody please check if it is already fixed in FPC trunk.
I use FPC 2.6.2 myself.

Juha

Afaik both * and *.* work on Windows and Dos-with-lfn.

Marco van de Voort wrote

Afaik both * and *.* work on Windows and Dos-with-lfn.

Do you mean the value is correct and only documentation is wrong?

Juha

On Windows using '*' as filemask will find (FindFirst/FindNext) all files, with or without extension.

The problem with TFileSearcher is in the MatchesMask function. This interprets '*.*' as _must_ have a period.
So it will return false on 'foo' , but true on 'foo.'

On Windows, if you use MatchesMask you must remove a dangling '.*' from the mask, if you want to use it for filenames.

Bart

Bart wrote

On Windows using '*' as filemask will find (FindFirst/FindNext) all files, with or without extension.

So does *.* on Windows. That is why the TFileSearcher behavior was changed.

Fortunately it is a rare situation. If needed, test for it can be added.

Juha

Juha Manninen wrote

So does *.* on Windows. That is why the TFileSearcher behavior was changed.

But I think it is wrong.

or ((ASearchMask = '*.*') and (ExtractFileExt(PathInfo.Name) = ''))

Now a Filemaks of ???.* will not find 'foo', but it should on Windows.

I think the best strategy is to remove dangling '.*' from ASearchMask and replace it by '*'

I used something like this in one of my libraries (it works on maskLists, not a single maks, so it's overcomplex for what we need):

procedure MaskListWinToLinuxWildCards(var AMaskList: String; const
ASeparator: Char);
//All we do here is change any mask that ends in '.*' so that it ends in '*'
//since in Linux if a mask ends in '.*', a '.' must be in the
filename, whilst in Windows
//this translates to may or may not have any extension
var
  i: Integer;
  S: String;
  SL: TStringList;
begin
  (*
  if (ASeparator = #0) then
  begin
    exit;
  end;
  *)
  SL := TStringList.Create;
  Try
    SL.Delimiter := ASeparator;
    SL.DelimitedText := AMaskList;
    for i := 0 to SL.Count - 1 do
    begin
      S := SL.Strings[i];
      //A Windows mask that ends in a period ('.') is the same as the
mask ending in '.*'
      if (S[Length(S)] = '.') then S := S + '*';
      //Wildcard end with '.*'  ?
      if (Length(S) > 1) and (S[Length(S)] = '*') and (S[Length(S)-1]
= '.') then
      begin//Replace a wildcard ending in '.*' in one that ends in '*'
if appropriate
        if (S = '.*') then S := '*'
        else
        begin//we know that Length(S) > 2 now...
          if (S[Length(S)-2] in ['?','*']) then
          begin
            System.Delete(S,Length(s),1);
            S[Length(S)] := '*';
            //Now the mask might end in '**' (if it ended in '*.*' before, so
            //if that is the case, we'll remove the last '*'
            if (Length(S) > 1) and (S[Length(S)-1] = '*') then
System.Delete(S,Length(S),1);
          end
          else
          begin
            //at this point we know that S is something like abc.*
            //converting this to abc* is wrong, this would also find
files like abcdef.*
            //we want to find in Linux terms both the file abc as well as abc.*
            //so we add the entry with the filename without the '.*'
            //and leave the current entry as it is
            SL.Add(System.Copy(S,1,Length(S)-2));
          end;
        end;//S  '.*'
        SL.Strings[i]:= S;
      end;
    end;
    AMaskList := SL.DelimitedText;
  Finally
    SL.Free;
  end;
end;

This approach has worked for me on Win and Linux with my backup program.

Bart

Notwithstanding your statement about masks vs. maskLists, I'm amazed. Is there really not some simple variable assignment that can be used to override this in one line of code? This seems to me to be very programmer-unfriendly.

Leslie

Maybe, but must be discussed first, th eproper place to fix this is in the masks unit itself?

But that may break existing programs...

Bart

Bart wrote

Maybe, but must be discussed first, th eproper place to fix this is in the masks unit itself?

AFAICS Delphi MatchesMask behaves the same:
MatchesMask('foo', '*.*') = False.
MatchesMask('foo.', '*.*') = True.

(Can someone test with recent Delphi please?)

So for any file searching function on Windows, this needs adjusting in "userland".

Bart

Bart wrote

So for any file searching function on Windows, this needs adjusting in "userland".

Turns out Windows is a bit odd:

Dir *. will show files without extension, but also files like .ext, but then not files like .test.ext

How are we ever going to emulate that using MatchesMask?

Dir .* will show only files starting with a dot (as expected)

Bart

Bart wrote

Turns out Windows is a bit odd: Dir *. will show files without extension, but also files like .ext, but then not files like .test.ext

If a file starts with a dot, it is *not* considered to separate name and extension on Windows. In this case the dot is part of the name and there is no extension (unless a second dot exists). A file *must* have a name part, only the extension can be omitted (on Windows).

J?rgen Hestermann wrote

If a file starts with a dot, it is *not* considered to separate name and extension on Windows. In this case the dot is part of the name and there is no extension (unless a second dot exists). A file *must* have a name part, only the extension can be omitted (on Windows).

Thanks for explaining that.

I tried to use the PathMatchSpecEx function to see if that would help. (http://msdn.microsoft.com/en-us/library/windows/desktop/bb773728%28v=vs.85%29.aspx)

But, as I later also found on
http://qualapps.blogspot.com/2009/08/pathmatchspec-problems.html , it doesn't solve our problem.

We may need a seperate FileNameMatchesWindowsWildcards function or something alike, to handle these (borderline?) cases.

Bart

Bart wrote

We may need a seperate FileNameMatchesWindowsWildcards function or something alike, to handle these (borderline?) cases.

The problem with all this is, that Microsoft is not consistent to itself. So there is no right or wrong in some cases. It is ambiguous if you try to refer to Microsoft products. Over years there were similar discussions about filter masks in the Ztree Forum (see a recent discussions in http://www.ztw3.com/forum/forum_entry.php?id=115457 and an overview of older discussions in http://www.ztw3.com/forum/forum_entry.php?id=115501 ). We always tried to find a "correct" interpretion of how Microsoft wants it to be done. But this does not exist.

The only solution I see is to be consistent within Free Pascal and use an internal routine to define a (logical and pratical) syntax of file masks. This way it is also possible to have the same mask for different OS's because all Free Pascal programmers can rely on the same routine to find matches.

J?rgen Hestermann wrote

The problem with all this is, that Microsoft is not consistent to itself.

:-)

But you must consider the end user.
If on Windows someone types *.* in my program, he expects to find all files, but on Linux, the user will expect to find only files that have a dot in it's name somewhere.

Bart

If on Windows someone types *.* in my program, he expects to find all files, but on Linux, the user will expect to find only files that have a dot in it's name somewhere.

+1

The same for the coder, who wants all files found when he uses the platform-specific mask constant.

Where's the problem?

DoDi

Bart wrote

But you must consider the end user. If on Windows someone types *.* in my program, he expects to find all files, but on Linux, the user will expect to find only files that have a dot in it's name somewhere.

If you just want to pass through the mask to the underlying OS then you don't need to worry about it. Just let the OS API do it. Someone on Linux will get what he expects on Linux and a Windows user will get what he expects on Windows (so nothing predictable ;-)). But if you want a user of your program to get the same results for both versions you need to unify them. It depends on the expectations.

Bart wrote

But you must consider the end user. If on Windows someone types *.* in my program, he expects to find all files,

and yet those average joe windows users do not and are very unaware that they are not finding all files...

but on Linux, the user will expect to find only files that have a dot in it's name somewhere.

So what *would* the average Joe type and how is that handled right now?

I think there are different cases to look at:

1.) The Free Pascal constant AllFilesMask:
This should be "*" on Linux and "*.*" on Windows (and maybe even different on other OS'es). A programmer who uses this constant should rely on finding all files (the mask should match all files).

2.) A filter mask that is typed in by a user of a program: This masked should be passed through to the OS API functions without change. The user should know what files match the mask for the OS he is currently using.

3.) A filter mask syntax defined by the programmer: Sometimes it may be desireable to unify filter masks in a program that exists under different platforms. If the average user does not know about mask standards for each OS the programmer may define his own mask syntax which works the same on all OS'es. For example, there is a permanent disput (and different behaviour) about the question mark and whether it should match exactly one character (which I find logical) or one or less (which is implemented in many but not all OS shell tools).

So in 1.) the programmer decides what the user will find, in 2.) the user itself decides and in 3.) it's a mix.

they would most likely use *.* because that's all they are used to using... they would remain oblivious to the files they miss with that mask, though... unless they are explicitly aware of one/some in particular they are looking for...

as they say, ignorance is bliss...

they would most likely use *.* because that's all they are used to using... they would remain oblivious to the files they miss with that mask

Which files would be missed with this mask? IMO all files fit this mask (at least for Windows).

J?rgen Hestermann wrote

Which files would be missed with this mask? IMO all files fit this mask (at least for Windows).

In the context of the original bugreport, they would miss all files without an extension, beause the *.* isn't handed to FindFirst, but to the MatchesMask that filters FindFirst with * as a mask.

MatchesMask('foo','*.*') -> False
TMask.Matches matches filenames in a Unix way (as does Delphi's TMask).

Bart

So (int the context of the bug report) the programmer has no chance to do it right and all his knowledge about OS filter masks is of no use because an intermedate routine silently changes his mask? This is more than strange. What is the reason for doing this?

quick testing form the command line of w2k shows that you are correct... what some GUI tool does is a different matter, though, and i have seen some that simply do not find all the files with certain masks... i agree that there should be consistency and i get the same results in my command line tests of

dir
dir *
dir *.*

where each return all the same files in the same order...

i also tried

dir .*

which gave me all the files starting with a '.' whether they had more dots or not...

eg:
C:\Documents and Settings\username>dir .*
Volume in drive C is Local Disk
Volume Serial Number is 9CAB-2377

Directory of C:\Documents and Settings\username

08/09/2012 22:18 <DIR> .
08/09/2012 22:18 <DIR> ..
01/19/2012 00:10 7,569 .bash_history 01/17/2012 19:01 103 .gitconfig
03/31/2009 18:28 218 .recently-used.xbel 01/17/2012 11:28 <DIR> .ssh
3 File(s) 7,890 bytes
3 Dir(s) 6,939,553,792 bytes free

note that the 3rd file has another dot in its name...

J?rgen Hestermann wrote

So (int the context of the bug report) the programmer has no chance to do it right and all his knowledge about OS filter masks is of no use because an intermedate routine silently changes his mask? This is more than strange. What is the reason for doing this?

No, the mask does not change but it gets interpreted in the Unix way (which is more logical but it is irrelevant here). We must adjust the interpretation for the OS, Windows that is.

Juha

J?rgen Hestermann wrote

So (int the context of the bug report) the programmer has no chance to do it right and all his knowledge about OS filter masks is of no use because an intermedate routine silently changes his mask? This is more than strange. What is the reason for doing this?

Not exactly.
User supplies SomeMask
FileSearcher does FF/FN with AllFilesMask and filters output with SomeMask. It currently is the only way to do a filesearch with multiple masks ('*.jpg;*.png;*.bmp').

It fails in some conditions because of the workings on TMask.Matches.

It can be partially fixed by altering the supplied mask, but it still fails in borderline conditions like 'foo*.'

It can only be "really" fixed if we have a different Matches function. But even then, the results may differ across Windows versions (as explained earlier).

For the time being I suggest implementing the partial fix and discuss if we want/need a Matches function that behaves the same as "Dir" (and FF/FN) does.

Bart

ewww... why not break the masks out into three (for the given example) separate FF/FN runs and stuff the results into the same TStringList (removing duplicates of course) or whatever the results are returned in?

Bart wrote

User supplies SomeMask FileSearcher does FF/FN with AllFilesMask and filters output with SomeMask.

But wasn't the original problem that AllFilesMask was wrong on Windows? So no matter what SomeMask would be, some files were missing.

It currently is the only way to do a filesearch with multiple masks ('*.jpg;*.png;*.bmp').

But this means that the underlying OS will never see the mask that the programmer has given. Mask matching would be completely done by some FP internal routines. So no matter what I know about OS specific mask handling, it would not apply here. I need to know FP specific mask handling.

It can only be "really" fixed if we have a different Matches function. But even then, the results may differ across Windows versions (as explained earlier).

I don't understand. Before you wrote that FP is using the AllFilesMask and then filters further internaly. Now you say that the outcome would be different on different OS's.

J?rgen Hestermann wrote

But wasn't the original problem that AllFilesMask was wrong on Windows?

No.
Supplying either '*.*' or '*' to FindFirst() function on Windows will correctly list all files.
The prblem was that the mask supplied to TMaskList.Matches() did not give the expected results, because Matches() treats at the given mask like Linux does (as apparently does Delphi).

But this means that the underlying OS will never see the mask that the programmer has given.

You cannot call FindFirst() with a masklist ('*.jpg;*.bmp') so you must either filter yourself or make repeated calls to FindFirst with each and every mask of the makslist.

No, I did not say that.
Windows itself behaves different across versions, in the way it treats borderline cases (like 'foo.'), when supplying it to the dir command (or FindFirst function).

I implemented a new MatchesWindowsMask function in TMask that ATM seems to mimic the behaviour of "Dir", and I made TFileSearcher use that on if you are on Windows.
It now handles masks like 'foo*.*', 'foo.' and 'foo.*' etc. as exected. (See r40970 and r40973)

This implementation of course does not depend on Windows version, because we programmed it ourselves.

In future, however, it may differ form the way "Dir" treats wildcards, since MS has not documented how borderline cases should be treated in the frist place.

Bart

And I've now checked why this is the case:

If you look at the implementation of FindFirstFileExW (which is in the end called by all other FindFirstFile* functions) in ReactOS ( http://svn.reactos.org/svn/reactos/trunk/reactos/dll/win32/kernel32/client/file/find.c?revision=58833&view=markup ) you'll see at line 822 the following code:

=== code begin ===

/* Change pattern: "*.*" --> "*" */
if (FilePattern.Length == 6 &&
RtlCompareMemory(FilePattern.Buffer, L"*.*", 6) == 6) {

     FilePattern.Length = 2;

}

=== code end ===

So this means it does not matter whether you pass '*.*' or '*' to FindFirstFile*, because the Win32 subsystem will adjust the mask anyway (FPC's FindFirst internally uses FindFirstFileA on Windows).

Regards,
Sven