Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / lazarus / 2021-02

unit Masks vs. unit FPMasks

39 messages · 4 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Juha Manninen

Hello

LazUtils has unit Masks with classes TMask and TMaskList. FPC's packages/fpindexer has unit FPMasks also with classes TMask and TMaskList.
A comment in FPMasks says "Moved here from LCL".
Revision control shows it was added 9 years ago in 2012. Since 2.5 years ago it supports Unicode by using UTF8string for all strings.

LCL has a related MaskEdit. It was first added in 2002 by Mattias with comment
"added TMaskEdit from Tony"
I don't know who Tony is.
Unit Masks was first added to LCL in 2007 by tombo with comment "LCL: implemented TMask, MatchesMask, added Masks docs" I don't know who Tombo is either.
In 2011 Masks was moved to LazUtils by Felipe.
Masks and MaskEdit have some identical code. IMO MaskEdit should reuse some code from Masks.
MaskEdit is maintained by Bart but he didn't know details of the history.

Masks in LazUtils has a slow implementation.
I planned to optimize it but now I realize we may have overlapping code. Q: Are Masks (LazUtils) and FPMasks (fpindexer) compatible? If they are, we should dump the LazUtils Masks and use code from FPC's libs.

Regards,
Juha
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20210223/871bd6fd/attachment.html>

Reply by email

Michael Van Canneyt
to Juha Manninen
Juha Manninen wrote 25 lines
Hello LazUtils has unit Masks with classes TMask and TMaskList. FPC's packages/fpindexer has unit FPMasks also with classes TMask and TMaskList. A comment in FPMasks says "Moved here from LCL". Revision control shows it was added 9 years ago in 2012. Since 2.5 years ago it supports Unicode by using UTF8string for all strings. LCL has a related MaskEdit. It was first added in 2002 by Mattias with comment "added TMaskEdit from Tony" I don't know who Tony is. Unit Masks was first added to LCL in 2007 by tombo with comment "LCL: implemented TMask, MatchesMask, added Masks docs" I don't know who Tombo is either. In 2011 Masks was moved to LazUtils by Felipe. Masks and MaskEdit have some identical code. IMO MaskEdit should reuse some code from Masks. MaskEdit is maintained by Bart but he didn't know details of the history. Masks in LazUtils has a slow implementation. I planned to optimize it but now I realize we may have overlapping code. Q: Are Masks (LazUtils) and FPMasks (fpindexer) compatible? If they are, we should dump the LazUtils Masks and use code from FPC's libs.

Since it comes from Lazarus in the first place, they are supposed to be compatible, yes. If not, then the unit in FPC can be updated to add missing things.

Michael.

Reply by email

Juha Manninen
to Michael Van Canneyt

A related thing. I am confused with different mask classes. Embarcadero docs for TMask
http://docwiki.embarcadero.com/Libraries/Sydney/en/System.Masks.TMask says:
"*Note:* Do not confuse TMask with the EditMask of a field or masked edit object. While both are used for comparing strings to a symbolic description of valid values, the special mask symbols and matching rules are completely different."

Is the "*masked edit object*" the same as MaskEdit? Is the syntax really different?

Juha
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20210223/50bed859/attachment-0001.html>

Reply by email

Juha Manninen
to Michael Van Canneyt: Since it comes from Lazarus in the first place, they are supposed to be compatible, yes.…

On Tue, Feb 23, 2021 at 12:00 PM Michael Van Canneyt via lazarus <

Show quoted text 6 lines
wrote: Since it comes from Lazarus in the first place, they are supposed to be compatible, yes. If not, then the unit in FPC can be updated to add missing things.

Oops, now I understand that FPMasks does not really support Unicode. Type UTF8String only converts encoding automatically when assigning values. There is no code to identify codepoints.
The LazUtils Masks iterates codepoints, although in a very slow way. Has anybody tested FPMasks with multibyte-codepoints? I guess it may not work correctly.

BTW, the UTF8String change is not in FPC 3.0.4 which we still must support. Even if FPC libs get new code that can be used in Lazarus, It takes many years before we can use it due to the slow release cycle. I hope FPC 3.2.2 comes out soon.

Juha
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20210223/b030dd0b/attachment.html>

Reply by email

Juha Manninen
to Juha Manninen
PM Juha Manninen wrote 7 lines
Oops, now I understand that FPMasks does not really support Unicode. Type UTF8String only converts encoding automatically when assigning values. There is no code to identify codepoints. The LazUtils Masks iterates codepoints, although in a very slow way. Has anybody tested FPMasks with multibyte-codepoints? I guess it may not work correctly.

I am not sure if iterating multibyte codepoints is even necessary. I must create a unit test.
Does anybody have a list of use cases or some example code? The Embarcadero documentation is very plain.
It is difficult to find comprehensive examples. I must confess I have not used TMask or TMaskList myself.

Juha
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20210223/8ea4c6af/attachment.html>

Reply by email

Bart
to Juha Manninen: Hello LazUtils has unit Masks with classes TMask and TMaskList. FPC's packages/fpindexer…
AM Juha Manninen wrote
LazUtils has unit Masks with classes TMask and TMaskList. FPC's packages/fpindexer has unit FPMasks also with classes TMask and TMaskList.

MaskEdit is a LCL control and hence has nothing to do in fpc. I have in the past copied bits of TMaskEdit to a related fpc unit (cannot remeber exactly which), so that they work the same (setting and removing masks IIRC), with the exception that TMaskEdit does this on UTF-8, while the fpc unit assumes 1-byte ANSI encoding (so it won't work on UTF-8).
Given the release cycle of fpc, I would strongly advise to have this code on "our side" (Lazarus), so bugs can be eliminated much faster.

The Masks unit is not related to TMaskEdit.
For the Masks unit: the same UTF-8 problems exist with it's fpc counterpart IIRC (did not study the code recently). So, I would like to keep it "here" as well.

Bart

Reply by email

Juha Manninen
to Bart

On Tue, Feb 23, 2021 at 2:54 PM Bart via lazarus <

wrote: MaskEdit is a LCL control and hence has nothing to do in fpc.

I am not suggesting to move MaskEdit to FPC libs obviously.

I have in the past copied bits of TMaskEdit to a related fpc unit

Show quoted text 9 lines
(cannot remeber exactly which), so that they work the same (setting and removing masks IIRC), with the exception that TMaskEdit does this on UTF-8, while the fpc unit assumes 1-byte ANSI encoding (so it won't work on UTF-8). Given the release cycle of fpc, I would strongly advise to have this code on "our side" (Lazarus), so bugs can be eliminated much faster. The Masks unit is not related to TMaskEdit.

Does it mean the mask syntax is different? I found this:

http://docwiki.embarcadero.com/Libraries/Sydney/en/System.MaskUtils.TEditMask Is TEditMask the same as TMaskEdit? Or is TEditMask used for TMask? Can you please explain it so that I understand?
Is it all documented somewhere?

For the Masks unit: the same UTF-8 problems exist with it's fpc

counterpart IIRC (did not study the code recently). So, I would like to keep it "here" as well.

Yes, I realized it must stay in LazUtils. Iterating codepoints is needed. FPMasks is good for ASCII only despite having UTF8String type. Does Michael have a plan for that unit?

Fortunately I found a unit test for Masks unit under LCL tests directory. I moved it under LazUtils in r64653.
I will add tests for multibyte codepoint text. Then optimizing TMask will be safe. Nothing can go wrong... :)

Juha
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20210223/d0e288ca/attachment.html>

Reply by email

José Mejuto
to Juha Manninen: On Tue, Feb 23, 2021 at 12:00 PM Michael Van Canneyt via lazarus < Oops, now I understand…
Juha Manninen wrote 5 lines
Masks in LazUtils has a slow implementation. I planned to optimize it but now I realize we may have overlapping code. Q: Are Masks (LazUtils) and FPMasks (fpindexer) compatible? If they are, we should dump the LazUtils Masks and use code from

FPC's libs.
Hello,

fpMasks and Masks are quite the same from my point of view, in fact fpMasks (if my memory serves me) have various problems with UTF8 handling.

I'll send you (direct mail) my TMaskAnsi, TMaskUTF8 and TMaskUnicode so you can do whatever you want with them :-)

Based in RTL's encoding you can alias TMask to any of them.

Reply by email

Bart
to Juha Manninen: On Tue, Feb 23, 2021 at 2:54 PM Bart via lazarus < I am not suggesting to move MaskEdit…
PM Juha Manninen wrote
Does it mean the mask syntax is different? I found this: http://docwiki.embarcadero.com/Libraries/Sydney/en/System.MaskUtils.TEditMask Is TEditMask the same as TMaskEdit? Or is TEditMask used for TMask? Can you please explain it so that I understand? Is it all documented somewhere?

I have copied in the past some of the TMaskEdit logic and methods to the MaskUtils unit.
Things like SplitEditMask etc.
So, basically the behave the same (on 1-byte ANSI strings).

The code in MaskEdit unit is the father of the code in MaskUtils.

From a Lazarus point of view, the code in MaskEdit is leading.

Fpc to some extent follows what we do here.

I do NOT want to use the code from MaskUtils in the MaskEdit unit, for reasons I explained in my previous code.

Bart

Reply by email

Juha Manninen
to Bart

On Tue, Feb 23, 2021 at 7:38 PM Bart via lazarus <

Show quoted text 5 lines
wrote: I have copied in the past some of the TMaskEdit logic and methods to the MaskUtils unit.

Ok, the TEditMask thing was in MaskUtils. I didn't pay attention. How about TMask? Does it have the same syntax as TMaskEdit or are they different?

Juha
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20210223/7dffc9ab/attachment-0001.html>

Reply by email

Bart
to Juha Manninen
PM Juha Manninen wrote
How about TMask? Does it have the same syntax as TMaskEdit or are they different?

TMask (unit masks) deals with masks with wildcards (*,? and sets of single byte chars).
It is mainly used for matching filenames (similar to the Path supplied to FindFirst).

TMaskEdit gives you the possibility to constrain user input to almost anything you like,
Can be used for e.g. ZIP codes, only numbers etc.
It also handles pasting in the control.
If the text the user enters does not match the specified mask when the control looses focus (or user presses enter) an exception is raised. In the Delphi 1,2,3 years I used it to force numeric input (integers and floats), for which we nowadays have better controls. There is a data aware counterpart as well.

As you have pointed out before, the GetCodePoint function in the Masks unit needs overhoaling.
(It is the same as in TMaskEdit, but that only reacts to user input with strings <=255 chars, so speed is not required there: I'ld love to see someone typing faster that the code in TMaskEdit calculates what needs to be done.)

Bart

Reply by email

Juha Manninen
to Bart

On Wed, Feb 24, 2021 at 12:08 AM Bart via lazarus <

Show quoted text 11 lines
wrote: TMask (unit masks) deals with masks with wildcards (*,? and sets of single byte chars). It is mainly used for matching filenames (similar to the Path supplied to FindFirst). TMaskEdit gives you the possibility to constrain user input to almost anything you like, Can be used for e.g. ZIP codes, only numbers etc.

TMask also supports ranges and sets. See the unit test. Eg. '[a-b]', '[!a-b]', '[abc]', '[0-9]'

Now I found documentation for TCustomMaskEdit.EditMask. It explains the syntax and it looks like the MaskUtils syntax.
It was documented, good! I missed it earlier.
I know filename wildcards and I know regular expressions. Now learning this Mask thingy...

As you have pointed out before, the GetCodePoint function in the Masks

unit needs overhoaling.

It is much worse than that!
Yes, GetCodePoint does its own nested loops and useless copies. But then it and other UTF8...() functions are called inside a loop, effectively causing many nested loops.
The scalability is maybe O(n^3) or O(n^4).
José Mejuto's Mask unit looks promising. He mentioned in a private mail (which should be public IMO, no deep secrets there) that a pattern

* "*something*to*write*here*""which with current mask it takes a lot of time to be processed. If matchable string is of more than 200 chars long it could take seconds to be resolved. My classes are typically O(n)."* Many seconds in a modern computer is a lot.

(It is the same as in TMaskEdit, but that only reacts to user input

with strings <=255 chars, so speed is not required there: I'ld love to see someone typing faster that the code in TMaskEdit calculates what needs to be done.)

True, but the code should be cleaned anyway and maybe reuse some other code. Code has aesthetic values, too.

Juha
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20210224/a192df4c/attachment.html>

Reply by email

Bart
to Juha Manninen
AM Juha Manninen wrote
> TMask (unit masks) deals with masks with wildcards (*,? and sets of > single byte chars).

...

TMask also supports ranges and sets. See the unit test. Eg. '[a-b]', '[!a-b]', '[abc]', '[0-9]'

By single byte chars I meant ASCII only.
You cannot have '[ä..ë]' in a TMask (a constraint that is a side effect of the implementation, but this would be sort of an undefined range as well).

Now I found documentation for TCustomMaskEdit.EditMask. It explains the syntax

It is the soucecode (has been there from the beginning) and in the wiki.

and it looks like the MaskUtils syntax.

Again: it's the other way around: the code of MaskUtils looks like the code of MaskEdit.

Show quoted text 12 lines
> As you have pointed out before, the GetCodePoint function in the Masks > unit needs overhoaling. It is much worse than that! Yes, GetCodePoint does its own nested loops and useless copies. But then it and other UTF8...() functions are called inside a loop, effectively causing many nested loops. The scalability is maybe O(n^3) or O(n^4). José Mejuto's Mask unit looks promising. He mentioned in a private mail (which should be public IMO, no deep secrets there) that a pattern "*something*to*write*here*" "which with current mask it takes a lot of time to be processed. If matchable string is of more than 200 chars long it could take seconds to be resolved. My classes are typically O(n)." Many seconds in a modern computer is a lot.

I use this Mask unit extensively for my backup program. Resolving TMaskMatches even for long strings and mask take orders of magnitude less time then accessing the file (just opening it).

Of course that is NOT a reason not to improve it: O(n^4) is just terrible. Mind you, the GetCodePoint/SetCodePoint originally was just a quick (as in: simple, stupis, short code) hack to get the UTF8 functionality in MaskEdit.
After changing all SomeString[i] to either GetCodePoint(SomeString,i) or SetCodePoint(SomeString,i, ACodePoint) the MaskEdit unit was UTF8 a major rewrite (which increases the cange of breaking compatibility).

Mind you that the first implementation of GetCodePoint was even more "simple", it simply called Utf8Copy(SomeString, i, 1)...

So, yes re-implement GetCodePoint/SetCodePoint or the internal logic of the Masks unit by all means, but as far as the MaskEdit unit is concerned the function signature should not change. There is no need for a major rewrite of that unit: it deals with user input and even if you make it 100 times slower as it is now, user will not notice it.

Bart

Reply by email

Juha Manninen
to Bart

I will not touch MaskEdit. Don't worry.

On Wed, Feb 24, 2021 at 11:03 AM Bart via lazarus <

Show quoted text 5 lines
wrote: Without a major rewrite (which increases the cange of breaking compatibility).

José Mejuto's code is a major rewrite for Masks. It supports Unicode in masks, too.
I try to make it compatible by changing some class and method names, and then run the unit tests.
Comprehensive unit tests are a way to prevent breaking things. Please everybody provide more test cases. The project is in components/lazutils/test.
There are no tests for MatchesWindowsMask() yet.

Juha
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20210224/74bf7cff/attachment.html>

Reply by email

José Mejuto
to Juha Manninen
Juha Manninen wrote
José Mejuto's code is a major rewrite for Masks. It supports Unicode in masks, too. I try to make it compatible by changing some class and method names, and then run the unit tests.

Hello,

In my code there is non 100% unicode compatibility when using the "CaseInsensitive" mode as as it uses lowercase mask and lowercase string to perform the test which is wrong by definition but I was unable to find a method to test codepoints case insensitive without pulling in big unicode tables.

I was thinking in import the NTFS (the filesystem) case comparison tables which are 128 KB "only".

Comprehensive unit tests are a way to prevent breaking things.

And also define if a compatibility break is a bug in the new code or in the old code. In example my mask supports (there is a define to disable) "[z-a]" converting it to "[a-z]" which is a compatibility break. Also there is the support (also can be disabled) for the mask "[?]" which is the counterpart for "*" but with one char position.

There are no tests for MatchesWindowsMask() yet.

Who defines which are right and which are wrong ? There is no official DOS/Windows mask strategies, only inherited behaviour since CP/M. Maybe the behaviour of CMD.EXE ? Which version of CMD.EXE ?

Reply by email

Bart
to Bart: On Wed, Feb 24, 2021 at 9:11 AM Juha Manninen via lazarus ... By single byte chars I…
AM Bart wrote
Of course that is NOT a reason not to improve it: O(n^4) is just terrible.

To put this discussion in a little perspective.
Given a string S (UTF8 encoded) with Utf8Length=1000. GetCodePoint(S,1000) on my laptop takes 0.00439 msecs to perform. So 10 thousand of these lookups cost appr. 44 ms.
This is kind of a worst case scenario.
You are not very likely to have strings that long in TMask.MatchesMask (and certainly not in a MaskEdit).

Bart

Reply by email

Bart
to José Mejuto: In my code there is non 100% unicode compatibility when using the "CaseInsensitive" mode…
AM José Mejuto wrote
In my code there is non 100% unicode compatibility when using the "CaseInsensitive" mode as as it uses lowercase mask and lowercase string to perform the test which is wrong by definition

Currently Masks unit does the same.

Show quoted text 5 lines
And also define if a compatibility break is a bug in the new code or in the old code. In example my mask supports (there is a define to disable) "[z-a]" converting it to "[a-z]" which is a compatibility break. Also there is the support (also can be disabled) for the mask "[?]" which is the counterpart for "*" but with one char position.

Current behaviour of sets and wildcards should not be changed by default. E.g. TShellTreeView and TShellListView us the Masks unit to populate the tree/view.
An option to have the behaviour you described would be OK, the TMaskOption can be extended for that.

Sometimes I wish we would migrate to using UnicodeString by default. It would make life a bit easier.
(And yes I know you would have to deal with composed characters (grapheme defined by more than 1 16-bit word)).

> There are no tests for MatchesWindowsMask() yet.

I tested that extensively on my machine with all scenarios I could think of. But others most likely can think of scenarios I did not test. It was based on current behaviour of Windows NT platform (Win7 at the time to be precise).

Who defines which are right and which are wrong ?

Well, I did ;-)
(Nobody else bothered at the time, and nobody complained either.)

Bart

Reply by email

José Mejuto
to Bart: To put this discussion in a little perspective. Given a string S (UTF8 encoded) with…
Bart wrote 11 lines
On Wed, Feb 24, 2021 at 10:02 AM Bart wrote: > Of course that is NOT a reason not to improve it: O(n^4) is just terrible. To put this discussion in a little perspective. Given a string S (UTF8 encoded) with Utf8Length=1000. GetCodePoint(S,1000) on my laptop takes 0.00439 msecs to perform. So 10 thousand of these lookups cost appr. 44 ms. This is kind of a worst case scenario. You are not very likely to have strings that long in TMask.MatchesMask (and certainly not in a MaskEdit).

Hello,

The worst case scenario is not based in GetCodePoint time, is in the TMask logic when "*" is found, it enters in recursive scan from that point, if it founds a new "*" it recurses again and if it finally fails in comparison it rolls back to the first "*". So specially crafted masks makes it recurse a lot.

String:='This is a test string';
Mask:='*T*h*s*n*x';

Of course this is not a day by day use ;-)

Note: Just to put in context, my "explore" in the TMask world started when writing my NTFS filesystem reader, when all file names are read (400,000) I can search for them using masks. When compiled in fpc (Lazarus) the "*.txx" search takes 1-2 seconds (not measured) and when compiled with Delphi it takes +/- 0.3 seconds, so I stated to write my own TMask.

Reply by email

Juha Manninen
to Bart: On Wed, Feb 24, 2021 at 11:22 AM José Mejuto via lazarus Currently Masks unit does the…

On Wed, Feb 24, 2021 at 1:00 PM Bart via lazarus <

Show quoted text 6 lines
wrote: > > There are no tests for MatchesWindowsMask() yet. I tested that extensively on my machine with all scenarios I could think of.

Please add your tests to the project I mentioned.

Juha
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20210224/1a4c0397/attachment.html>

Reply by email

José Mejuto
to Bart: On Wed, Feb 24, 2021 at 11:22 AM José Mejuto via lazarus Currently Masks unit does the…

El 24/02/2021 a las 11:58, Bart via lazarus escribió:

Hello,

Show quoted text 5 lines
> In my code there is non 100% unicode compatibility when using the > "CaseInsensitive" mode as as it uses lowercase mask and lowercase string > to perform the test which is wrong by definition Currently Masks unit does the same.

Yes, but in example in my case I can not success test mask "ä*" vs string "Ä*" because "Ä" is not lowercased to "ä" (Windows 7).

Sometimes I wish we would migrate to using UnicodeString by default. It would make life a bit easier. (And yes I know you would have to deal with composed characters (grapheme defined by more than 1 16-bit word)).

That's a can of worms! UTF8 forces you to write "correct code" (at least try it) for any character >127, with UnicodeString you get the false apparence that everything magically works until everything cracks when a string with surrogate pairs come in play :-) and ALL you text handling must be rewritten, and most of them completly rewritten.

Show quoted text 8 lines
>> There are no tests for MatchesWindowsMask() yet. I tested that extensively on my machine with all scenarios I could think of. But others most likely can think of scenarios I did not test. It was based on current behaviour of Windows NT platform (Win7 at the time to be precise). > Who defines which are right and which are wrong ? Well, I did ;-) (Nobody else bothered at the time, and nobody complained either.)

And mostly will not as almost everything matches the expected behaviour for an user, like typical "*.txt" but there are some non supported cases like:

Filename:='test.txt'
Mask:='test??.txt?'
Match must be true

This is the doc from my code about Windows matching, Quirks can be enabled or disabled for compatibility:

----------------8<----------------------------8<--------------------- Windows mask works in a different mode than regular mask, it has too many quirks and corner cases inherited from CP/M, then adapted to DOS (8.3) filenames and adapted again for long file names.

Anyth?ng.abc = "?" matches exactly 1 char
Anyth*ng.abc = "*" matches 0 or more of chars

------- Quirks -------

--eWindowsQuirk_AnyExtension
Anything*.* = ".*" is removed.

--eWindowsQuirk_FilenameEnd
Anything??.abc = "?" matches 1 or 0 chars (except '.') (Not the same as "Anything*.abc", but the same
as regex "Anything.{0,2}\.abc")
Internally converted to "Anything[??].abc"

--eWindowsQuirk_Extension3More
Anything.abc = Matches "Anything.abc" but also
"Anything.abc*" (3 char extension)
Anything.ab = Matches "Anything.ab" and never
"anything.abcd"

--eWindowsQuirk_EmptyIsAny
"" = Empty string matches anything "*"

--eWindowsQuirk_AllByExtension (Not in use anymore) .abc = Runs as "*.abc"

--eWindowsQuirk_NoExtension
Anything*. = Matches "Anything*" without extension

----------------8<----------------------------8<---------------------

Reply by email

Juha Manninen
to José Mejuto: In my code there is non 100% unicode compatibility when using the "CaseInsensitive" mode…

On Wed, Feb 24, 2021 at 12:22 PM José Mejuto via lazarus <

Show quoted text 11 lines
wrote: In my code there is non 100% unicode compatibility when using the "CaseInsensitive" mode as as it uses lowercase mask and lowercase string to perform the test which is wrong by definition but I was unable to find a method to test codepoints case insensitive without pulling in big unicode tables. I was thinking in import the NTFS (the filesystem) case comparison tables which are 128 KB "only".

That is not necessary.
LazUTF8 has functions like UTF8CompareText(), UTF8CompareTextP() and the latest UTF8CompareLatinTextFast().
UTF8CompareLatinTextFast supports full Unicode but is optimized for mostly Latin text.
We should add a PChar version UTF8CompareLatinTextFastP() and use it in your mask code.

Show quoted text 5 lines
Comprehensive unit tests are a way to prevent breaking things. And also define if a compatibility break is a bug in the new code or in the old code. In example my mask supports (there is a define to disable) "[z-a]" converting it to "[a-z]" which is a compatibility break.

Your code does not compile when RANGES_AUTOREVERSE is not defined. cMask is not found.
The reverse logic can be enabled by default. It does not break anybody's masks as I understand it. Earlier it was an error, now it does something sensible.

Also there is the support (also can be disabled) for the mask "[?]"

which is the counterpart for "*" but with one char position.

Where did you get this "[?]" syntax? There must be a reference documentation somewhere but I have not seen it.
What is the difference between "?" and "[?]" ?

On Wed, Feb 24, 2021 at 1:28 PM José Mejuto via lazarus <

Show quoted text 13 lines
wrote: > Sometimes I wish we would migrate to using UnicodeString by default. > It would make life a bit easier. > (And yes I know you would have to deal with composed characters > (grapheme defined by more than 1 16-bit word)). That's a can of worms! UTF8 forces you to write "correct code" (at least try it) for any character >127, with UnicodeString you get the false apparence that everything magically works until everything cracks when a string with surrogate pairs come in play :-) and ALL you text handling must be rewritten, and most of them completly rewritten.

Exactly. UnicodeString uses UTF-16 which is also a variable length encoding. The same rules should be applied but often they are not. There is plenty of sloppy UTF-16 code out there.
Writing proper code UTF-8 is not difficult once you wrap your mind around the concept. There is a learning curve, true. I also scratched my head for some time when studying it.

Juha
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20210224/3490307a/attachment-0001.html>

Reply by email

José Mejuto
to Juha Manninen

El 24/02/2021 a las 13:31, Juha Manninen via lazarus escribió:

Hello,

I was thinking in import the NTFS (the filesystem) case comparison tables which are 128 KB "only". That is not necessary. LazUTF8 has functions like UTF8CompareText(), UTF8CompareTextP() and the

The code was originally designed to be fpc only, not Lazarus libs (even when compatibles with console apps).

Show quoted text 5 lines
Your code does not compile when RANGES_AUTOREVERSE is not defined. cMask is not found. The reverse logic can be enabled by default. It does not break anybody's masks as I understand it. Earlier it was an error, now it does something sensible.

Probably cMask falls by error inside one $IFDEF, I'll check but must be a trivial error.

Show quoted text 5 lines
Also there is the support (also can be disabled) for the mask "[?]" which is the counterpart for "*" but with one char position. Where did you get this "[?]" syntax? There must be a reference documentation somewhere but I have not seen it. What is the difference between "?" and "[?]" ?

Wildcards except "*" and "?" do not have any kind of standard reference, some use "[]" others "{}", some negate with "!", others with "^", some ranges using "-" others "<", there is a jungle of widcards.

In the develop of the Windows matching I found that there is no option for zero or one char which is needed in the Windows matching, so started to googling about and find some that uses the "?" inside brackets to notate the zero or one and it is compatible with the idea of ranges and charsets, so used it.

The difference is:

? = 1 char
[?] = 1 or 0 chars.

?? = 2 chars
[??] = 0,1 or 2 chars

In the DOS/Windows world (behaviour, not syntax) it is used like:

--------------------------
File name: "test.txta"
DOS mask: "test.txt?"
Match: True

File name: "test.txt"
DOS mask: "test.txt?"
Match: True
--------------------------

Also in the code I have added the escape character "\" (also selectable use or not) to allow string matching of stars, question mark, brackets, etc. characters. As other wildcards there is no standard character and/or behaviour.

Have a nice day.

Reply by email

Juha Manninen
to Juha Manninen: On Wed, Feb 24, 2021 at 12:22 PM José Mejuto via lazarus < That is not necessary.…

MaskList is used in TCustomShellTreeView.GetFilesInDir(). It sets a mask option *moDisableSets*. Why are sets disabled? Is it to stay compatible with FindFirstUTF8() syntax?
I don't fully understand the comment. Why would EConvertError be raised?

      //Disable the use of sets in the masklist.
      //this behaviour would be incompatible with the situation if no
MaskList was used
      //and it would break backwards compatibilty and could raise
unexpected EConvertError where it did not in the past.
      //If you need sets in the MaskList, use the OnAddItem event for that.
(BB)
      MaskOptions := [moDisableSets];
      {$ifdef NotLiteralFilenames}
      if (ACaseSensitivity = mcsCaseSensitive) then
        MaskOptions := [moDisableSets, moCaseSensitive];
      {$else}
      if (ACaseSensitivity  mcsCaseInsensitive) then
        MaskOptions := [moDisableSets, moCaseSensitive];
      {$endif}
      MaskList := TMaskList.Create(MaskStr, ';', MaskOptions);  //False by
default

Anyway I don't see why a useful Mask syntax should be disabled. Can this be solved by other means?
Can José's mask do a similar thing?

Juha
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20210224/2beb9b1b/attachment.html>

Reply by email

Bart
to Juha Manninen: On Wed, Feb 24, 2021 at 1:00 PM Bart via lazarus < Please add your tests to the project I…
PM Juha Manninen wrote
> I tested that extensively on my machine with all scenarios I could think of. Please add your tests to the project I mentioned.

I did that when writing the code.
I never saved the tested masks I used anywhere AFAIK.

The comments in the code state what scenario's I have come up with. The scenarios described were made to behave like the dir command of windows.

Bart

Reply by email

Juha Manninen
to José Mejuto: The worst case scenario is not based in GetCodePoint time, is in the TMask logic when "*"…

On Wed, Feb 24, 2021 at 1:05 PM José Mejuto via lazarus <

Show quoted text 9 lines
wrote: Note: Just to put in context, my "explore" in the TMask world started when writing my NTFS filesystem reader, when all file names are read (400,000) I can search for them using masks. When compiled in fpc (Lazarus) the "*.txx" search takes 1-2 seconds (not measured) and when compiled with Delphi it takes +/- 0.3 seconds, so I stated to write my own TMask.

I am interested in how well your TMask version compares with Delphi's version.
Does it match the speed or even surpass it?

Anyway this unit looks very promising!
It works with Unicode as advertised. It passes all cases in the updated unit test. Our current implementation fails miserably. See components/lazutils/test/testmasks.lpi
For example this

  TestMask('ö', '[!☺-☂]', True);

means that 'ö' is not between a happy face and an umbrella. Case-insensitive matching of Unicode can be fixed later with functions found in LazUTF8.

With José's approval the license will be LGPL with a linking exception. It will be part of the LazUtils package. Author's name will be mentioned of course.
Is that OK?
I will not copy the whole original unit but use the UTF-8 parts + rename and tweak some things.

Regards
Juha
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20210224/eba6c526/attachment.html>

Reply by email

Bart
to Juha Manninen: On Wed, Feb 24, 2021 at 1:05 PM José Mejuto via lazarus < I am interested in how well…
PM Juha Manninen wrote 5 lines
I am interested in how well your TMask version compares with Delphi's version. Does it match the speed or even surpass it? Anyway this unit looks very promising! It works with Unicode as advertised. It passes all cases in the updated unit test. Our current implementation fails miserably.

IIRC then fpc has some wildcard matching code in one of it's *nix units. At least it used to have it.

Bart

Reply by email

José Mejuto
to Bart
Bart wrote 6 lines
> Filename:='test.txt' > Mask:='test??.txt?' > Match must be true That sucks big time. A ? is supposed to match EXACTLY 1 character (not optional). Bloody @ Micro$uck,

X-D

This quirk has its explanation which is 8.3 backwards compatibility. The old 8.3 masking (well the system itself) resolves masks setting the name and extension in a 11 bytes array where space is a "no-char", so file name "TEST.TXT" is stored as "TEXT " and "TXT" and if you apply this mask "TEST????.TXT" its logical that it evaluates to a positive match.

I'm quite sure that this comes from CP/M times.

Have a nice day.

Reply by email

José Mejuto
to Juha Manninen: On Wed, Feb 24, 2021 at 1:05 PM José Mejuto via lazarus < I am interested in how well…

El 24/02/2021 a las 21:53, Juha Manninen via lazarus escribió:

Hello,

I am interested in how well your TMask version compares with Delphi's version. Does it match the speed or even surpass it?

Not tested because in my code strings are allways UTF8 stored so for Delphi comparison I must convert them to Unicode before. I'll try to perform a simple benchmark.

Case-insensitive matching of Unicode can be fixed later with functions found in LazUTF8.

As this code is not a priority I think its better to make it work with Lazutf8 functions before the first commit. The problem is the support for sametext in UnicodeString.

With José's approval the license will be LGPL with a linking exception. It will be part of the LazUtils package. Author's name will be mentioned of course. Is that OK?

Yes, of course. Put the name if it is used for a reference, otherwise put standard headers.

I will not copy the whole original unit but use the UTF-8 parts + rename and tweak some things.

In fact I think that the whole unit is needed. It has (info for other readers) 3 classes TMaskUTF8, TMaskAnsi and TMaskUnicode, maybe TMaskAnsi can be omitted but UTF8 and Unicode should be present. Many times TMask is used over zillions of strings, converting Unicode to UTF8 (for UnicodeStrings and WideStrings) is time consuming, much more than the masking itself in most cases.

I suggest to keep the 3 classes and create a new TMask one which mimic the behaviour of current TMask, disabling the masking extensions (escape char, [?],...) which is very simple as you only need to subclass the Create method, mask compilation happens at first use time, not at creation time.

This way current code using TMask will behave 99.9% identical, but an user that needs to mask other strings can use TMaskUnicode, in example, and activate or deactivate other extensions.

Reply by email

Juha Manninen
to José Mejuto

On Thu, Feb 25, 2021 at 10:47 AM José Mejuto via lazarus <

Show quoted text 9 lines
wrote: In fact I think that the whole unit is needed. It has (info for other readers) 3 classes TMaskUTF8, TMaskAnsi and TMaskUnicode, maybe TMaskAnsi can be omitted but UTF8 and Unicode should be present. Many times TMask is used over zillions of strings, converting Unicode to UTF8 (for UnicodeStrings and WideStrings) is time consuming, much more than the masking itself in most cases.

UTF8 is also Unicode, one of its encodings.
The name UnicodeString is misleading. It should be UTF16String. Please remember our Unicode solution uses UTF-8. It is done by changing the default encoding of AnsiString and triggered by the same LazUTF8 unit that is used by Masks unit. Everything is UTF-8.

I suggest to keep the 3 classes and create a new TMask one which mimic

Show quoted text 5 lines
the behaviour of current TMask, disabling the masking extensions (escape char, [?],...) which is very simple as you only need to subclass the Create method, mask compilation happens at first use time, not at creation time.

I can include the TMaskUnicode class there if you want, although its name is also misleading.
TMaskAnsi must be left out. It has no use with our Unicode solution. TMaskUTF8 I have renamed to TMask in my tests. It replaces the current TMask which supports Unicode only partially. I could make an alias type
TMask = class(TMaskUTF8)
but why should I? Basically every String in our Unicode system has UTF-8 encoding. No need to have a special mask class for UTF-8.

This way current code using TMask will behave 99.9% identical, but an

user that needs to mask other strings can use TMaskUnicode, in example, and activate or deactivate other extensions.

Where do the other strings come from? Anyway TMaskUnicode can be included, no problem.
I am not sure we want a new TMask to behave 99.9% identical with the current one. The new one has some clear improvements. Interestingly there does not seem to be any standard for the mask syntax. So we cannot be compliant to any "standard".

Regards,
Juha
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20210225/b49178a2/attachment-0001.html>

Reply by email

José Mejuto
to Juha Manninen

El 25/02/2021 a las 10:39, Juha Manninen via lazarus escribió:

Hello,

Show quoted text 5 lines
UTF8 is also Unicode, one of its encodings. The name UnicodeString is misleading. It should be UTF16String. Please remember our Unicode solution uses UTF-8. It is done by changing the default encoding of AnsiString and triggered by the same LazUTF8 unit that is used by Masks unit. Everything is UTF-8.

Yes, Unicode is a very, very bad name. I've developed the code with fpc in mind, not Lazarus, that's the reason of the three versions. From the point of view of Lazarus, UTF8 is enough.

Show quoted text 10 lines
This way current code using TMask will behave 99.9% identical, but an user that needs to mask other strings can use TMaskUnicode, in example, and activate or deactivate other extensions. Where do the other strings come from? Anyway TMaskUnicode can be included, no problem. I am not sure we want a new TMask to behave 99.9% identical with the current one. The new one has some clear improvements. Interestingly there does not seem to be any standard for the mask syntax. So we cannot be compliant to any "standard".

Backwards compatibility, in special the escape character which can be used in old masks like: "C:\*.*". The other functions can be kept active, but escape char could be a compatibility problem.

Anyway, revisiting code I've found a bug in the escape character in char groups, which simply are ignored. It has been fixed in UTF8 version, I'm now porting to UTF16 and Ansi and I'll send you the updated version. Also fixed the "[!]" mask to raise an exception and the "RANGES_AUTOREVERSE" (was a variable rename from "cMask" to "lMask" which was missed in the IFDEF).

Have a nice day.

Reply by email

Juha Manninen
to José Mejuto

On Thu, Feb 25, 2021 at 12:44 PM José Mejuto via lazarus <

Show quoted text 6 lines
wrote: Backwards compatibility, in special the escape character which can be used in old masks like: "C:\*.*". The other functions can be kept active, but escape char could be a compatibility problem.

Ok, true.
Escaping special characters would be very handy. A pity. I kept the name TMaskUTF8 after all and inherited TMask from it. It helps synchronise changes between our versions, among other things. Now I need instructions or a piece of code into TMask constructor to make it ~ backwards compatible. Later we can see if the advanced features can be used.
I will look at the case-insensitive match of Unicode next...

Anyway, revisiting code I've found a bug in the escape character in char

Show quoted text 6 lines
groups, which simply are ignored. It has been fixed in UTF8 version, I'm now porting to UTF16 and Ansi and I'll send you the updated version. Also fixed the "[!]" mask to raise an exception and the "RANGES_AUTOREVERSE" (was a variable rename from "cMask" to "lMask" which was missed in the IFDEF).

I renamed "cMask" to "lMask" here. Earlier I made other changes. I renamed UTF8Length to UTF8CodepointSizeFast. LazUTF8 has that function for the purpose (and UTF8CodepointSize). Then I used the LazUTF8 version. UTF8Length in LazUTF8 means the number of codepoints in a whole string. I also changed PByte to PChar because of that function change. It seems to compile everywhere.
An exception from "[!]" mask would be good, yes. It is clearly an error from a user. The current TMask also complains about it.

Juha
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20210225/9bd9881e/attachment.html>

Reply by email

Bart
to Bart
PM Bart wrote
TMask.MatchesWindowsMask now raises an exception. Seriously?? This is INSANE.

Let me explain.
My backup and some related programs still compile, but instatly raise an exception when they start to perform their main task. Thank you very much.

The normal way of doing this is:
Deprecate the function in question, but d NOT kill it's functionality. Add a useful deprecated message.
Remove the function in the next major release (deprecate in 2.1, and so 2.2, only remove in 2.3, si't be gone in 2.4).
Simply removing functionality like you have done now will alienate users from Lazarus, since apparently "we" cannot be trusted.

Juha: you seem to be obsessed with speeding up string handling code. This is not really a problem as long as you are not deaf to arguments against your changes.
You introduce new bugs, remove old features, all for the sake of speed. All that when, in my perception, this code is mostly used in conjunction with file IO, which is orders of magnitude slower than even slopy string handling.

Bart

Reply by email

Juha Manninen
to Bart

On Fri, Feb 26, 2021 at 10:24 PM Bart via lazarus <

Show quoted text 23 lines
wrote: On Fri, Feb 26, 2021 at 7:15 PM Bart wrote: My backup and some related programs still compile, but instatly raise an exception when they start to perform their main task. Thank you very much. The normal way of doing this is: Deprecate the function in question, but d NOT kill it's functionality. Add a useful deprecated message. Remove the function in the next major release (deprecate in 2.1, and so 2.2, only remove in 2.3, si't be gone in 2.4). Simply removing functionality like you have done now will alienate users from Lazarus, since apparently "we" cannot be trusted. Juha: you seem to be obsessed with speeding up string handling code. This is not really a problem as long as you are not deaf to arguments against your changes. You introduce new bugs, remove old features, all for the sake of speed. All that when, in my perception, this code is mostly used in conjunction with file IO, which is orders of magnitude slower than even slopy string handling.

True, it created more conflicts than I anticipated. I reverted the new TMask in r64675. It must be worked later in trunk. Now a 2.2 fork will happen in few weeks.
Sorry for the hassle.

José and others. you can see my adaptation of your code in Lazarus trunk just before the revert, eg. r64674.
I also attach the unit here.
I fixed the case-insensitive Unicode match by simply replacing LowerCase() with UTF8LowerCase(). It is a well optimized function. First I planned to use UTF8CompareLatinTextFast() but it did not fit here. There is a unit test project in components/lazutils/test/. The code passes all tests there!
Unicode is fully supported also in mask ranges.

Let's continue the integration later.

Regards,
Juha
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20210227/558820c4/attachment-0001.html> -------------- next part --------------
A non-text attachment was scrubbed...
Name: masks_José.pas
Type: text/x-pascal
Size: 39836 bytes
Desc: not available
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20210227/558820c4/attachment-0001.pas>

Reply by email

Bart
to José Mejuto
PM José Mejuto wrote
I've uploaded last TMask* code to my github, this way its easier to track changes. https://github.com/JoshyFun/pascal_tmask

First we will fork Lazarus 2.2
Then work in mask unit can proceed.

I wonder if we could have both implementations sied by side for a while (so masks = old one, the new one whould have an appropriate new neame like lazmasks or similar). The component should still be called TMask and TMaskList and the function MatchesMask(list) should remain. This way backwards compatibility is preserved.
And we buy ourselves time.
Later on we can then deprecate the old masks unit (in 2.4) and only use the new one (remove the old one in 2.6).

The advantage of this is that the new implementation is not hindered by the old implementation.
So, no need to implement TMask.MatchesWindowsMask in the new implementation.

I did not study th enew implementation in depth, but things like escaping sound OK, if configurable.
Case sensitivity must be configurable, and in contrast to TMask we might choose to implement is as
CaseSensitive/CaseInsensitive/OSDefault.
Doing it in such a way would also make adapting the TShell* controls easier, we can simply publish a property that controls the options for the mask.

Bart

Reply by email

José Mejuto
to Bart

El 02/03/2021 a las 15:22, Bart via lazarus escribió:

Hello,

Show quoted text 6 lines
First we will fork Lazarus 2.2 Then work in mask unit can proceed. The advantage of this is that the new implementation is not hindered by the old implementation. So, no need to implement TMask.MatchesWindowsMask in the new implementation.

It's implemented as a new class, descendant from TMask*. Of course it needs some unit testing.

I did not study th enew implementation in depth, but things like escaping sound OK, if configurable.

It is, character and use it or not.

Case sensitivity must be configurable, and in contrast to TMask we might choose to implement is as CaseSensitive/CaseInsensitive/OSDefault.

Currently not OSDefault as for files it should be filesystem default which is unknown (filesystem decides on each call) and for regular strings I do not know which could be "OSDefault" :-?

Doing it in such a way would also make adapting the TShell* controls easier, we can simply publish a property that controls the options for the mask.

No problem, you can use the code or move it to the trash bin :-)

Reply by email