when rebuilding FPC (version 3.2.2, Lazarus 2.2.6, "x86_64-linux-gtk2") from sources, the compiler processes the symbol "fnmatch" that is 'paired' with the library "libgtk-x11-2.0.so". i have traced this back to the source file /usr/share/fpcsrc/3.2.2/packages/gtk2/src/gtk+/gtk/fnmatch.pas and in particular the following function declaration therein:

    { Match STRING against the filename pattern PATTERN,
       returning zero if it matches, FNM_NOMATCH if not.   }
    function fnmatch(__pattern: char; __string: char;
      __flags: gint): gint; cdecl; external gtklib;

 located at around line 36. the constant gtklib is defined in the file /usr/share/fpcsrc/3.2.2/packages/gtk2/src/gtk+/gtk/gtk2.pas as:

const
      gtklib = 'libgtk-x11-2.0.so';

however, the library "libgtk-x11-2.0.so" does NOT contain a symbol/function "fnmatch".

is anyone able to tell me where the symbol "fnmatch" resolves to? i have spent a number of days working on this without success. note that editing fnmatch.pas to either of the following variations:

function fnmatch( ... ) ... external;
function fnmatch( ... ) ... external 'libc';

both yield a compiler that, while capable of compiling a working terminal application, can NOT compile a (gtk2) GUI application. so the symbol "fnmatch" is NOT resolving to a match in "libc.so.6" where one would normally expect to find fnmatch defined.

many thanks for any help that can be offered...

cheers,
rob :-)

Op 16-10-2023 om 17:03 schreef robert rozee via fpc-devel:

See the discussion on the forum.

Summary:

- the compiler doesn't link anything with GTK2 or libc for that matter, so working compiler or not shouldn't depend on a symbol in GTK2 header.

- Since it has perfectly fine definition in libc, my guess that this is some old workaround of GTK2 for some defunct version (or multiple varying prototypes) of fnmatch, something might have been resolved a decade ago already.

- the symbol is never used in Lazarus to my best knowledge. Simply omitting the line from the GTK2 header should allow you to continue with your patch evaluation.

- without the patches you are reviewing it works all fine, suggesting a problem in those patches that somehow forces linking to the symbol without it being used.

CEST Marco van de Voort wrote

so the symbol "fnmatch" is NOT resolving to a match in "libc.so.6" where one would normally expect to find fnmatch defined

Usually, it is:

$ nm -D /lib64/libc.so.6 | grep fnmatch
00000000000e493e T fnmatch

Date: Tue, 17 Oct 2023 09:32:16 +0200
From: Marco van de Voort

- without the patches you are reviewing it works all fine, suggesting a problem in those patches that somehow forces linking to the symbol without it being used.

to dispel any doubt, i've just created a new Linux Mint 21.2 (XFCE) 64-bit VM, using an ISO downloaded from the Mint website. after running all updates on the VM i then downloaded FPC and Lazarus .DEB install packages from: https://sourceforge.net/projects/lazarus/files/Lazarus%20Linux%20i386%20DEB/Lazarus%202.2.6/ and installed the three packages (fpc-laz, fpc-src, lazarus-project) as per instructions. this gives a completely unpatch, unadulterated 64-bit Linux, running a completely unpatch, unadulterated FPC and Lazarus.

i next created a small GUI application that just displays a digital clock in a window. nothing fancy. this is the same 'GUI test application' i've been using throughout, although i created it anew to ensure there was no possibility of external contamination. i made NO changes to any of the Lazarus IDE settings, beyond selecting Tools->Options->Window->Show single button in TaskBar, and NO changes to the default project settings. the test program builds correctly, yielding a working GUI application that when run displays a small running digital clock display in a window.

after the above verification that everything was working, i then made a SINGLE change to the file fnmatch.inc located in /usr/share/fpcsrc/3.2.2/packages/gtk2/src/gtk+/gtk/ at line 37:

//// original declaration, start ////
function fnmatch(__pattern: char; __string: char;
__flags: gint): gint; cdecl; external gtklib;
//// original declaration, finish ////
//// altered declaration, start ////
function fnmatch(__pattern: char; __string: char;
__flags: gint): gint; cdecl; external; // gtklib;
//// altered declaration, finish ////

as you can see, the edit involved removing "gtklib" from after the word "external". i next rebuilt the compiler, RTL, etc using Fred vS's shell script: #!/bin/sh
COMPILER=fpc
make clean
make all FPC=$COMPILER OPT="-Fl/usr/local/lib"
make FPC=$COMPILER install INSTALL_PREFIX=/usr/

NOW, the test application FAILS TO BUILD, with lazbuild reporting:

lazbuild ./project1.lpr
Hint: (lazarus) [RunTool] "/usr/bin/fpc" "-iWTOTP" Hint: (lazarus) [RunTool] "/usr/bin/fpc" "-va" "compilertest.pas" Hint: (lazarus) [RunTool] "/usr/bin/fpc" "-iWTOTP" "-Px86_64" "-Tlinux" Hint: (lazarus) [RunTool] "/usr/bin/fpc" "-va" "compilertest.pas" "-Px86_64" "-Tlinux" TProject.DoLoadStateFile Statefile not found: /home/user/Pascal/Clock/lib/x86_64-linux/project1.compiled Hint: (11030) Start of reading config file /etc/fpc.cfg Hint: (11031) End of reading config file /etc/fpc.cfg Free Pascal Compiler version 3.2.2 [2023/10/17] for x86_64 Copyright (c) 1993-2021 by Florian Klaempfl and others (1002) Target OS: Linux for x86-64
(3104) Compiling project1.lpr
(10001) PPU Loading /usr/share/lazarus/2.2.6/lcl/units/x86_64-linux/gtk2/gtk2int.ppu (10011) PPU Source: gtk2int.pas not found
(10011) PPU Source: gtk2defines.inc not available
(10011) PPU Source: gtk2winapih.inc not available
(10011) PPU Source: gtk2lclintfh.inc not available (10011) PPU Source: gtk2listslh.inc not available
(10011) PPU Source: gtk2widgetset.inc not available (10011) PPU Source: gtk2winapi.inc not available
(10011) PPU Source: gtk2lclintf.inc not available
(10011) PPU Source: gtk2listsl.inc not available
(10028) Recompiling Gtk2Int, checksum changed for /usr/lib/fpc/3.2.2/units/x86_64-linux/gtk2/gtk2.ppu /usr/share/lazarus/2.2.6/lcl/units/x86_64-linux/gtk2/gtk2int.ppu:gtk2int.pas(14,36) Fatal: (10022) Can't find unit Gtk2Int used by Interfaces Fatal: (1018) Compilation aborted
Error: /usr/bin/ppcx64 returned an error exitcode
Error: (lazarus) Compile Project, Target: project1: stopped with exit code 1 Error: (lazbuild) failed compiling of project /home/user/Pascal/Clock/project1.lpi

reversing the edit, rebuilding the compiler, RTL, etc, and now when run lazbuild reports:

lazbuild ./project1.lpr
Hint: (lazarus) [RunTool] "/usr/bin/fpc" "-iWTOTP" Hint: (lazarus) [RunTool] "/usr/bin/fpc" "-va" "compilertest.pas" Hint: (lazarus) [RunTool] "/usr/bin/fpc" "-iWTOTP" "-Px86_64" "-Tlinux" Hint: (lazarus) [RunTool] "/usr/bin/fpc" "-va" "compilertest.pas" "-Px86_64" "-Tlinux" Hint: (11030) Start of reading config file /etc/fpc.cfg Hint: (11031) End of reading config file /etc/fpc.cfg Free Pascal Compiler version 3.2.2 [2023/10/17] for x86_64 Copyright (c) 1993-2021 by Florian Klaempfl and others (1002) Target OS: Linux for x86-64
(3104) Compiling project1.lpr
(3104) Compiling unit1.pas
/home/user/Pascal/Clock/unit1.pas(18,27) Hint: (5024) Parameter "Sender" not used /home/user/Pascal/Clock/unit1.pas(17,26) Hint: (5024) Parameter "Sender" not used (9022) Compiling resource /home/user/Pascal/Clock/lib/x86_64-linux/project1.or (9015) Linking /home/user/Pascal/Clock/project1
(1008) 92 lines compiled, 2.2 sec
(1022) 4 hint(s) issued

... and the result is a working GUI application. i can see no way the above results could exist if fnmatch were resolving to libc.so.6. the ONLY conclusion is that it is resolving to SOMEWHERE ELSE, and nobody seems to know where that somewhere else is.

would one of the FPC developers please care to duplicate the above tests?

cheers,
rob :-)

Op 17-10-2023 om 13:17 schreef robert rozee via fpc-devel:

And that works.

This is not a linking error or mis-resolved symbol or something similar. It is about incrementally rebuilding the LCL automatically as a dependency of your project after

(1) installing a new compiler, possibly erasing a release (-Ur) compiler with one that isn't.

(2) modifying LCL source.

Which is not _that_ surprising as you leave a lot of state behind from your earlier compilation. A Lazarus developer might give you some tips on that how to do this more systematic.  I usually first build lazarus with make bigide and then with lazbuild (to reenable packages) after such mods.

Another, simpler thing might be to simply erase all .o and .ppu from the lazarus repo before doing the lazbuild. That erases state (that hopefully will then be rebuilt in a consistent way by lazbuild)

reversing the edit, rebuilding the compiler, RTL, etc, and now when run lazbuild reports:

This seems to indicate the cause of the state problem is the source modification rather than the compiler recompile. Which is logical since it invalidates the RTL.

Op 17-10-2023 om 13:17 schreef robert rozee via fpc-devel:

why on earth is it logical??? one of the main functions of the FPC compiler (tm, the department of duplicate words department) should surely be the ability to rebuilt itself. if it can NOT rebuild itself successfully then there is something _really_ wrong down in the gubbins. i can flip back and forth between working and not working, as many times as i wish, without any deviation from the outcome. furthermore, it is my understanding that the compiler is DESIGNED to be rebuilt by the end user.

what i am demonstrating is that if an ERROR in the source code is REMOVED, then the compiler breaks. surely this is very, very wrong?

cheers,
rob :-)

Do you realize that neither the LCL nor the Lazarus are "the compiler"? It might be better to discuss what is designed for Lazarus / LCL in some Lazarus list.

what i am demonstrating is that if an ERROR in the source code is REMOVED, then the compiler breaks. surely this is very, very wrong?

The compiler doesn't break, it reports an error situation. The compiler also provides means for analyzing this situation if needed (-vt and -s compiler optionsin this case). Alternatively, you can choose the end user approach, but then you should probably follow the documentation coming from Lazarus regarding how things should be done.

Tomas

lazbuild should have noticed the changed file date of the compiler and recompile all used packages.

What packages are listed in the project inspector?

Mattias

Sent: Wednesday, October 18, 2023 at 1:13 AM
From: "Mattias Gaertner via fpc-devel"

What packages are listed in the project inspector?

you mean for my test application when it is opened in the Lazarus IDE?

Files
project1.lpr
unit1.pas
Required Packages
LCL

i would strongly recommend you simply creating a x86 64-bit Linux VM and duplicating the process i outlined at the start. the problem manifests when trying to compile _any_ GTK2 GUI application - although i must admit i've not tried with just a blank form! as Marco has pointed out, i discovered the problem while developing a patch to the compiler, but the problem is not related to the patch. my suspicion is that the function fnmatch (as defined in /usr/share/fpcsrc/3.2.2/packages/gtk2/src/gtk+/gtk/fnmatch.inc, line 36-37) is used by the compiler when it is compiling any GTK2 GUI application, and that the compiler is dependent upon the function doing or returning something very different to what glibc's fnmatch does. but figuring the what/how is contingent on finding out where lines 36-37 resolve to.

cheers,
rob :-)

Let's see if I got this right:
You installed the debs and then installed a new compiler, effectively replacing deb files.

The problem is that the debs install a version (Makefile.compiled files) that is blind for changes to the compiler.

If you need a custom compiler, you need to compile lazarus yourself too, either clone it yourself with git or use fpcupdelux.

Mattias

Sent: Wednesday, October 18, 2023 at 2:17 AM
From: "Mattias Gaertner via fpc-devel"
To: 

i start out with a clean Linux install, then install the three .deb packages.

next i place in /usr/share/fpcsrc/3.2.2/ the following script and run it as root:

#!/bin/sh
COMPILER=fpc
make clean
make all FPC=$COMPILER OPT="-Fl/usr/local/lib"
make FPC=$COMPILER install INSTALL_PREFIX=/usr/

this script, according to Fred vS, rebuilds: FPC, the FPC RTL, and the Lazarus LCL. i have every reason to believe that this script does rebuild everything, but i am open to any additional steps that should be added to the script to rebuild anything that has been missed out.

cheers,
rob :-)

On 2023-10-17 15:34, robert rozee via fpc-devel wrote:

.
.

No, Lazarus LCL is not rebuilt this way, this is just for FPC and FPC RTL.

Tomas

No, it updates fpc and all its packages. It does not touch any Lazarus package. They must be rebuilt too. Since all packages needs rebuilding, it makes no sense to use the lazarus deb. Just use the git version or fpcupdelux.

Mattias

Sent: Wednesday, October 18, 2023 at 3:05 AM
From: "Mattias Gaertner via fpc-devel"

interesting. the above script has been widely circulating on the lazarus/FPC forums, and to date none of the 'experts' there, including Marco, have suggest it is deficient. this silence disappoints me.

what is required to rebuild the "Lazarus packages" locally? i do notice that from within the Lazarus IDE there is a menu option to 'rebuild all', but that this does not work (for Linux users at least) as, in part at least, doing so requires running as root. simply downloading the pre-compiled git version of anything is pointless, as anything pre-built from there is going to contain the same pairing of fnmatch/libgtk-x11-2.0.so.

cheers,
rob :-)

The script is ok to update fpc. And if you use lazarus from git then lazarus and lazbuild automatically rebuild the lazarus packages. The script is not fine in combination with the lazarus deb.

[...] simply downloading the pre-compiled git version of anything is pointless

Where is this coming from?
I'm not aware of a pre-compiled git version of lazarus.

Clone the lazarus git and build it with "make distclean bigide".

Mattias

Sent: Wednesday, October 18, 2023 at 9:58 AM
From: "Mattias Gaertner via fpc-devel"

my mistake, i have never used the git repository, and just work from the three .deb packages from sourceforge. i assumed you meant that nightly builds existed and were available, and i should use these.

i prefer to work with a non-moving target, that is the sourceforge releases. this way, others can duplicate exactly my environment, and see what i see. i also believe that the vast majority of users get their FPC and Lazarus from sourceforge, hence it is important that any patches can be applied to such installs.

given that i have installed from the three .deb packages, how do i perform a rebuild of the Lazarus packages? do i add the line "make distclean bigide" to the end of Fred vS's script?

cheers,
rob :-)

robert rozee wrote

[...] i prefer to work with a non-moving target, that is the sourceforge releases. this way, others can duplicate exactly my environment, and see what i see. i also believe that the vast majority of users get their FPC and Lazarus from sourceforge, hence it is important that any patches can be applied to such installs.

Patches don't work with debs.

git can reproduce any release, e.g.

cd your/lazarusgit
git checkout lazarus_3_0_RC2

given that i have installed from the three .deb packages, how do i perform a rebuild of the Lazarus packages? do i add the line "make distclean bigide" to the end of Fred vS's script?

cd your/lazarusgit
make distclean bigide

Mattias

Sent: Wednesday, October 18, 2023 at 9:03 PM
From: "Mattias Gaertner via fpc-devel"

brilliant! the following works for me:

cd /usr/share/lazarus/2.2.6
sudo make distclean bigide

doing the above after rebuilding FPC (with fnmatch.inc changed to remove the reference to gtklib) results in lazbuild being able to build a GTK2 GUI application!

regarding patches and debs, i think we are talking cross-purposes. i am not patching the .deb files or their contents. i am using the three .debs to install FPC and Lazarus, then patching (now two) source files: pdecsub.pas and fnmatch.inc in usr/share/fpcsrc. the patch applied to pdecsub changes the way 'external' statements are processed, converting unversioned glibc symbols into BASE versioned ones. once FPC and Lazarus have been rebuilt, this results in the compiler now generating binaries that will run against _all_ versions of glibc, including those earlier than the one installed on the machine that created the binary.

i hope you can see, my patch will NOT EVER be applied against the sources on gitlab, nor is EVER intended to be applied in that way. it is to be applied to a _local_ installation on an individual user's computer so that said user can themself decide how to deal with glibc version compatibility.

Mattias: many many thanks for helping put the last piece of the puzzle in place! "make distclean bigide" was the missing piece needed to solve the puzzle. if i offended you or anyone else while getting to the solution, please accept my sincerest apologies.

cheers,
rob :-)

Hello.

@ Rob:

Hum, if I may, I never said that this script will rebuild Lazarus LCL, it will only rebuild FPC and RTL.

Fre;D

Sent: Thursday, October 19, 2023 at 1:56 AM
From: "Fred vS via fpc-devel"

my bad - Fred is entirely correct, i got confused about who told me the script would rebuild the LCL. over the course of the past few months hundreds of emails have been exchanged with people from all over the place, and in this case i lost track of who said what! my apologies, Fred.

btw, if any of the fpc-developers are interested, see the below posting for details of the patch: https://forum.lazarus.freepascal.org/index.php/topic,64731.msg494276.html#msg494276

cheers,
rob :-)

fnmatch is a libc function and AFAICT, GTK never exported a function with the name "fnmatch". GTK has a file named fnmatch.c which declares "_gtk_fnmatch" with different parameters than libc fnmatch. There is no GTK header file named fnmatch.h. GTK "_gtk_fnmatch" is only exposed in gtkprivate.h (likely for private / internal use only).

_gtk_fnmatch current version:
https://gitlab.gnome.org/GNOME/gtk/-/blob/main/gtk/fnmatch.c#L249 https://gitlab.gnome.org/GNOME/gtk/-/blob/main/gtk/gtkprivate.h#L63

_gtk_fnmatch 2004 version (just before fpc did their gtk header conversion): https://gitlab.gnome.org/GNOME/gtk/-/blob/3127f29ef626097dd861376f36a0914a20f4598c/gtk/fnmatch.c#L253 https://gitlab.gnome.org/GNOME/gtk/-/blob/3127f29ef626097dd861376f36a0914a20f4598c/gtk/gtkprivate.h#L102

To promote good housekeeping, and prevent strange linking behavior and errors I propose the removal of the incorrect declaration from ./packages/gtk2/src/gtk+/gtk/fnmatch.inc:

function fnmatch(__pattern: AnsiChar; __string: AnsiChar;
  __flags: gint): gint; cdecl; external gtklib;

or remove the entire file (or move it out of gtk2 and put it where it belongs?): ./packages/gtk2/src/gtk+/gtk/fnmatch.inc

Trunk fpc fnmatch.inc version:
https://gitlab.com/freepascal.org/fpc/source/-/blob/main/packages/gtk2/src/gtk+/gtk/fnmatch.inc

Original 2005 fpc fnmatch.inc version:
https://gitlab.com/freepascal.org/fpc/source/-/blob/3c9295f2856b775a6db83a2fde40b9992b492cb9/packages/extra/gtk2/gtk+/gtk/fnmatch.inc

AFAIK, the GTK fnmatch can be found here:

https://codebrowser.dev/gtk/include/fnmatch.h.html

But I am 100% in favour of only using fnmatch from glibc !

Additional source-link:
https://gitlab.gnome.org/rburton/gdk-pixbuf/-/blob/GTK_MULTIHEAD_MERGEPOINT_28_03_02/gtk/fnmatch.c#L61

AFAIK, the GTK fnmatch can be found here: https://codebrowser.dev/gtk/include/fnmatch.h.html Additional source-link: https://gitlab.gnome.org/rburton/gdk-pixbuf/...

Ok, I searched again in the official/upstream repo and found the function fnmatch and file fnmatch.h. In 2002 fnmatch.c was changed to remove the function name "fnmatch" and replace it with "gtk_fnmatch_intern" and "_gtk_fnmatch". In the same commit fnmatch.h was deleted. The fpc header import apparently did not pick up these changes.

GTK commit:
https://gitlab.gnome.org/GNOME/gtk/-/commit/73b15ba391b3a533786e2a2f4f80274b80866822 https://gitlab.gnome.org/GNOME/gtk/-/commit/73b15ba391b3a533786e2a2f4f80274b80866822#8af485fae929f582a6202cf54a88b00a0cc9824b_60_78

System fnmatch wasn't going to be UTF-8 clean, neither was our version.

Fri Dec 13 17:45:40 2002 Owen Taylor

* gtk/fnmatch.c gtk/gtkprivate.h gtk/gtkfilesel.c: System fnmatch wasn't going to be UTF-8 clean, neither was our version. Redo our fnmatch.c to be UTF-8, add test cases, fix all sorts of bugs inherited
from the antique GNU fnmatch code. Change interface to get rid of fnmatch.h constants. Fixes basic
non-workingness of filesel with non-ASCII filenames.

* gtk/fnmatch.h: No longer needed.