Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-pascal / 2014-03

fpc in symlinked directory

45 messages · 7 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Mattias Gaertner

Hi,

When I start fpc in a directory containing symlinks the compiler uses the resolved directory, leading to wrong source file names. For example:

/tmp/orig/project1.pas
/tmp/link -> /tmp/orig

cd /tmp/link
fpc -vb project1.pas
/tmp/orig/project1.pas(6,5) Error: Identifier not found "w"

Is there a workaround?
For example a way to tell fpc to use the PWD environment variable?

Mattias

Michael Van Canneyt
to Mattias Gaertner
Mattias Gaertner wrote 14 lines
Hi, When I start fpc in a directory containing symlinks the compiler uses the resolved directory, leading to wrong source file names. For example: /tmp/orig/project1.pas /tmp/link -> /tmp/orig cd /tmp/link fpc -vb project1.pas /tmp/orig/project1.pas(6,5) Error: Identifier not found "w" Is there a workaround?

No.

For example a way to tell fpc to use the PWD environment variable?

No.

Because if you do a Chdir('ABC') inside a FPC program, the CWD variable is no longer valid. So it would require us to keep track of what you do. We could hook the Chdir and SetDirectory calls. But that would go wrong at the first direct syscall or library call that changes the current directory.

So, FPC uses a system call to get the correct (resolved) CWD directory. This is the only guaranteed mechanism.

So, the best Lazarus can do is resolve all directories before calling the compiler.

Since you brought it up:
IMHO Lazarus must do this resolving anyhow, because now Lazarus does not work at all with symlinks, e.g. to find filenames and resolve packages. Regardless of what the compiler does.

I submitted bugreports on the list about this in the past.

Michael.

Mattias Gaertner
to Michael Van Canneyt
Show quoted text 7 lines
[...] > For example a way to tell fpc to use the PWD environment > variable? No. Because if you do a Chdir('ABC') inside a FPC program,

'a FPC program' sounds as if you are talking about console programs in general. My question is only about the compiler.

the CWD variable is no longer valid. So it would require us to keep track of what you do.

What can I do to the compiler, that needs tracking?

Show quoted text 6 lines
We could hook the Chdir and SetDirectory calls. But that would go wrong at the first direct syscall or library call that changes the current directory. So, FPC uses a system call to get the correct (resolved) CWD directory. This is the only guaranteed mechanism.
What's wrong with this:
  if GetEnvironmentVariable('PWD')'' then
    SetCurrentDir(GetEnvironmentVariable('PWD'));
?
So, the best Lazarus can do is resolve all directories before calling the compiler.

If you mean with "resolve" to convert relative paths to absolute paths: That is not sufficient.
For example when the program uses a unit in the same directory, the compiler uses its current directory.

Since you brought it up: IMHO Lazarus must do this resolving anyhow, because now Lazarus does not work at all with symlinks, e.g. to find filenames and resolve packages. Regardless of what the compiler does. I submitted bugreports on the list about this in the past.

I remember some problems. Formerly the IDE resolved symlinks, which was wrong. This was fixed.

Mattias

Michael Van Canneyt
to Mattias Gaertner
Mattias Gaertner wrote 7 lines
> > So, FPC uses a system call to get the correct (resolved) CWD directory. > This is the only guaranteed mechanism. What's wrong with this: if GetEnvironmentVariable('PWD')'' then SetCurrentDir(GetEnvironmentVariable('PWD'));

It's a hack ?

Show quoted text 6 lines
? > So, the best Lazarus can do is resolve all directories before calling the compiler. If you mean with "resolve" to convert relative paths to absolute paths: That is not sufficient.

Yes it is.
But you must do it correctly for ALL paths, and take care to use absolute paths only. Relative paths and symlinks do not work.

For example when the program uses a unit in the same directory, the compiler uses its current directory.

What does that mean ? Can you give a command-line example ? (sources and all)

Show quoted text 7 lines
> Since you brought it up: > IMHO Lazarus must do this resolving anyhow, because now Lazarus does not work at all with symlinks, e.g. to find filenames and resolve packages. Regardless of what the compiler does. > > I submitted bugreports on the list about this in the past. I remember some problems. Formerly the IDE resolved symlinks, which was wrong. This was fixed.

That depends on what you call 'fixed' :)

I use a lot of symlinks. To this day, the IDE cannot handle them correctly.

I must always take great care not to select a project/package using a symlink, but always must use the 'correct' resolved path.

The problem is the use of relative paths to indicate units of dependent packages. I have asked in the past to specify paths to packages as absolute paths, never relative:
if you use relative paths, things will get messed up when using symlinks.

And it has nothing to do with the compiler, the compiler isn't called when loading a package or a project, or specifying paths to packages.

Michael.

Mattias Gaertner
to Michael Van Canneyt
Show quoted text 13 lines
On Tue, 25 Mar 2014, Mattias Gaertner wrote: >> >> So, FPC uses a system call to get the correct (resolved) CWD directory. >> This is the only guaranteed mechanism. > > What's wrong with this: > if GetEnvironmentVariable('PWD')'' then > SetCurrentDir(GetEnvironmentVariable('PWD')); It's a hack ?

AFAIK the PWD is the official thing and used by other console tools as well. For example when I compile a file with gcc in a symlinked directory it creates debugging info with the unresolved file name.

But SetCurrentDir does not work. It seems to resolve directories. The documentation does not mention this behavior. Is this by design?

Show quoted text 14 lines
> ? > >> So, the best Lazarus can do is resolve all directories before calling the compiler. > > If you mean with "resolve" to convert relative paths to absolute paths: > That is not sufficient. Yes it is. But you must do it correctly for ALL paths, and take care to use absolute paths only. Relative paths and symlinks do not work. > For example when the program uses a unit in the same directory, the > compiler uses its current directory. What does that mean ? Can you give a command-line example ?

I used this one:
cd /tmp/link
/usr/bin/fpc -g -gl -vb -Fi/tmp/link -Fu/tmp/link/ -FU/tmp/link/ /tmp/link/project1.lpr

unit1.o:
<33> DW_AT_comp_dir : /tmp/orig/

(sources and all)

Pretty straightforward:

-----------------------------
program project1;
uses sysutils, Unit1;
begin
  DoSomething('');
end.
------------------------------
unit Unit1;
{$mode objfpc}{$H+}
interface
uses
  Classes, SysUtils;
procedure DoSomething(s: string);
implementation
procedure DoSomething(s: string);
begin
  writeln('DoSomething ',s);
end;
end.
--------------------------------
Show quoted text 19 lines
>> Since you brought it up: >> IMHO Lazarus must do this resolving anyhow, because now Lazarus does not work at all with symlinks, e.g. to find filenames and resolve packages. Regardless of what the compiler does. >> >> I submitted bugreports on the list about this in the past. > > I remember some problems. Formerly the IDE resolved symlinks, which > was wrong. This was fixed. That depends on what you call 'fixed' :) I use a lot of symlinks. To this day, the IDE cannot handle them correctly. I must always take great care not to select a project/package using a symlink, but always must use the 'correct' resolved path. The problem is the use of relative paths to indicate units of dependent packages. I have asked in the past to specify paths to packages as absolute paths, never relative: if you use relative paths, things will get messed up when using symlinks.

It works here. Please create a bug report.

Keep in mind that /tmp/link/unit1.pas and /tmp/orig/uni1.pas must be treated as two different files and within one project you must use only one of them.
Only when writing a file the IDE needs to check for symlinks.

And it has nothing to do with the compiler, the compiler isn't called when loading a package or a project, or specifying paths to packages.

No, but with the current state of the compiler, symlinks are not supported.

Mattias

Mattias Gaertner
to Mattias Gaertner
[...] But SetCurrentDir does not work. It seems to resolve directories. The documentation does not mention this behavior. Is this by design?

The setcwd of C does the same, so I guess it is by design. It should be documented.

Mattias

Michael Van Canneyt
to Mattias Gaertner: On Wed, 26 Mar 2014 09:10:09 +0100 (CET) AFAIK the PWD is the official thing and used by…
Mattias Gaertner wrote 20 lines
On Wed, 26 Mar 2014 09:10:09 +0100 (CET) Michael Van Canneyt wrote: > > > On Tue, 25 Mar 2014, Mattias Gaertner wrote: > >>> >>> So, FPC uses a system call to get the correct (resolved) CWD directory. >>> This is the only guaranteed mechanism. >> >> What's wrong with this: >> if GetEnvironmentVariable('PWD')'' then >> SetCurrentDir(GetEnvironmentVariable('PWD')); > > It's a hack ? AFAIK the PWD is the official thing and used by other console tools as well. For example when I compile a file with gcc in a symlinked directory it creates debugging info with the unresolved file name.

This is not correct.
You seem to think bash-specific behaviour is the correct way to do things. It is not, it is specific to bash.

I use tcsh:

cadwal: >cd /tmp
cadwal: >ls -ld src1 link
lrwxrwxrwx 1 michael michael 4 Mar 26 11:09 link -> src1/ drwxrwxr-x 2 michael michael 4096 Mar 26 11:12 src1/ cadwal: >cd link
cadwal: >pwd
/tmp/src1

As you see, it has "resolved" the link.

But SetCurrentDir does not work. It seems to resolve directories. The documentation does not mention this behavior. Is this by design?

Yes.

It matches GetCurrentDir.

Show quoted text 8 lines
>> compiler uses its current directory. > > What does that mean ? Can you give a command-line example ? I used this one: cd /tmp/link /usr/bin/fpc -g -gl -vb -Fi/tmp/link -Fu/tmp/link/ -FU/tmp/link/ /tmp/link/project1.lpr

cadwal: >fpc -g -gl -vb -Fi/tmp/link -Fu/tmp/link -FU/tmp/link /tmp/link/p.pp /usr/bin/ld: warning: link.res contains output sections; did you forget -T?

So your example works fine here.

Show quoted text 5 lines
> I have asked in the past to specify paths to packages as absolute > paths, never relative: > if you use relative paths, things will get messed up when using symlinks. It works here. Please create a bug report.

Will do.

Keep in mind that /tmp/link/unit1.pas and /tmp/orig/uni1.pas must be treated as two different files and within one project you must use only one of them.

There you are wrong. They must be treated as the same file. They ARE the same file, after all.

That is why FPC resolves everything.
/proc/self/cwd does the same: it resolves the directory.

cadwal: >ls /proc/self/cwd -l
lrwxrwxrwx 1 michael michael 0 Mar 26 11:18 /proc/self/cwd -> /tmp/src1/

So I still think that the FPC behaviour is the only correct behaviour.

And likewise, I think that Lazarus is still wrong using relative paths.

Michael.

Mattias Gaertner
to Michael Van Canneyt
Show quoted text 21 lines
[...] > AFAIK the PWD is the official thing and used by other console tools as > well. For example when I compile a file with gcc in a symlinked > directory it creates debugging info with the unresolved file name. This is not correct. You seem to think bash-specific behaviour is the correct way to do things. It is not, it is specific to bash. I use tcsh: cadwal: >cd /tmp cadwal: >ls -ld src1 link lrwxrwxrwx 1 michael michael 4 Mar 26 11:09 link -> src1/ drwxrwxr-x 2 michael michael 4096 Mar 26 11:12 src1/ cadwal: >cd link cadwal: >pwd /tmp/src1 As you see, it has "resolved" the link.

You seem to think tcsh-specific behaviour is the correct way to do things. ;)

tcsh uses -P as default, while bash uses -L as default:

pwd -L
/tmp/link

And using gcc in tcsh uses /tmp/link for debugging info as well.

Show quoted text 14 lines
[...] >>> compiler uses its current directory. >> >> What does that mean ? Can you give a command-line example ? > > I used this one: > cd /tmp/link > /usr/bin/fpc -g -gl -vb -Fi/tmp/link -Fu/tmp/link/ > -FU/tmp/link/ /tmp/link/project1.lpr cadwal: >fpc -g -gl -vb -Fi/tmp/link -Fu/tmp/link -FU/tmp/link /tmp/link/p.pp /usr/bin/ld: warning: link.res contains output sections; did you forget -T? So your example works fine here.

It compiles, but it creates wrong debugging info.
What shows "objdump -g unit1.o | grep tmp" for you? /tmp/link/ or /tmp/orig/?

[...]

Mattias

Marco van de Voort
to Mattias Gaertner: On Wed, 26 Mar 2014 09:10:09 +0100 (CET) AFAIK the PWD is the official thing and used by…

In our previous episode, Mattias Gaertner said:

Show quoted text 5 lines
> > It's a hack ? AFAIK the PWD is the official thing and used by other console tools as well.

afaik:

1) it is an official thing only of the shell, not the "system".
2) it assumes that any program doesn't change paths inbetween. (either the executed program before pwd test, or sh->exe1->exe2 scenarios, excluding freak stuff like daemons)

Michael Van Canneyt
to Mattias Gaertner: On Wed, 26 Mar 2014 11:25:19 +0100 (CET) You seem to think tcsh-specific behaviour is the…
Mattias Gaertner wrote 27 lines
On Wed, 26 Mar 2014 11:25:19 +0100 (CET) Michael Van Canneyt wrote: > [...] >> AFAIK the PWD is the official thing and used by other console tools as >> well. For example when I compile a file with gcc in a symlinked >> directory it creates debugging info with the unresolved file name. > > > This is not correct. > You seem to think bash-specific behaviour is the correct way to do things. > It is not, it is specific to bash. > > I use tcsh: > > cadwal: >cd /tmp > cadwal: >ls -ld src1 link > lrwxrwxrwx 1 michael michael 4 Mar 26 11:09 link -> src1/ > drwxrwxr-x 2 michael michael 4096 Mar 26 11:12 src1/ > cadwal: >cd link > cadwal: >pwd > /tmp/src1 > > As you see, it has "resolved" the link. You seem to think tcsh-specific behaviour is the correct way to do things. ;)

Well, actually I don't think in such terms. I think that the system is right. And the system (the linux kernel) has resolved the symlink to find the correct file/dir. which is the only correct behaviour in my opinion.

Show quoted text 6 lines
tcsh uses -P as default, while bash uses -L as default: pwd -L /tmp/link And using gcc in tcsh uses /tmp/link for debugging info as well.

Maybe they cater for it. I think this is a noble, but misplaced, intention. If they would specify full pathnames, there would not be a problem.

cadwal: >objdump -g a.o

a.o: file format elf64-x86-64

Show quoted text 17 lines
> [...] >>>> compiler uses its current directory. >>> >>> What does that mean ? Can you give a command-line example ? >> >> I used this one: >> cd /tmp/link >> /usr/bin/fpc -g -gl -vb -Fi/tmp/link -Fu/tmp/link/ >> -FU/tmp/link/ /tmp/link/project1.lpr > > cadwal: >fpc -g -gl -vb -Fi/tmp/link -Fu/tmp/link -FU/tmp/link /tmp/link/p.pp > /usr/bin/ld: warning: link.res contains output sections; did you forget -T? > > So your example works fine here. It compiles, but it creates wrong debugging info.

Eh ? You didn't say anything about debugging info ?

What shows "objdump -g unit1.o | grep tmp" for you? /tmp/link/ or /tmp/orig/?

cadwal: >objdump -g a.o

a.o: file format elf64-x86-64

:)

Aha...
I suspect the "leading to wrong source file names." of your initial post is referring to the source filenames in debug info.

If this is the case: it is just an additional indication that Lazarus must simply resolve all filenames. If it does so, then there is no problem, since FPC and the OS do the same.

As a consequence, It will get rid of the annoying myfile.pp(2) when opening a source file during a debug session.

BTW: The Firebird database server also does this when resolving database filenames. It checks the inode number. If it didn't, it would open the same DB file twice, thinking they are different DBs - and corrupting them in the process. So it needs to uniquely identify each file, to make sure it opens each DB file only once.

Which, again, I think is the only sensible behaviour.

Michael.

Tomas Hajny
to Michael Van Canneyt: This is not correct. You seem to think bash-specific behaviour is the correct way to do…
Michael Van Canneyt wrote 9 lines
On Wed, 26 Mar 2014, Mattias Gaertner wrote: > On Wed, 26 Mar 2014 09:10:09 +0100 (CET) > Michael Van Canneyt wrote: >> On Tue, 25 Mar 2014, Mattias Gaertner wrote: >> >>>> >>>> So, FPC uses a system call to get the correct (resolved) CWD >>>> directory. >>>> This is the only guaranteed mechanism.

.
.

Show quoted text 9 lines
cadwal: >cd /tmp cadwal: >ls -ld src1 link lrwxrwxrwx 1 michael michael 4 Mar 26 11:09 link -> src1/ drwxrwxr-x 2 michael michael 4096 Mar 26 11:12 src1/ cadwal: >cd link cadwal: >pwd /tmp/src1 As you see, it has "resolved" the link.

Please note that this is not portable / consistent among platforms - if you create a junction under MS Windows (which is more or less equivalent to Unix links), the respective operating system API call returns the path with the junction name rather than the real name.

.
.

Show quoted text 6 lines
> Keep in mind that /tmp/link/unit1.pas and /tmp/orig/uni1.pas must be > treated as two different files and within one project you must use > only one of them. There you are wrong. They must be treated as the same file. They ARE the same file, after all.

They are obviously the same file, but the application may not be able to detect it easily in all cases on all platforms (at least not using platform independent API like our RTL). In my view, that's in fact the point of links - allowing applications to work as if the real directory setup was different from the reality. Assuming that the RTL should uncover this is in my view not a good idea.

.
.

That is why FPC resolves everything.

.
.

That probably depends on what you mean by "everything". E.g. references to used units in debug information are stored as relative paths (and include junction names if you used those when passing the path to the compiler).

Tomas

Mattias Gaertner
to Michael Van Canneyt: This is not correct. You seem to think bash-specific behaviour is the correct way to do…
Show quoted text 7 lines
[...] > Keep in mind that /tmp/link/unit1.pas and /tmp/orig/uni1.pas must be > treated as two different files and within one project you must use > only one of them. There you are wrong. They must be treated as the same file. They ARE the same file, after all.

When a user tells a program to open file /tmp/link/unit1.pas, he expects it to work with /tmp/link/unit1.pas
not /tmp/orig/unit1.pas. Almost all editors do that and so does Lazarus.

That is why FPC resolves everything.

Not true.
For example when I leave out the 'cd /tmp/link' and my cwd is /home/mattias, then fpc creates the debugging info "../../tmp/link/unit1.pas".

/proc/self/cwd does the same: it resolves the directory.

This is the physical cwd.

cadwal: >ls /proc/self/cwd -l lrwxrwxrwx 1 michael michael 0 Mar 26 11:18 /proc/self/cwd -> /tmp/src1/ So I still think that the FPC behaviour is the only correct behaviour.

What is incorrect with using the logical cwd?

And likewise, I think that Lazarus is still wrong using relative paths.

AFAIK the only issue with symlinks are the '..', as this works on the physical parent. There is no problem with a relative path like 'foo/bar'.
I don't see how absolute paths can work platform independently. Relative paths are more readable.

Mattias

Michael Van Canneyt
to Tomas Hajny: . . Please note that this is not portable / consistent among platforms - if you create a…
Tomas Hajny wrote 26 lines
On Wed, March 26, 2014 11:25, Michael Van Canneyt wrote: > On Wed, 26 Mar 2014, Mattias Gaertner wrote: >> On Wed, 26 Mar 2014 09:10:09 +0100 (CET) >> Michael Van Canneyt wrote: >>> On Tue, 25 Mar 2014, Mattias Gaertner wrote: >>> >>>>> >>>>> So, FPC uses a system call to get the correct (resolved) CWD >>>>> directory. >>>>> This is the only guaranteed mechanism. . . > cadwal: >cd /tmp > cadwal: >ls -ld src1 link > lrwxrwxrwx 1 michael michael 4 Mar 26 11:09 link -> src1/ > drwxrwxr-x 2 michael michael 4096 Mar 26 11:12 src1/ > cadwal: >cd link > cadwal: >pwd > /tmp/src1 > > As you see, it has "resolved" the link. Please note that this is not portable / consistent among platforms - if you create a junction under MS Windows (which is more or less equivalent to Unix links), the respective operating system API call returns the path with the junction name rather than the real name.

I didn't mean to imply this.

I reason within the frame of reference of Unix.

Show quoted text 13 lines
>> Keep in mind that /tmp/link/unit1.pas and /tmp/orig/uni1.pas must be >> treated as two different files and within one project you must use >> only one of them. > > There you are wrong. They must be treated as the same file. > They ARE the same file, after all. They are obviously the same file, but the application may not be able to detect it easily in all cases on all platforms (at least not using platform independent API like our RTL). In my view, that's in fact the point of links - allowing applications to work as if the real directory setup was different from the reality. Assuming that the RTL should uncover this is in my view not a good idea.

The RTL does not 'uncover' anything. It uses strictly what the OS offers.

In the case of set/get CWD, the OS is queried/instructed.

So does Libc.

PWD is a convention, and IMHO belongs on the application level.

Show quoted text 7 lines
> That is why FPC resolves everything. . . That probably depends on what you mean by "everything". E.g. references to used units in debug information are stored as relative paths (and include junction names if you used those when passing the path to the compiler).

I was not speaking about compiler and debug info, but about the RTL.

As for paths used in the compiler: This is the application level. I don't think something needs to be done, but maybe the compiler developers think differently.

Michael.

Michael Van Canneyt
to Mattias Gaertner: On Wed, 26 Mar 2014 11:25:19 +0100 (CET) When a user tells a program to open file…
Mattias Gaertner wrote 14 lines
On Wed, 26 Mar 2014 11:25:19 +0100 (CET) Michael Van Canneyt wrote: > [...] >> Keep in mind that /tmp/link/unit1.pas and /tmp/orig/uni1.pas must be >> treated as two different files and within one project you must use >> only one of them. > > There you are wrong. They must be treated as the same file. > They ARE the same file, after all. When a user tells a program to open file /tmp/link/unit1.pas, he expects it to work with /tmp/link/unit1.pas not /tmp/orig/unit1.pas. Almost all editors do that and so does Lazarus.

I am not arguing with this.
You can store the filename with the path as specified by the user. Relative if you want, for files within the same project.

But I am arguing that when passing filenames/paths to other tools (including the compiler), you should always specify full filenames.

This way, you don't need to worry about the distinction between physical/logical.

Show quoted text 11 lines
> That is why FPC resolves everything. Not true. For example when I leave out the 'cd /tmp/link' and my cwd is /home/mattias, then fpc creates the debugging info "../../tmp/link/unit1.pas". > /proc/self/cwd does the same: it resolves the directory. This is the physical cwd.

Yes. It is the only one that makes sense.

Show quoted text 8 lines
> cadwal: >ls /proc/self/cwd -l > lrwxrwxrwx 1 michael michael 0 Mar 26 11:18 /proc/self/cwd -> /tmp/src1/ > > So I still think that the FPC behaviour is the only correct behaviour. What is incorrect with using the logical cwd?

That passing relative paths such as ../../xyz become ambiguous, since they rely on the receiving tool obeying the same convention as the sending tool.

Show quoted text 5 lines
> And likewise, I think that Lazarus is still wrong using relative paths. AFAIK the only issue with symlinks are the '..', as this works on the physical parent. There is no problem with a relative path like 'foo/bar'.

Exactly. Lazarus specifies ALL links to required packages as relative paths.

They go horribly wrong in the presence of symlinks.

I don't see how absolute paths can work platform independently.

But they don't have to be platform independent ?

I have a project A, depending on package B.
I don't have to specify a path in the dependencies.

Lazarus KNOWS where package B is located on my current system. So, it can specify the full path.
Whether this path contains symlinks or not is then totally irrelevant.

However, if lazarus wants to specify a relative path, it must calculate this.

Then suddenly, the difference between "logical" (in fact, badly chosen name, it should be 'historical" IMHO) and pĥysical path becomes relevant, and depends on how the 'receiving' tool reacts on this difference.

Ergo: Don't use relative paths, always absolute,
when passing paths to other tools. All problems are solved.

If I MANUALLY specify some path as a relative path, only then Lazarus may assume that I know what I am doing, and pass the path unmodified to external tools.

I don't see what the problem is with this reasoning. I am in fact surprised that you can think any other way... (but that is my shortcoming ;) )

Michael.

Mattias Gaertner
to Michael Van Canneyt: Well, actually I don't think in such terms. I think that the system is right. And the…
Show quoted text 8 lines
On Wed, 26 Mar 2014, Mattias Gaertner wrote: [...] > You seem to think tcsh-specific behaviour is the correct way to do > things. ;) Well, actually I don't think in such terms. I think that the system is right. And the system (the linux kernel) has resolved the symlink to find the correct file/dir. which is the only correct behaviour in my opinion.

The system finds the right file by itself. That's why programs can work on a higher level.

[...] Maybe they cater for it. I think this is a noble, but misplaced, intention. If they would specify full pathnames, there would not be a problem.

What problem do they have? /tmp/link is a valid file name.

cadwal: >objdump -g a.o a.o: file format elf64-x86-64

objdump -g a.out | grep tmp
<15> DW_AT_comp_dir : (indirect string, offset: 0x7d): /tmp/link

Show quoted text 18 lines
[...] >>>>> compiler uses its current directory. >>>> >>>> What does that mean ? Can you give a command-line example ? >>> >>> I used this one: >>> cd /tmp/link >>> /usr/bin/fpc -g -gl -vb -Fi/tmp/link -Fu/tmp/link/ >>> -FU/tmp/link/ /tmp/link/project1.lpr >> >> cadwal: >fpc -g -gl -vb -Fi/tmp/link -Fu/tmp/link -FU/tmp/link /tmp/link/p.pp >> /usr/bin/ld: warning: link.res contains output sections; did you forget -T? >> >> So your example works fine here. > > It compiles, but it creates wrong debugging info. Eh ? You didn't say anything about debugging info ?

Yes, sorry. The wrong file names appear in error messages and debug info.
Just add a syntax error to unit1.pas and compile once in /tmp/link and once outside. You see once the physical directory and once the logical.

Show quoted text 12 lines
> What shows "objdump -g unit1.o | grep tmp" for you? > /tmp/link/ or /tmp/orig/? cadwal: >objdump -g a.o a.o: file format elf64-x86-64 :) Aha... I suspect the "leading to wrong source file names." of your initial post is referring to the source filenames in debug info.

It's both. Syntax errors and debug info.

If this is the case: it is just an additional indication that Lazarus must simply resolve all filenames. If it does so, then there is no problem, since FPC and the OS do the same.

The problem is the following:
The user opens /tmp/link/project1.lpr in the source editor and runs it. When there is a syntax error or when the debugger stops they report a source file name '/tmp/orig/unit1.pas' and the user wonders why. The IDE automatically opens the file '/tmp/orig/unit1.pas', but since it shows in the tab only the 'unit1.pas' the user may confuse that with the right file.
It is possible to implement some heuristic search to find the file that was "meant". But it would be nicer if fpc would use the logical directory, as it does for all other paths.

Show quoted text 10 lines
As a consequence, It will get rid of the annoying myfile.pp(2) when opening a source file during a debug session. BTW: The Firebird database server also does this when resolving database filenames. It checks the inode number. If it didn't, it would open the same DB file twice, thinking they are different DBs - and corrupting them in the process. So it needs to uniquely identify each file, to make sure it opens each DB file only once. Which, again, I think is the only sensible behaviour.

That's about writing to a file. I already wrote, that the source editor does that too. Some places of the IDE does not do the check.

For instance:
One include file used in different contexts. Depending on its file name different units use it and parse it differently. FPC reports the logical file name on an error and so the IDE knows the right context. The IDE opens the file two times and with different settings.

Mattias

Mattias Gaertner
to Michael Van Canneyt: I am not arguing with this. You can store the filename with the path as specified by the…
[...] But I am arguing that when passing filenames/paths to other tools (including the compiler), you should always specify full filenames.

The IDE passes all search paths as full file paths, especially no '..'. No problem here. Symlinks don't need to be resolved for this.

Show quoted text 7 lines
[...] >> That is why FPC resolves everything. > > Not true. > For example when I leave out the 'cd /tmp/link' and my cwd > is /home/mattias, then fpc creates the debugging info > "../../tmp/link/unit1.pas".

No comment here?

Show quoted text 5 lines
>> /proc/self/cwd does the same: it resolves the directory. > > This is the physical cwd. Yes. It is the only one that makes sense.

It makes sense for the kernel, because that works on the lower level.

Michael, why do use symlinks? Why don't you always specify the fully resolved paths? You must see some sense in logical paths.

Show quoted text 8 lines
[...] >> And likewise, I think that Lazarus is still wrong using relative paths. > > AFAIK the only issue with symlinks are the '..', as this works on the > physical parent. There is no problem with a relative path like > 'foo/bar'. Exactly. Lazarus specifies ALL links to required packages as relative paths.

Not here.
Please create a bug report with an example where Lazarus passes '..' to a tool.

[...]

Mattias

Michael Van Canneyt
to Mattias Gaertner: On Wed, 26 Mar 2014 12:07:01 +0100 (CET) The system finds the right file by itself.…
Mattias Gaertner wrote 22 lines
On Wed, 26 Mar 2014 12:07:01 +0100 (CET) Michael Van Canneyt wrote: > On Wed, 26 Mar 2014, Mattias Gaertner wrote: > [...] >> You seem to think tcsh-specific behaviour is the correct way to do >> things. ;) > > Well, actually I don't think in such terms. I think that the system is right. > And the system (the linux kernel) has resolved the symlink to find the correct file/dir. > which is the only correct behaviour in my opinion. The system finds the right file by itself. That's why programs can work on a higher level. > [...] > Maybe they cater for it. I think this is a noble, but misplaced, intention. > If they would specify full pathnames, there would not be a problem. What problem do they have? /tmp/link is a valid file name.

Only because it is a full path. Then it doesn't matter whether you resolve or not.

But things like ../link may end up wildly on wildly different places depending on whether the start path is logical or physical.

Example:

/home/michael/projects/lazarus/lazarus.lpi
/home/michael/lazarus -> /home/michael/projects/lazarus /home/michael/packages/synapse/laz_synapse.lpk

If I open
/home/michael/projects/lazarus/lazarus.lpi
then the path to laz_synapse is
../../../synapse/laz_synapse

If I open
/home/michael/lazarus/lazarus.lpi
then the path to laz_synapse is (omitting the units/platform for simplicity) ../../synapse/laz_synapse

Specifying the latter to the compiler fails, because the compiler uses physical paths.

But specifying
/home/michael/packages/synapse/laz_synapse.lpk
will work correctly in both cases.

It gets complicated even more when the packages are in directories that have symlinks and I opened through the symlink or not.

Show quoted text 9 lines
>> It compiles, but it creates wrong debugging info. > > Eh ? You didn't say anything about debugging info ? Yes, sorry. The wrong file names appear in error messages and debug info. Just add a syntax error to unit1.pas and compile once in /tmp/link and once outside. You see once the physical directory and once the logical.

So ? You just need to check the inode.

Show quoted text 9 lines
> and the OS do the same. The problem is the following: The user opens /tmp/link/project1.lpr in the source editor and runs it. When there is a syntax error or when the debugger stops they report a source file name '/tmp/orig/unit1.pas' and the user wonders why. The IDE automatically opens the file '/tmp/orig/unit1.pas', but since it shows in the tab only the 'unit1.pas' the user may confuse that with the right file.

? it IS the right file. It is the SAME file, after all.

Show quoted text 20 lines
> As a consequence, It will get rid of the annoying myfile.pp(2) when opening a source > file during a debug session. > > BTW: The Firebird database server also does this when resolving database > filenames. It checks the inode number. If it didn't, it would open the > same DB file twice, thinking they are different DBs - and corrupting them > in the process. So it needs to uniquely identify each file, to make sure > it opens each DB file only once. > > Which, again, I think is the only sensible behaviour. That's about writing to a file. I already wrote, that the source editor does that too. Some places of the IDE does not do the check. For instance: One include file used in different contexts. Depending on its file name different units use it and parse it differently. FPC reports the logical file name on an error and so the IDE knows the right context. The IDE opens the file two times and with different settings.

Which IMHO is totally misplaced behaviour.

Lazarus sometimes tries to be too clever. This is one of these times...

Michael.

Michael Van Canneyt
to Mattias Gaertner: On Wed, 26 Mar 2014 13:37:32 +0100 (CET) The IDE passes all search paths as full file…
Mattias Gaertner wrote 9 lines
On Wed, 26 Mar 2014 13:37:32 +0100 (CET) Michael Van Canneyt wrote: > [...] > But I am arguing that when passing filenames/paths to other tools > (including the compiler), you should always specify full filenames. The IDE passes all search paths as full file paths, especially no '..'. No problem here. Symlinks don't need to be resolved for this.

Euh ? Last time I checked, the paths were full of ../ ? (admittedly, that was some time ago, I stopped opening files through symlinks...)

Show quoted text 9 lines
> [...] >>> That is why FPC resolves everything. >> >> Not true. >> For example when I leave out the 'cd /tmp/link' and my cwd >> is /home/mattias, then fpc creates the debugging info >> "../../tmp/link/unit1.pas". No comment here?

I tested, and indeed:

# include_directories
         .ascii  "../../tmp/link\000"

I think this is an error in FPC, because the linker file contains full paths:

INPUT(
/usr/local/lib/fpc/2.7.1/units/x86_64-linux/rtl/prt0.o /tmp/link/p.o
/usr/local/lib/fpc/2.7.1/units/x86_64-linux/rtl/system.o ...

The two should at least match; and you will not be surprised to hear that I think the linker file is correct :)

Show quoted text 7 lines
>>> /proc/self/cwd does the same: it resolves the directory. >> >> This is the physical cwd. > > Yes. It is the only one that makes sense. It makes sense for the kernel, because that works on the lower level.

There is only 1 level: the system level.

The rest are mirages.

Michael, why do use symlinks? Why don't you always specify the fully resolved paths? You must see some sense in logical paths.

Stop: There is a difference between symlinks and logical paths.

Did I say I was against symlinks ? No. I use them a lot, and all the time.

They're just typing shortcuts, for convenience.

So, if I open (because it is faster)

/home/michael/lazarus/lazarus.lpi

and then the IDE caption tells me I have actually opened

/home/michael/projects/lazarus/lazarus.lpi

I will only be happy.

Similarly, if I do on the commandline:

cd ~
cd lazarus

and then pwd answers me:

/home/michael/projects/lazarus/

I am not surprised and actually quite happy that the tool correctly figured out the symlink.

By contrast, I am surprised (one might even say worried) that there exist people which expect (no, they insist) that pwd reports

/home/michael/lazarus/

In my highly personal opinion, these people do not understand symlinks.

And apparently such people not only exist, they seem to thrive...

The whole dicsussion is about the difference between knowing where you are (physical) and how you got there (logical).

The second concept is simply alien to me, because I consider it utterly irrelevant.

Show quoted text 12 lines
> [...] >>> And likewise, I think that Lazarus is still wrong using relative paths. >> >> AFAIK the only issue with symlinks are the '..', as this works on the >> physical parent. There is no problem with a relative path like >> 'foo/bar'. > > Exactly. Lazarus specifies ALL links to required packages as relative paths. Not here. Please create a bug report with an example where Lazarus passes '..' to a tool.

No problem. I will test again, and report.

Michael.

Mattias Gaertner
to Michael Van Canneyt: Only because it is a full path. Then it doesn't matter whether you resolve or not. But…
Show quoted text 12 lines
[...] > What problem do they have? /tmp/link is a valid file name. Only because it is a full path. Then it doesn't matter whether you resolve or not. But things like ../link may end up wildly on wildly different places depending on whether the start path is logical or physical. Example: /home/michael/projects/lazarus/lazarus.lpi /home/michael/lazarus -> /home/michael/projects/lazarus

Note: Two different lazarus directories should use two different config directories, because the config directory also contains absolute file names.

Show quoted text 13 lines
/home/michael/packages/synapse/laz_synapse.lpk If I open /home/michael/projects/lazarus/lazarus.lpi then the path to laz_synapse is ../../../synapse/laz_synapse If I open /home/michael/lazarus/lazarus.lpi then the path to laz_synapse is (omitting the units/platform for simplicity) ../../synapse/laz_synapse Specifying the latter to the compiler fails, because the compiler uses physical paths.

When does Lazarus pass '../../synapse/laz_synapse' to the compiler?

Show quoted text 6 lines
[...] > Just add a syntax error to unit1.pas and compile once in /tmp/link and > once outside. You see once the physical directory and once the > logical. So ? You just need to check the inode.

Is there a function to list all files pointing to an inode?

Show quoted text 11 lines
>> and the OS do the same. > > The problem is the following: > The user opens /tmp/link/project1.lpr in the source editor and runs > it. When there is a syntax error or when the debugger stops they report > a source file name '/tmp/orig/unit1.pas' and the user wonders why. > The IDE automatically opens the file '/tmp/orig/unit1.pas', but since > it shows in the tab only the 'unit1.pas' the user may confuse that > with the right file. ? it IS the right file. It is the SAME file, after all.

We could make a poll. I guess the majority expects that when the IDE compiles /tmp/link/project1.pas then it should work with /tmp/link/uni1.pas, not /somewhereelse/unit1.pas.

Show quoted text 8 lines
[...] > For instance: > One include file used in different contexts. Depending on its file > name different units use it and parse it differently. FPC reports the > logical file name on an error and so the IDE knows the right context. > The IDE opens the file two times and with different settings. Which IMHO is totally misplaced behaviour.

Yes, and still: some people do that.

Lazarus sometimes tries to be too clever. This is one of these times...

Actually the symlink solution is a workaround for Lazarus not being so clever.

Mattias

Tomas Hajny
to Michael Van Canneyt: Euh ? Last time I checked, the paths were full of ../ ? (admittedly, that was some time…
Michael Van Canneyt wrote 44 lines
On Wed, 26 Mar 2014, Mattias Gaertner wrote: > On Wed, 26 Mar 2014 13:37:32 +0100 (CET) > Michael Van Canneyt wrote: > >> [...] >> But I am arguing that when passing filenames/paths to other tools >> (including the compiler), you should always specify full filenames. > > The IDE passes all search paths as full file paths, especially no > '..'. No problem here. Symlinks don't need to be resolved for this. Euh ? Last time I checked, the paths were full of ../ ? (admittedly, that was some time ago, I stopped opening files through symlinks...) >> [...] >>>> That is why FPC resolves everything. >>> >>> Not true. >>> For example when I leave out the 'cd /tmp/link' and my cwd >>> is /home/mattias, then fpc creates the debugging info >>> "../../tmp/link/unit1.pas". > > No comment here? I tested, and indeed: # include_directories .ascii "../../tmp/link\000" I think this is an error in FPC, because the linker file contains full paths: INPUT( /usr/local/lib/fpc/2.7.1/units/x86_64-linux/rtl/prt0.o /tmp/link/p.o /usr/local/lib/fpc/2.7.1/units/x86_64-linux/rtl/system.o ... The two should at least match; and you will not be surprised to hear that I think the linker file is correct :)

.
.

It makes no sense to use absolute paths in debug information, because it would make distribution of units compiled with debug information to other machines impossible (or more precisely - it would be possible to distribute them, but debugging would not work :/ ).

Tomas

Michael Van Canneyt
to Mattias Gaertner: On Wed, 26 Mar 2014 14:31:19 +0100 (CET) Note: Two different lazarus directories should…
Mattias Gaertner wrote 10 lines
On Wed, 26 Mar 2014 14:31:19 +0100 (CET) > [...] >> Just add a syntax error to unit1.pas and compile once in /tmp/link and >> once outside. You see once the physical directory and once the >> logical. > > So ? You just need to check the inode. Is there a function to list all files pointing to an inode?

Actually, you just need to know if 2 filenames point to the same inode.

You don't need all filenames pointing to the same inode.

Show quoted text 15 lines
>>> and the OS do the same. >> >> The problem is the following: >> The user opens /tmp/link/project1.lpr in the source editor and runs >> it. When there is a syntax error or when the debugger stops they report >> a source file name '/tmp/orig/unit1.pas' and the user wonders why. >> The IDE automatically opens the file '/tmp/orig/unit1.pas', but since >> it shows in the tab only the 'unit1.pas' the user may confuse that >> with the right file. > > ? it IS the right file. It is the SAME file, after all. We could make a poll. I guess the majority expects that when the IDE compiles /tmp/link/project1.pas then it should work with /tmp/link/uni1.pas, not /somewhereelse/unit1.pas.

I certainly don't, but I guess we differ in our view. See my other mail.

Show quoted text 10 lines
> [...] >> For instance: >> One include file used in different contexts. Depending on its file >> name different units use it and parse it differently. FPC reports the >> logical file name on an error and so the IDE knows the right context. >> The IDE opens the file two times and with different settings. > > Which IMHO is totally misplaced behaviour. Yes, and still: some people do that.

So ?

Let's say 90% of people use a screwdriver as a hammer. It doesn't make them right. It means 90% of people are abusing the screwdriver.

(just pulling something out of my hat, it may well be 95% or 75% or less)

Michael.

Michael Van Canneyt
to Tomas Hajny: . . It makes no sense to use absolute paths in debug information, because it would make…
Tomas Hajny wrote 9 lines
> The two should at least match; and you will not be surprised to hear that > I think the linker file is correct :) . . It makes no sense to use absolute paths in debug information, because it would make distribution of units compiled with debug information to other machines impossible (or more precisely - it would be possible to distribute them, but debugging would not work :/ ).

I was waiting for this argument, because it is totally wrong :-)

The paths on another system may be wildly different anyway. Just take a debian vs. fedora system.

Paths are valid only on the system where the compiler/debugger are running;
anything else is wishful thinking.

Michael.

Martin Frb
to Tomas Hajny: . . It makes no sense to use absolute paths in debug information, because it would make…
Tomas Hajny wrote 5 lines
It makes no sense to use absolute paths in debug information, because it would make distribution of units compiled with debug information to other machines impossible (or more precisely - it would be possible to distribute them, but debugging would not work :/ ).

Absolute filenames (or at least more than just the name) are a must for some cases.

iF you debug a library, then you start some launcher app, which loads the library. Both launcher app, and library may contain a unit called "Foo". Yet the 2 Foo may be different units from different locations. With just the filename, the debugger can not handle this.

As for debugging the exe on a diff computer. The IDE will try by filename only, if full path can not be found.
This fallback is also needed, because older gdb have some problems finding a unit by full path, if .. are in the name (which is the case for at least the lazarus.exe).

Mattias Gaertner
to Michael Van Canneyt: Euh ? Last time I checked, the paths were full of ../ ? (admittedly, that was some time…
Show quoted text 16 lines
[...] I tested, and indeed: # include_directories .ascii "../../tmp/link\000" I think this is an error in FPC, because the linker file contains full paths: INPUT( /usr/local/lib/fpc/2.7.1/units/x86_64-linux/rtl/prt0.o /tmp/link/p.o /usr/local/lib/fpc/2.7.1/units/x86_64-linux/rtl/system.o ... The two should at least match; and you will not be surprised to hear that I think the linker file is correct :)

Well, the /tmp/link is a logical, not a physical path. So I'm not yet sure, why you think that physical cwd is the "only correct" way, but it's ok for all other to be logical. Isn't that inconsistent?

[...] There is only 1 level: the system level. The rest are mirages.

lol :)

[...] Did I say I was against symlinks ? No. I use them a lot, and all the time. They're just typing shortcuts, for convenience.

You can do more with symlinks than that. Famous examples: /tmp -> /var/tmp or the links in /lib.

Show quoted text 9 lines
So, if I open (because it is faster) /home/michael/lazarus/lazarus.lpi and then the IDE caption tells me I have actually opened /home/michael/projects/lazarus/lazarus.lpi I will only be happy.

And bash users will report it as a bug.

[...bash bashing, irrelevant to this thread...]

Mattias

Marco van de Voort
to Tomas Hajny: . . It makes no sense to use absolute paths in debug information, because it would make…

In our previous episode, Tomas Hajny said:

Show quoted text 9 lines
> The two should at least match; and you will not be surprised to hear that > I think the linker file is correct :) . . It makes no sense to use absolute paths in debug information, because it would make distribution of units compiled with debug information to other machines impossible (or more precisely - it would be possible to distribute them, but debugging would not work :/ ).

If you don't you might neither, since the unit could be included by

uses xx in  'some/rel/path/xx.pas';

where currentdir is in -Fu and some/rel/path is not.

Mattias Gaertner
to Michael Van Canneyt: Actually, you just need to know if 2 filenames point to the same inode. You don't need…
Show quoted text 6 lines
[...] >> So ? You just need to check the inode. > > Is there a function to list all files pointing to an inode? Actually, you just need to know if 2 filenames point to the same inode.

And what 2 files should I check? There are thousands of files in the search path.
IMO we can shorten this: Going from logical to physical is easy, going back is more difficult. It is more complicated than just checking the inode. Especially if you want to do that on all thousands of messages from the compiler. That's an implementation detail and off-topic.

Show quoted text 9 lines
[...] >> ? it IS the right file. It is the SAME file, after all. > > We could make a poll. I guess the majority expects that when the > IDE compiles /tmp/link/project1.pas then it should > work with /tmp/link/uni1.pas, not /somewhereelse/unit1.pas. I certainly don't, but I guess we differ in our view. See my other mail.

So, you would be happy, if the IDE opens /tmp/orig/unit1.pas instead of /tmp/link/unit1.pas?

>> [...] Let's say 90% of people use a screwdriver as a hammer. It doesn't make them right. It means 90% of people are abusing the screwdriver.

Well, bash is wrong, logical paths are wrong, sharing an include file is wrong, ..., 90% are abusing the screwdriver. ;)
I guess we are getting off-topic.

Mattias

Michael Van Canneyt
to Mattias Gaertner: On Wed, 26 Mar 2014 15:19:29 +0100 (CET) Well, the /tmp/link is a logical, not a physical…
Mattias Gaertner wrote 23 lines
On Wed, 26 Mar 2014 15:19:29 +0100 (CET) Michael Van Canneyt wrote: > [...] > I tested, and indeed: > > # include_directories > .ascii "../../tmp/link\000" > > I think this is an error in FPC, because the linker file contains full paths: > > INPUT( > /usr/local/lib/fpc/2.7.1/units/x86_64-linux/rtl/prt0.o > /tmp/link/p.o > /usr/local/lib/fpc/2.7.1/units/x86_64-linux/rtl/system.o > ... > > The two should at least match; and you will not be surprised to hear that > I think the linker file is correct :) Well, the /tmp/link is a logical, not a physical path. So I'm not yet sure, why you think that physical cwd is the "only correct" way, but it's ok for all other to be logical. Isn't that inconsistent?

Not in the least:

Like I said, it is the difference between 'where am i' and 'how do I get here'.

CWD is clearly 'where am i'. It is a location.
Making CWD return 'how did I get to this location' is wrong.

Symlinks make 'getting somewhere' easier, they provide shortcuts. Their use is convenient, but no more than that.

(yes, I know you can do some other nifty tricks with them)

Show quoted text 6 lines
> Did I say I was against symlinks ? No. I use them a lot, and all the time. > > They're just typing shortcuts, for convenience. You can do more with symlinks than that. Famous examples: /tmp -> /var/tmp or the links in /lib.

Sure. All of them specify how to get somewhere. No problem with that.

Show quoted text 11 lines
> So, if I open (because it is faster) > > /home/michael/lazarus/lazarus.lpi > > and then the IDE caption tells me I have actually opened > > /home/michael/projects/lazarus/lazarus.lpi > > I will only be happy. And bash users will report it as a bug.

<quip>
Bash users should be bashed on the head till they understand symlinks :) Why do you think I use tcsh ?
</quip>

Seriously: I think the analogy using 'where am I' and 'how did I get here' expresses best my view on the whole issue.

Michael.

Michael Van Canneyt
to Mattias Gaertner: On Wed, 26 Mar 2014 15:33:38 +0100 (CET) And what 2 files should I check? There are…
Mattias Gaertner wrote 12 lines
On Wed, 26 Mar 2014 15:33:38 +0100 (CET) Michael Van Canneyt wrote: > [...] >>> So ? You just need to check the inode. >> >> Is there a function to list all files pointing to an inode? > > Actually, you just need to know if 2 filenames point to the same inode. And what 2 files should I check? There are thousands of files in the search path.

To avoid duplicates, only the files that are already open in the IDE must be checked. For the others, it is irrelevant how you opened them.

There are 2 scenarios:

1. A file with a similar name (filename part matches) is already open in the IDE. If you collect the inode when opening, there is no problem (I'm guessing you already do a stat on open.
2. No such file is opened:
You can open it using whatever path.

The next time a file must be opened, you check the file at the 'locical' location, and if its inode matches an opened file, you know it is open already.

Show quoted text 20 lines
IMO we can shorten this: Going from logical to physical is easy, going back is more difficult. It is more complicated than just checking the inode. Especially if you want to do that on all thousands of messages from the compiler. That's an implementation detail and off-topic. > [...] >>> ? it IS the right file. It is the SAME file, after all. >> >> We could make a poll. I guess the majority expects that when the >> IDE compiles /tmp/link/project1.pas then it should >> work with /tmp/link/uni1.pas, not /somewhereelse/unit1.pas. > > I certainly don't, but I guess we differ in our view. > See my other mail. So, you would be happy, if the IDE opens /tmp/orig/unit1.pas instead of /tmp/link/unit1.pas?

Absolutely.

The only reason I would use a symlink is to avoid traversing N levels of directory. It is the end result that matters.

Michael.

Marc Weustink
to Michael Van Canneyt
Michael Van Canneyt wrote 33 lines
On Wed, 26 Mar 2014, Mattias Gaertner wrote: > On Wed, 26 Mar 2014 15:33:38 +0100 (CET) > Michael Van Canneyt wrote: > >> [...] >>>> So ? You just need to check the inode. >>> >>> Is there a function to list all files pointing to an inode? >> >> Actually, you just need to know if 2 filenames point to the same inode. > > And what 2 files should I check? There are thousands of files in > the search path. To avoid duplicates, only the files that are already open in the IDE must be checked. For the others, it is irrelevant how you opened them. There are 2 scenarios: 1. A file with a similar name (filename part matches) is already open in the IDE. If you collect the inode when opening, there is no problem (I'm guessing you already do a stat on open. 2. No such file is opened: You can open it using whatever path. The next time a file must be opened, you check the file at the 'locical' location, and if its inode matches an opened file, you know it is open already.

This will fail if the file is located on some smb or nfs mount.

Marc

Mattias Gaertner
to Mattias Gaertner: When I start fpc in a directory containing symlinks the compiler uses the resolved…

Here is a summarize so far:

When the compiler is started in a directory with a symlink, the directory is resolved and used for error messages and debug info. All other directories are not resolved.
If you want fpc to use only unresolved paths, you can start fpc in a directory without sources. But this requires adapting all relative paths, which can not be done automatically in general.

GetCurrentDir always returns the resolved physical file name.

Mattias

Show quoted text 21 lines
Hi, When I start fpc in a directory containing symlinks the compiler uses the resolved directory, leading to wrong source file names. For example: /tmp/orig/project1.pas /tmp/link -> /tmp/orig cd /tmp/link fpc -vb project1.pas /tmp/orig/project1.pas(6,5) Error: Identifier not found "w" Is there a workaround? For example a way to tell fpc to use the PWD environment variable? Mattias _______________________________________________ fpc-pascal maillist - http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Tomas Hajny
to Michael Van Canneyt: I was waiting for this argument, because it is totally wrong :-) The paths on another…
Michael Van Canneyt wrote 21 lines
On Wed, 26 Mar 2014, Tomas Hajny wrote: >> The two should at least match; and you will not be surprised to hear >> that >> I think the linker file is correct :) > . > . > > It makes no sense to use absolute paths in debug information, because it > would make distribution of units compiled with debug information to > other > machines impossible (or more precisely - it would be possible to > distribute them, but debugging would not work :/ ). I was waiting for this argument, because it is totally wrong :-) The paths on another system may be wildly different anyway. Just take a debian vs. fedora system. Paths are valid only on the system where the compiler/debugger are running; anything else is wishful thinking.

OK, I enjoy being wrong. ;-) My point is that the system where the compiler is (was) running may be different from the system where the debugger is running. Among others, this scenario is explicitly supported by FPC with its parameter supporting creation of a script for linking on a (different) host machine.

And I'd indeed wish to keep possibility of copying my compiled units to a different machine (which may not follow the same rules for location of the home directories, or may provide some other designated space for my work outside of the standard home directory) while still being able to debug the created executable there (as easily as possible - that is fulfilled by the current situation of using relative paths within debug information, but would be negatively impacted by a potential change to absolute paths everywhere in FPC including specifically debug information).

Tomas

Michael Van Canneyt
to Tomas Hajny
Tomas Hajny wrote 28 lines
On Wed, March 26, 2014 15:35, Michael Van Canneyt wrote: > On Wed, 26 Mar 2014, Tomas Hajny wrote: >>> The two should at least match; and you will not be surprised to hear >>> that >>> I think the linker file is correct :) >> . >> . >> >> It makes no sense to use absolute paths in debug information, because it >> would make distribution of units compiled with debug information to >> other >> machines impossible (or more precisely - it would be possible to >> distribute them, but debugging would not work :/ ). > > I was waiting for this argument, because it is totally wrong :-) > > The paths on another system may be wildly different anyway. > Just take a debian vs. fedora system. > > Paths are valid only on the system where the compiler/debugger are > running; > anything else is wishful thinking. OK, I enjoy being wrong. ;-) My point is that the system where the compiler is (was) running may be different from the system where the debugger is running. Among others, this scenario is explicitly supported by FPC with its parameter supporting creation of a script for linking on a (different) host machine.

I understand the logic.

Show quoted text 9 lines
And I'd indeed wish to keep possibility of copying my compiled units to a different machine (which may not follow the same rules for location of the home directories, or may provide some other designated space for my work outside of the standard home directory) while still being able to debug the created executable there (as easily as possible - that is fulfilled by the current situation of using relative paths within debug information, but would be negatively impacted by a potential change to absolute paths everywhere in FPC including specifically debug information).

At the very least I think that we should be consistent and use the same filenames in link script and debug info ?

For all other problems: making an option telling the compiler to use only absolute/relative paths seems like an option which can be used to cater for all situations.

Michael.

Jonas Maebe
to Michael Van Canneyt
Michael Van Canneyt wrote
At the very least I think that we should be consistent and use the same filenames in link script and debug info ?

How directories are stored in the debug info is generally defined by the debug info format itself. E.g. the DWARF standard explicitly mentions that search directories for source files can be either relative or absolute to the current working directory (there's even an attribute to specify a default CWD, so that the files can be easily found on the compilation system itself even when the program is started from another location). Specifying the directories in a relative way means that you can easily copy over a debug binary and all related source code from one system to another (or move things around) without breaking anything. For that reason, we explicitly add relative directory names to the DWARF debug info rather than absolute ones.

For linking, we use absolute paths in link.res to ensure that the parameters given by the programmer to the compiler (such as -Fl and -XR) are taken into account when linking, and that the linker doesn't accidentally link in some other library with the same name that it found in its default search path.

I don't see what the advantage of being "consistent" between the generation of debug info and linker scripts would be. Making the paths in the debug info absolute would make things less flexible without increasing usability (I've never seen bug reports about problems with the wrong source files being found). Making the paths in the linker script relative would add a lot of complexity (the compiler first searches for the library files on the files system, and then it would have to make the paths relative to the CWD for no gain) and special cases (when linking on a remote system, the compiler has no clue about what the relative position of e.g. /usr/lib will be compared to the CWD, so then you would still have to use absolute paths).

Jonas

Michael Van Canneyt
to Mattias Gaertner: Here is a summarize so far: When the compiler is started in a directory with a symlink,…
Mattias Gaertner wrote 10 lines
Here is a summarize so far: When the compiler is started in a directory with a symlink, the directory is resolved and used for error messages and debug info. All other directories are not resolved. If you want fpc to use only unresolved paths, you can start fpc in a directory without sources. But this requires adapting all relative paths, which can not be done automatically in general. GetCurrentDir always returns the resolved physical file name.

The compiler does not do anything special: it determines its current working directory using the RTL, meaning it gets the resolved current working directory.

Additionally, source filenames in debug info are relative to (it seems) the current working directory.

Meanwhile I tested the IDE, and indeed the paths are passed using full paths. I am glad this is so, and wonder how long I've been in the dark erroneously :(

In that case, may I suggest to leave the 'Show relative paths' unchecked by default in the "show options" dialog (or save its state)? It will then by default match the parameters as they are actually passed to the compiler.

It seems then that the relative filenames in the debug info are the cause of the IDE opening the same file twice when using symlinks.

Since this is a major annoyance I think this behaviour should become configurable in FPC.

Michael.

Michael Van Canneyt
to Jonas Maebe: How directories are stored in the debug info is generally defined by the debug info…
Jonas Maebe wrote 14 lines
On 26/03/14 17:23, Michael Van Canneyt wrote: > At the very least I think that we should be consistent and use the same > filenames in link script and debug info ? How directories are stored in the debug info is generally defined by the debug info format itself. E.g. the DWARF standard explicitly mentions that search directories for source files can be either relative or absolute to the current working directory (there's even an attribute to specify a default CWD, so that the files can be easily found on the compilation system itself even when the program is started from another location). Specifying the directories in a relative way means that you can easily copy over a debug binary and all related source code from one system to another (or move things around) without breaking anything. For that reason, we explicitly add relative directory names to the DWARF debug info rather than absolute ones.

As I wrote to Tomas:
This assumes that the directory structure on the other system is identical.

An assumption that doesn't hold between e.g. 64-bit debian and fedora.

But I am not asking to change the way things are done unconditionally, I understand your reasoning.

Show quoted text 15 lines
For linking, we use absolute paths in link.res to ensure that the parameters given by the programmer to the compiler (such as -Fl and -XR) are taken into account when linking, and that the linker doesn't accidentally link in some other library with the same name that it found in its default search path. I don't see what the advantage of being "consistent" between the generation of debug info and linker scripts would be. Making the paths in the debug info absolute would make things less flexible without increasing usability (I've never seen bug reports about problems with the wrong source files being found). Making the paths in the linker script relative would add a lot of complexity (the compiler first searches for the library files on the files system, and then it would have to make the paths relative to the CWD for no gain) and special cases (when linking on a remote system, the compiler has no clue about what the relative position of e.g. /usr/lib will be compared to the CWD, so then you would still have to use absolute paths).

The problem is easy to reproduce, as Mattias demonstrated.

During debugging, Lazarus consistently opens a file twice (or doesn't find it at all) when symlinks are used in the path to the project file, because there is a mismatch between the relative path as lazarus sees it (logical), and the relative path as FPC sees it (physical), and lazarus uses the debug info to open the file.

That's what started this whole thread.

Specifying source files in the debug info using full paths would solve the issue.

I'm not advocating to always do this, I understand your reasoning for having the current behaviour.

I'm advocating to make this behaviour configurable:

Most people do not copy the files to another system to debug, for them the current exercise is pointless. I suspect much more people use symlinks, for them it would be a convenience to have everything absolute. It would avoid problems. You have the names in absolute form available anyway (for the linker), so it should not be a major problem to make the behaviour configurable.

The alternative is that Lazarus also resolves the directory of the project, as FPC does. (It probably needs to resolve all other files in the process, which is a big effort, but that is conjecture on my part)

Michael.

Michael Van Canneyt
to Michael Van Canneyt
Michael Van Canneyt wrote 8 lines
The problem is easy to reproduce, as Mattias demonstrated. During debugging, Lazarus consistently opens a file twice (or doesn't find it at all) when symlinks are used in the path to the project file, because there is a mismatch between the relative path as lazarus sees it (logical), and the relative path as FPC sees it (physical), and lazarus uses the debug info to open the file.

To demonstrate that the problem is real:

Attached is a screenshot. Remark that the same files are opened twice.

How did I get there ? Simple:

I have /home/michael/FPC/trunk as my copy of FPC trunk.

I have a symlink /home/michael/fpc pointing to the trunk directory.

I opened a project /home/michael/fpc/utils/pas2js/pas2js.lpr (used the symlink), with all files in /home/michael/fpc/packages/fcl-js/src opened in the ide. I put a breakpoint in jsparser.pp.

I hit run, and when hitting the breakpoints, Lazarus opens the source files AGAIN, but now with base path /home/michael/FPC/trunk, and appends a (2) after the filename.

Extremely annoying, and since I do a lot of work in trunk, I stopped using the symlinked directories when opening files.

Just to show that this is not some freak occurrence, but a major PITA.

Michael.

Jonas Maebe
to Michael Van Canneyt: As I wrote to Tomas: This assumes that the directory structure on the other system is…
Michael Van Canneyt wrote 14 lines
On Wed, 26 Mar 2014, Jonas Maebe wrote: > Specifying the directories in a > relative way means that you can easily copy over a debug binary and > all related source code from one system to another (or move things > around) without breaking anything. For that reason, we explicitly add > relative directory names to the DWARF debug info rather than absolute > ones. As I wrote to Tomas: This assumes that the directory structure on the other system is identical. An assumption that doesn't hold between e.g. 64-bit debian and fedora.

The global directory structure of a system is irrelevant. Surely the directory structure of the lazarus or fpc source tree itself is identical on both systems. Those lazarus and fpc source directories themselves can be located anywhere, exactly because all paths to source files are relative to base directories rather than absolute.

But I am not asking to change the way things are done unconditionally, I understand your reasoning.

I don't understand yours :)

Show quoted text 6 lines
During debugging, Lazarus consistently opens a file twice (or doesn't find it at all) when symlinks are used in the path to the project file, because there is a mismatch between the relative path as lazarus sees it (logical), and the relative path as FPC sees it (physical), and lazarus uses the debug info to open the file.

Well, it's the first I've heard about this problem and I don't remember seeing any bug reports about it either. I never use Lazarus to debug though.

I'm advocating to make this behaviour configurable:

File a bug report/feature request :) I also don't know by heart the ramifications for Stabs and/or Stabx. And e.g. for the JVM target, adding any kind of directory information is forbidden by the specifications.

Jonas

Michael Van Canneyt
to Jonas Maebe
Jonas Maebe wrote 22 lines
On 26/03/14 19:07, Michael Van Canneyt wrote: > > On Wed, 26 Mar 2014, Jonas Maebe wrote: > >> Specifying the directories in a >> relative way means that you can easily copy over a debug binary and >> all related source code from one system to another (or move things >> around) without breaking anything. For that reason, we explicitly add >> relative directory names to the DWARF debug info rather than absolute >> ones. > > As I wrote to Tomas: > This assumes that the directory structure on the other system is identical. > > An assumption that doesn't hold between e.g. 64-bit debian and fedora. The global directory structure of a system is irrelevant. Surely the directory structure of the lazarus or fpc source tree itself is identical on both systems. Those lazarus and fpc source directories themselves can be located anywhere, exactly because all paths to source files are relative to base directories rather than absolute.

Correct, UNLESS you use symlinks.

Show quoted text 5 lines
> But I am not asking to change the way things are done unconditionally, > I understand your reasoning. I don't understand yours :)

There is only a problem in the case of symlinks.

Show quoted text 10 lines
> During debugging, Lazarus consistently opens a file twice (or doesn't > find it at all) when symlinks are used in the path to the project file, > because there is a mismatch > between the relative path as lazarus sees it (logical), and the relative > path as FPC sees it (physical), and lazarus uses the debug info to open > the file. Well, it's the first I've heard about this problem and I don't remember seeing any bug reports about it either. I never use Lazarus to debug though.

If you don't use symlinks, there will not be a problem.

But I have first complained about the problem in lazarus already many years ago. That there still is a problem should be clear from the screenshot I sent in my other mail.

Show quoted text 6 lines
> I'm advocating to make this behaviour configurable: File a bug report/feature request :) I also don't know by heart the ramifications for Stabs and/or Stabx. And e.g. for the JVM target, adding any kind of directory information is forbidden by the specifications.

Well, I have been playing with GDB and bash/tcsh on the command-line.

I cannot reproduce the problem using command-line tools, which reinforces my belief that FPC is right in using physical (and not logical) paths. That in turn means to me that the IDE should simply also resolve the directories of projects to physical directories.

(unless Mattias can still make a convincing case for another solution...)

Michael.

Mattias Gaertner
to Michael Van Canneyt: The compiler does not do anything special: it determines its current working directory…
[...] The compiler does not do anything special: it determines its current working directory using the RTL, meaning it gets the resolved current working directory.

I described the situation from a user's pov.
Somewhere it should be documented.

BTW, is there already a RTL function to resolve a file name? Using SetCurrentDir is not thread safe.

Mattias

Tomas Hajny
to Mattias Gaertner
Mattias Gaertner wrote 13 lines
On Wed, 26 Mar 2014 18:47:06 +0100 (CET) Michael Van Canneyt wrote: >[...] > The compiler does not do anything special: it determines its current > working directory using the RTL, meaning it gets the resolved current > working directory. I described the situation from a user's pov. Somewhere it should be documented. BTW, is there already a RTL function to resolve a file name? Using SetCurrentDir is not thread safe.

Is there any problem with ExpandFileName, or do you mean something different by "resolving a file name"?

Tomas

Mattias Gaertner
to Tomas Hajny
Show quoted text 17 lines
On 26 Mar 14, at 23:05, Mattias Gaertner wrote: > On Wed, 26 Mar 2014 18:47:06 +0100 (CET) > Michael Van Canneyt wrote: > > >[...] > > The compiler does not do anything special: it determines its current > > working directory using the RTL, meaning it gets the resolved current > > working directory. > > I described the situation from a user's pov. > Somewhere it should be documented. > > BTW, is there already a RTL function to resolve a file name? > Using SetCurrentDir is not thread safe. Is there any problem with ExpandFileName, or do you mean something different by "resolving a file name"?

There many ways to resolve a file name. ;)
I just tried: ExpandFilename('/tmp/link') returns '/tmp/link'. It does not resolve symlinks.

Mattias

Michael Van Canneyt
to Mattias Gaertner: On Wed, 26 Mar 2014 18:47:06 +0100 (CET) I described the situation from a user's pov.…
Mattias Gaertner wrote 10 lines
On Wed, 26 Mar 2014 18:47:06 +0100 (CET) Michael Van Canneyt wrote: > [...] > The compiler does not do anything special: it determines its current > working directory using the RTL, meaning it gets the resolved current > working directory. I described the situation from a user's pov. Somewhere it should be documented.

I will do so.

BTW, is there already a RTL function to resolve a file name?

fpReadLink ?

Or do you mean a complete filename ? No. It would be a welcome addition, though.

Using SetCurrentDir is not thread safe.

What do you mean ? The CWD is process wide, so obviously setting it is not thread-safe ?

Michael.

Mattias Gaertner
to Michael Van Canneyt
Show quoted text 5 lines
[...] > I described the situation from a user's pov. > Somewhere it should be documented. I will do so.

Thanks.

Show quoted text 6 lines
> > BTW, is there already a RTL function to resolve a file name? fpReadLink ? Or do you mean a complete filename ? No. It would be a welcome addition, though.

Yes.

> Using SetCurrentDir is not thread safe. What do you mean ? The CWD is process wide, so obviously setting it is not thread-safe ?

Yes.

Mattias

Tomas Hajny
to Mattias Gaertner: On Wed, 26 Mar 2014 23:19:57 +0100 There many ways to resolve a file name. ;) I just…
Mattias Gaertner wrote 25 lines
On Wed, 26 Mar 2014 23:19:57 +0100 "Tomas Hajny" wrote: > On 26 Mar 14, at 23:05, Mattias Gaertner wrote: > > On Wed, 26 Mar 2014 18:47:06 +0100 (CET) > > Michael Van Canneyt wrote: > > > > >[...] > > > The compiler does not do anything special: it determines its current > > > working directory using the RTL, meaning it gets the resolved > current > > > working directory. > > > > I described the situation from a user's pov. > > Somewhere it should be documented. > > > > BTW, is there already a RTL function to resolve a file name? > > Using SetCurrentDir is not thread safe. > > Is there any problem with ExpandFileName, or do you mean something > different by "resolving a file name"? There many ways to resolve a file name. ;) I just tried: ExpandFilename('/tmp/link') returns '/tmp/link'. It does not resolve symlinks.

Correct. However, the compiler doesn't use anything else as far as I know. This kind of suggests that you shouldn't get a different behaviour between the compiler and the IDE using this in my view? As an example, if you're inside '/tmp/link' and call ExpandFileName('../somebuggyfile'), it should give you the same result as show by the compiler in its error messages when compiling this ../somebuggyfile.

Tomas

Michael Van Canneyt
to Marc Weustink: This will fail if the file is located on some smb or nfs mount. Marc
Marc Weustink wrote 36 lines
Michael Van Canneyt wrote: > > > On Wed, 26 Mar 2014, Mattias Gaertner wrote: > >> On Wed, 26 Mar 2014 15:33:38 +0100 (CET) >> Michael Van Canneyt wrote: >> >>> [...] >>>>> So ? You just need to check the inode. >>>> >>>> Is there a function to list all files pointing to an inode? >>> >>> Actually, you just need to know if 2 filenames point to the same inode. >> >> And what 2 files should I check? There are thousands of files in >> the search path. > > To avoid duplicates, only the files that are already open in the IDE > must be checked. > For the others, it is irrelevant how you opened them. > > There are 2 scenarios: > > 1. A file with a similar name (filename part matches) is already open in > the IDE. > If you collect the inode when opening, there is no problem (I'm > guessing you already do a stat on open. > 2. No such file is opened: > You can open it using whatever path. > > The next time a file must be opened, you check the file at the 'locical' > location, and if its inode matches an opened file, you know it is open > already. This will fail if the file is located on some smb or nfs mount.

I know. There are some other cases where it will fail.

After all the tests I did, it seems to me that the easiest solution is to resolve the project filename (as FPC does), and work from there.

Michael.