Hi guys,
I've spent a couple of days porting FPC to DragonFly BSD with some success.

uname -a

DragonFly banshee.x.com 3.9-DEVELOPMENT DragonFly
v3.9.0.710.g6792a-DEVELOPMENT #0: Fri Oct 24 18:32:12 CEST 2014

x86_64

bin/fpc -i

Free Pascal Compiler version 2.6.4

Compiler Date : 2014/11/18
Compiler CPU Target: x86_64

Supported targets:
Linux for x86-64
FreeBSD for x86-64
Win64 for x64
Darwin for x86_64
Solaris for x86-64 (under development)
OpenBSD for x86-64 (under development)
NetBSD for x86-64 (under development)
DragonFly for x86-64

Supported CPU instruction sets:
ATHLON64

Supported FPU instruction sets:
SSE64
SSE3

Supported ABI targets:
DEFAULT
SYSV
AIX
EABI
ARMEB

Supported Optimizations:
REGVAR
STACKFRAME
LOOPUNROLL
TAILREC
CSE

Supported Whole Program Optimizations:
All
DEVIRTCALLS
OPTVMTS
SYMBOLLIVENESS

Supported Microcontroller types:

This program comes under the GNU General Public Licence For more information read COPYING.v2

Please report bugs in our bug tracker on:
http://bugs.freepascal.org

More information may be found on our WWW pages (including directions for mailing lists useful for asking questions or discussing potential new features, etc.):
http://www.freepascal.org

Since DragonFly uses the FreeBSD ports collection, the plan was to start with patches there and then get most of them into the repository (x86_64 only). However I hit a snag in the middle of the bootstrapping process and I could use some help. It won't compile because it can't find the units, but it can't find the units because it can't properly find out if a directory exists.

bin/fpc -vut

Configfile search: /home/marino/.fpc.cfg
Configfile search: /usr/local/etc/fpc.cfg
Reading options from file /usr/local/etc/fpc.cfg
Path "./lib/fpc/2.6.4/units/x86_64-dragonfly/" not found Path "./lib/fpc/2.6.4/units/x86_64-dragonfly/*/" not found Path "./lib/fpc/2.6.4/units/x86_64-dragonfly/rtl/" not found Path "/home/marino/.fppkg/lib/fpc/2.6.4/units/x86_64-dragonfly/*/" not found
Path "/usr/lib/" not found
Path "/lib/" not found
Free Pascal Compiler version 2.6.4 [2014/11/18] for x86_64 Copyright (c) 1993-2014 by Florian Klaempfl and others Fatal: No source file name in command line
Fatal: Compilation aborted
Error: bin/ppcx64 returned an error exitcode (normal if you did not specify a source file to be compiled)

I've narrowed down the problem to a malfunctioning strpas command, and further to the "move" command. I've temporarily disabled the assembly routines and I'm using the generic "move" command. What's going on is the move "dest" argument is receiving a different address location than the procedure that calls it gives. So the string copy works, but it's getting copied to the wrong location. See below

gdb bin/fpc

GNU gdb (GDB) 7.6.1
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-dragonfly".
For bug reporting instructions, please see:
<http://bugs.dragonflybsd.org/>...
Reading symbols from /usr/local/bin/fpc...done.
(gdb) set args -vut
(gdb) b generic.inc:886
Breakpoint 1 at 0x403b5f: file ../inc/generic.inc, line 886. (gdb) r
Starting program: /usr/local/bin/fpc -vut

Breakpoint 1, fpc_shortstr_to_shortstr (RES=..., highRES=255, SSTR=...) at ../inc/generic.inc:886
886 ../inc/generic.inc: No such file or directory. (gdb) p sstr
$1 = '/etc/localtime'
(gdb) p slen
$2 = 14
(gdb) p @sstr
$3 = (^SHORTSTRING) 0x658c48 '/etc/localtime'
(gdb) p @res
$4 = (^
OPENSTRING) 0x7ffffffff228
'????'#127#0#0'?????'#127#0#0#27'WD'#0#0#0#0#0'??
Z'#2#1#0#0#0#1#0#0#0'?#'#4#138'?'#129'R'#128#0#0#0#0#0#0#0#0'?????)kT'#0#0#0#0'? ?S&'#0#0#0#0':?qP'#0#0#0#0#8#129'|,'#0#0#0#0':?qP'#0#0#0#0#8#129'|,'#0#0#0#0'6'# 8#0#0#0#0#0#0#4#0#0#0#0#0#0#0#0'@', #0 <repeats 30 times>, #2#16#0#0#0#0#0#0#3#0
#0#0#0#0#0#0#0#128#0 <repeats 30 times>,
#16'????'#127#0#0#149#143'A'#0#0#0#0#0#
0'?t'#0#8#0#0#0
(gdb) s
MOVE (SOURCE=0, DEST=0, COUNT=15) at ../inc/generic.inc:30 30 in ../inc/generic.inc
(gdb) p @dest
$5 = (^formal) 0x7ffffffff3e0
(gdb) p @source
$6 = (^formal) 0x658c48
(gdb)

As you can see, the address of "dest" should be 0x7ffffffff228 but it comes as 0x7ffffffff3e0. The address of "source", a constant, is maintained through the call. For convenience, I've pasted the interesting code below:

   generic.inc: line 879
	procedure fpc_shortstr_to_shortstr(out res:shortstring; const sstr:
shortstring
	var
	  slen : byte;
	begin
	  slen:=length(sstr);
	  if slen>high(res) then
	    slen:=high(res);
	  move(sstr[0],res[0],slen+1);
	  res[0]:=chr(slen);
	end;
   generic.inc: line 25
	procedure Move(const source;var dest;count:SizeInt);[public, alias:
'FPC_MOVE']
	var
	  aligncount : sizeint;
	  pdest,psrc,pend : pbyte;
	begin
	// snipped
	end;

Maybe "dest" is getting passed by value rather than by reference. Does anyone have any idea where I can start looking in the code? gdb won't tell me what source code is responsible for this.

Getting FPC to run as a first-class citizen on DragonFly has been a long-term goal of mine (ENOTIME) but I'm so close so I hope someone can help. I also found a couple of mistakes in generic BSD code so my patches will improve support for other BSDs potentially.

Regards,
John

Such problems usually mean that the stat or statfs record needs adjustments. You can find the "stat" record definition in rtl/bsd/ostypes.inc. statfs is in rtl/<yourosdir>/ptypes.inc

I've narrowed down the problem to a malfunctioning strpas command, and further to the "move" command.

This is probably an issue of gdb being confused about formal parameters. I think @formalpara will get you the location on the stack of that parameter (it's passed in a register in that case, but FPC will copy it back to the stack locally if you compile without optimisations), not its value.

Parameter passing on x86-64 is identical for all platforms except for Win64, so it would be very strange if it went wrong in such a common routine only on your new platform.

Jonas

"John Marino" wrote

Hi guys, I've spent a couple of days porting FPC to DragonFly BSD with some success.

[...]

> bin/fpc -i Free Pascal Compiler version 2.6.4

I can't help you much with your specific problen, but new targets should be added to the development version of FPC (in this case 2.7.1) and not the release. Afterall you'll need to port it sooner or later to that version anyway and especially in the current case (2.6.4 vs 2.7.1) quite some changes happened.
Additionally we (core developers) only work with the development version anyway (the release compiler is basically only used for compiling trunk), so it will be easier for us to help you as well.

Regards,
Sven

Jonas Maebe wrote

Such problems usually mean that the stat or statfs record needs adjustments. You can find the "stat" record definition in rtl/bsd/ostypes.inc. statfs is in rtl/<yourosdir>/ptypes.inc

I think you nailed it. Actually I found an error on that file about half an hour ago and am regenerating everything.

Yeah, after finding the error above, I started wondering how accurate gdb was being. Will post back in a minute.

John

fixing the error in rtl/bsd/ostypes.inc didn't solve this particular issue. I'm pretty confident stat record is good though, I checked it thoroughly last night. However, I have very little confidence in rtl/dragonfly/ptypes.inc where statfs is concerned. In fact, I never touched it.

John

It doesn't matter, I'll just have to do it twice.
I have to patch the latest release -- otherwise we won't get it until the next release and subsequent import into ports. Plus it's more important to get FPC bootstrapped at this point.

I think the patches would apply to a dev branch without too much trouble though.

John

In our previous episode, Jonas Maebe said:

And
- getcwd (part of truename directory expansion)
- opendir/readdir and friends.

If something is wrong it sometimes shows up in ptrace.

getcwd is a likely candidate since afaik that changed FreeBSD 5+

Thanks, Marco.
The TStatfs structure needed updating but it wasn't the cause of my issues.

In case people are interested, I pushed my patches publicly: https://github.com/DragonFlyBSD/DeltaPorts/commit/117fa674817cc090a11b058d2f7bbf8e96b2e286

Don't worry about the patches to the generated makefiles. I had to do that based on the ports build order. I know those will get regenerated when they get incorporated into the dev branch.

The patch that defines OLDBINUTILS to avoid assembly is not a permanent patch (Dragonfly has binutils 2.24 in base, not an issue).

I'll check out your hints!

John

getcwd seems to work okay, I don't think that's a problem. This is where the problems start (cfileutl.pas, PathExists function):

if allowcache then
  Result:=DirCache.DirectoryExists(hs)

The DirCache.DirectoryExists function always returns false, even on valid paths.

Still looking...

John

more importantly,
if allowcache then
          Result:=DirCache.DirectoryExists(hs)
        else
          Result:=SysUtils.DirectoryExists(hs);

SysUtils.DirectoryExists(hs) returns false too.

so I'm back at rtl/unix/sysutils

Function DirectoryExists (Const Directory : String) : Boolean;
Var Info : Stat;
begin
  DirectoryExists:=(fpstat(pointer(Directory),Info)>=0) and
fpS_ISDIR(Info.st_mode);
end;

John

so DirectoryExists is only hit once -- on a file
(/usr/local/etc/fpc.cfg). The DirCache.DirectoryExists function seems to be used instead for all directories, all the time.

John

If you get it working with trunk fast enough then there might even be a remote chance that it will be on the next release, that will be branched soon (no guarantees though).

I think the patches would apply to a dev branch without too much trouble though.

In trunk there are quite some changes in the RTL, because of the introduction of codepage aware strings and RawByteString. I don't know in how far the specialized Posix RTLs are affected there (e.g. Linux, FreeBSD, etc. instead of the base directories bsd and unix), but you should be aware that it might involve a bit more work.

Regards,
Sven

If I delete "{$define usedircache}" from cfileutl.pas, then FPC can find the directories and it is is able to build helloworld program natively.

So it seems the problem is limited to dircache.
I tried that last night but it didn't work, probably because of the bad statfs type.

Maybe I'll go with dircache disabled for now just to see if I can bootstrap it.

John

In our previous episode, John Marino said:

/me points to the fallback code of when fpgetcwd fails in unix/sysdir.inc: function do_getdir.

Thanks, Marco.
In the meantime, FPC can fully bootstrap when dircache is disabled. I completely the package and it can build itself and 81 other FPC packages successfully.

http://muscles.dragonflybsd.org/bulk/bleeding-edge-potential/20141118_112715/

There were 9 failures but they could have been broken anyway or the quick-n-dirty replacement hack I did to make x86_64-dragonfly a valid OS for the makefile backfired.

I definitely want to figure out why cachedir isn't working, but at least FPC seems functional without it.

John

Hi Sven,
I'm mainly done.
I was planning to provide 3 patch sets:
1) rtl/dragonfly (all new, minus Makefile.fpc)
2) changes to existing files
3) Regenerated Makefile and *.inc files

I am hesitating on 3). I got the trunk fpcmake built and it generates Makefiles nicely (the -r option doesn't work, so I wrote a script for that). However there are tons of these, and some of the makefiles aren't even current now (e.g. they weren't regenerated for aros, mipsel, etc). So I am guessing I should not provide these and just the assume the committer will regenerate the makefiles.

What about any .ini files? Will somebody else regen the derivative .inc files too?

John

In our previous episode, John Marino said:

Yes. Keeping a list of such files to regenerate is a good thing though. (the .ini/inc files, don't bother with the makefiles)

Sounds good, though the generated Makefiles are indeed not needed :)

I am hesitating on 3). I got the trunk fpcmake built and it generates Makefiles nicely (the -r option doesn't work, so I wrote a script for that).

Normally it should work if you do "utils/fpcm/fpcmake -Tall -r" inside the main source directory. If you do this on DragonflyBSD and it doesn't work then you might still have a bug somewhere (in principal it should work, because I just used it this weekend on Mac OS X which is BSD as well).

However there are tons of these, and some of the makefiles aren't even current now (e.g. they weren't regenerated for aros, mipsel, etc). So I am guessing I should not provide these and just the assume the committer will regenerate the makefiles.

Yes, the committer will regenerate them (most likely in a separate commit).

Regards,
Sven

Hello,
I've applied the DragonFly BSD support to the trunk. The 2.6.4 compiler can bootstrap it and it builds itself and helloworld.

I've split it into 3 patch sets for the convenience of the committer:

http://leaf.dragonflybsd.org/~marino/df-fpc/dfly-support-existing-most.diff

http://leaf.dragonflybsd.org/~marino/df-fpc/dfly-support-existing-packages.diff

http://leaf.dragonflybsd.org/~marino/df-fpc/dfly-support-rtl-dragonfly.diff

The first diff are modifications to existing files minus anything in packages. The second diff are modifications to existing files in packages. The third diff is the entire rtl/dragonfly directory minus a makefile.

After committing,
1) all makefile needs need regenerated (and rtl/dragonfly/Makefile generated)
2) utils/fpcm/fpcmake.inc needs to be regenerated

Notes:
A) The change to compiler/cfileutl.pas is not intended to be permanent, but until I figure out why dircache isn't working, it's needed. B) rtl/bsd/ostypes: fixed constants S_IRUSR, S_IWUSR, S_IRGRP, S_IWGRP, S_IXGRP. These were actually incorrect.
C) rtl/bsd/ostypes: changed S_IFMT and friends to octal. They were correct but octal is a lot easier to check than decimal -- headers are written in octal.

I would be very elated if this could get committed prior to the next release as Free Pascal 2.6.4 is already available to via a derivative of the FreeBSD Ports Collection (called DPorts)

Thanks to all who provided advise to get me here.
Regards,
John

John Marino wrote

B) rtl/bsd/ostypes: fixed constants S_IRUSR, S_IWUSR, S_IRGRP, S_IWGRP, S_IXGRP. These were actually incorrect.

Not for Darwin at least:

/* Read, write, execute/search by owner */
#define S_IRWXU 0000700 /* [XSI] RWX mask for owner */ #define S_IRUSR 0000400 /* [XSI] R for owner */ #define S_IWUSR 0000200 /* [XSI] W for owner */ #define S_IXUSR 0000100 /* [XSI] X for owner */ /* Read, write, execute/search by group */
#define S_IRWXG 0000070 /* [XSI] RWX mask for group */ #define S_IRGRP 0000040 /* [XSI] R for group */ #define S_IWGRP 0000020 /* [XSI] W for group */ #define S_IXGRP 0000010 /* [XSI] X for group */ /* Read, write, execute/search by others */
#define S_IRWXO 0000007 /* [XSI] RWX mask for other */ #define S_IROTH 0000004 /* [XSI] R for other */ #define S_IWOTH 0000002 /* [XSI] W for other */ #define S_IXOTH 0000001 /* [XSI] X for other */

Since these are octal numbers, they correspond to the existing (binary notation) constants in the RTL afaics.

Jonas

You're right, I got burned by oct/hex.

John

I have updated the link in place to remove that change: http://leaf.dragonflybsd.org/~marino/df-fpc/dfly-support-existing-most.diff

thanks for resolving that issue,
John

I am wondering what the next step is. Are these patch sets currently under review? Is anything more needed from me?

thanks,
John

John Marino wrote

I am wondering what the next step is. Are these patch sets currently under review? Is anything more needed from me?

It's best to open a bug report at http://bugs.freepascal.org and attach them there.

Jonas

I opened the PR: http://mantis.freepascal.org/view.php?id=27091

This didn't fit a category well, so I just picked "compiler" and title is not the best but I couldn't edit it.

I see a lot of activity in the trunk thus I'm worried about bitrot (and not making the 2.8.0 release) so that's what is concerning me.

Thanks,
John

John Marino wrote

I see a lot of activity in the trunk thus I'm worried about bitrot (and not making the 2.8.0 release) so that's what is concerning me.

Committing large patches right before forking a stable branch is generally not done in any case (it's kind of a contraction in terms). At first sight there is nothing in your patches that can affect existing targets though, so it should be ok. Maybe remove constants that you changed to octal, just to avoid any accidental bug creeping in that way.

Jonas

I understand that POV.
However, at the very least, committing rtl/dragonfly and fpcmake changes and regenerating makefiles would cut the external patches necessary for dragonfly down tremendously (e.g ~500 kb => < 5kb).

I think the octal part is low risk (just a calculator is needed to verify, right?) but I won't object if the committer wants to omit that hunk for the reason stated above.

I think that hunk is the only code change that would affect other (BSD) targets, everything else was preprocessor stuff IIRC.

John

In our previous episode, Jonas Maebe said:

Maybe we should simply branch now, and let the rest of the fpmake release engineering be merged back as necessary?

That would free up trunk.

So a couple of weeks have passed. The PR wasn't claimed. This is a significant contribution; not only is it a fully working target that didn't exist before, but one that will be instantly supported through the FreeBSD ports collection (over 100 packages).

It would be great to understand what is the plan to handle this contribution, the associated timeline and who will be responsible for the commit. I'm in the process of setting up a FPC support team in FreeBSD but obviously if no target exists for our platform, I won't have a reason to do that.

Anyway, if somebody could provide a thoughtful update about the plan for this DragonFly target, I would be most appreciative.

Regards,
John

In our previous episode, John Marino said:

Anyway, if somebody could provide a thoughtful update about the plan for this DragonFly target, I would be most appreciative.

Committed. Sorry for the delay. The purchase of a new tablet ate a bit more time than expected.

In our previous episode, Marco van de Voort said:

Makefiles still have to be regenerated, but since branching is immintent and makefiles are regenerated for both new release as new trunk, I left that still to do.

Understood. Many thanks, Marco!
John