Hello,

Delphi 2010 has changed the RTTI mechanism, and it is said to be much more usable than the previous one.
(see here for example: http://www.malcolmgroves.com/blog/?p=476)

Some interesting tools are using this new approach, like DelphiOnRails: http://code.google.com/p/delphionrails/

Currently, the FPC doesn't implement this new mechanism (See Jonas' answer to my previous post, badly sent :-( ).
But it might be possible to add this mechanism, by making patches to FPC.

Would it interest somebody to have this new functionality?

Best regards,
Thierry

Thierry Coq wrote

Delphi 2010 has changed the RTTI mechanism, and it is said to be much more usable than the previous one. (see here for example: http://www.malcolmgroves.com/blog/?p=476)

I would said extended what they had before.

Currently, the FPC doesn't implement this new mechanism (See Jonas' answer to my previous post, badly sent :-( ). But it might be possible to add this mechanism, by making patches to FPC.

Yes.

Would it interest somebody to have this new functionality?

Yes.

Best regards,
Paul Ishenin

Indeed, it's an extension of what existed before.

I agree with Paul. The FPC team is certainly interested in having this, but finding time to implement this is another matter.

If you want to look into it, I think Paul can help on the compiler end, and I'm willing to help with the extra RTL functionality.

Michael.

On 11/12/2010 11:53, Michael Van Canneyt wrote:
...

If you want to look into it, I think Paul can help on the compiler end, and I'm willing to help with the extra RTL functionality.

How can I help?
I know a lot how the Delphi RTTI works, but I don't have a Delphi 2010 version (too expensive)!
I'm going to look into the specs and what are the differences with FPC's current implementation.
I can also download a DelphiOnRails version and see how it is used in practice.

Thierry

Hello,
I've started looking into Delphi 2010 RTTI scheme. For example, superobject uses it. Delphi 2010 declares a whole new API for RTTI. Would it be possible to reproduce the interface of this API and implement it using the existing RTTI system in FPC? Would it be legal?

I'm investigating further.
Thierry

Thierry Coq wrote

I've started looking into Delphi 2010 RTTI scheme. For example, superobject uses it. Delphi 2010 declares a whole new API for RTTI. Would it be possible to reproduce the interface of this API and implement it using the existing RTTI system in FPC? Would it be legal?

Yes, that would be legal.

Jonas

Yes, it will be possible. But there are few problems:
1. compiler does not produce required information. It already produces more than was needed for D7 compatibility but still less than D2010.
2. new switches needs to be added to compiler to tune what kind of info to generate in rtti
3. another thing is rtti unit. At the moment it can't be compiled in fpc because it intensively uses extended record syntax. Good thing is I'm currently implementing this syntax extensions in FPC. You already can play with methods and visibility sections in records but it is still not enough to be able to compile rtti unit.

Best regards,
Paul Ishenin

Paul,

Thanks for working on adding methods and visibility to record types. Could you also look into fixing generics? That is, to allow for generic constraints and remove the redundant generic and specialize keywords. For example:

{code}

type
  TList<T> = class { ... interface code ... }  end;
  TIntegerList = class(TList<Integer>) end;
  TObjectList<T: TObject> = class(TList<T>) end;
var
  Dates: TList<TDateTime>;
  Integers: TIntegerList;
  Threads: TObjectList<TThread>;

{code}

Note the lack of redundant generic and specialize keywords. It is obvious the type is generic by virtue of the angle brackets after the type identifier. Also notice the type constraint after T in the object list type declaration.

Thanks!

Anthony Walter wrote

Thanks for working on adding methods and visibility to record types. Could you also look into fixing generics? That is, to allow for generic constraints and remove the redundant generic and specialize keywords.

I hope someone else will do this. At least this has low priority for me.

Best regards,
Paul Ishenin

Hi,

what is so different on this mail than the one I sent a few months ago to the fpc-devel list?

anyway, it would require some packages (bpl) language/RTE additions as well. That was where I was hanging when spare time slipped away a few months back. I started on rtti unit with help in embarcadero's online wiki. Btw, during those few weeks, I noticed the XE updates. In theory it is possible to do some AOP in Delphi XE now with TVirtualMethodInterceptor or something like that.

the unit wasn't all that useful yet (mostly interface) and probably partially obsolete due to the XE upgrade :-).

kind regards,
Dimitri Smits

Would you care to explain why you think so ?

The 2 features have nothing to do with each other whatsoever. You can perfectly add extended RTTI to FPC without package support.

It may be that package support (if/whenever it appears) may need to take into account RTTI, but certainly not the other way round.

Michael.

Sure, but I was talking about the RTTI unit and API compatibility. (hierarchical starting point is program, then package, then unit, then type)

more pressing would be:
- list of units in program (only once, hence the packages or monolythic executable)
- list of types/vars/methods in that unit

a TClass should be able to get its unitname as well.

Your remark on the package support is invalid. The other way around IS valid as well, since the api of the rtti unit has some dependencies. If Packages are ever implemented, then the api must be corrected. Likewise, some parts of the api are made from their point of view (embarcadero's), where an executable is a package, as well as every bpl, and each add their unit-info's to the RTE on load of that "package". From there a logical hierarchy is possible, which shows a bit in the rtti unit api. It all depends on the direction of the relationship (tclass.unitname, tunitinfo.packagename, but also rtticontext.findtype('unit.type')) which makes the implementation a breeze or a bit harder.

from http://docwiki.embarcadero.com/VCL/en/RTTI.TRttiContext

  TRttiContext = record
  public
    class function Create: TRttiContext; static;
    procedure Free;
    function GetType(ATypeInfo: Pointer): TRttiType; overload;
    function GetType(AClass: TClass): TRttiType; overload;
    function GetTypes: TArray<TRttiType>;
    function FindType(const AQualifiedName: string): TRttiType;
    function GetPackages: TArray<TRttiPackage>;
  end;
where TArray<TRttiPackage> and TArray<TRttiType> is btw a generically defined => TArray<T>=array of T;

Since Delphi can have only one instance of a unit in memory (error on load otherwise regarding duplicate units), and can load/unload dll/bpl at runtime, the entire FindType method starts with the packages and the onload/onunload hooks must take into account that that array gets updated/invalidated. bidirectional dependencies are to be avoided, but here we are... make without packages => change interface, change trivialness of implementation, change rtti on types; implement packages later on => reimplement, reinstate interface, change rtti again on types (if still possible).

kind regards,
Dimitri Smits

Dimitri Smits wrote

a TClass should be able to get its unitname as well.

FPC already supports TClass.UnitName.

Regards,
Sven

So ?

Then we'll correct the API as far as needed. It's just a matter of installing some hooks now (they'll probably be in Delphi's RTTI API anyway). We did the same for threads, resourcestrings, variants and widestrings, after all.

Therefore, until the time someone finds a cross-platform way to implement packages, I see no reason not to start on RTTI. They can perfectly be implented independently or even in parallel.

Michael.

22.12.2010 21:52, Michael Van Canneyt пишет:

Wine has ability to load windows libraries on linux. This technology could be used to implement cross-platform packages.

In our previous episode, said:

.. into windows programs.

This technology could be used to implement cross-platform packages.

Yes, run the FPC/win32 binary in Wine :-)

23.12.2010 17:52, Marco van de Voort пишет:

I`ve heard that some cross-platform projects use this to build cross-platform plug-ins.

It would create an unwanted dependency on wine.
For such a basic functionality, that's simply out of the question.

Michael.

Am 23.12.2010 16:20, schrieb

Yeah... and only works on i386- and x86_64-linux.

Runtime packages will come sooner or later, maybe not this year (ok... that's hard to beat ^^) and maybe not next year. But when they come they'll be implemented in a good cross platform way without such hackish approaches like requiring Wine or some other big dependencies.

Regards,
Sven

24.12.2010 17:31, Sven Barth пишет:

I mentioned WINE only as as example of possibility to use native binaries from one platform in another. FPC should use own format of dynamic library and own dynamic linker code embedded into executable. This solution could be crossplaform.

Regards,
Alexander

Am 09.01.2011 22:03, schrieb

No, FPC should rely on the operating system for dynamic linking. There is no use in duplicating a functionality that is already there. One "just" needs to spot all problems that might arise on different platforms when using dynamic libraries (e.g. symbol resolution on Windows vs. ELF based systems).

Regards,
Sven

10.01.2011 12:05, Sven Barth пишет:

If so, there must be an executable format supported by all FPC target platforms natively. I don't know any such format. We have to duplicate some OS functions to make things crossplatform.

Alex.

Why ?
There is no need for that; You can perfectly use the OS functionality. All you need to do is add a layer on top which hides the OS specifics. Borland could do it, so we can do it too.

The main problem is the Windows target where a library is more like a self-contained exe. On Linux (and probably Mac OS), libraries already work as packages are intended to work.

Michael.

If so, there must be an executable format supported by all FPC target platforms natively. I don't know any such format. We have to duplicate some OS functions to make things crossplatform.

I don't think that is what Sven meant. I think the executable format would be that of the native platform. However, the interface to the library (the exported symbols and such) would be a common interface that abstracts away the specifics of the underlying OS Library loading routines.

M

10.01.2011 12:50, пишет:

Delphi package is standart DLL with PE file format - it is not portable without things like Wine for example.

Simply spoken, yes, that's what I meant.

Regards,
Sven

10.01.2011 12:58, Matt Emson пишет:

But I want packages to be binary portable between OS (on target processor architecture)
Alex

2011/1/10 :

> But I want packages to be binary portable between OS (on target processor architecture)

I don't think that is feasible, unless you don't use any OS features.

Vincent

Exactly.

Even just because FPC supports multiple CPUs; You can't use an i386 package on SPARC or ARM.

So you'll always have to recompile your package for all platforms that you want to support.

Michael.

Am 10.01.2011 11:00, schrieb

FPC packages won't use PE format on non Windows systems, but Mach-O on Mac OS X and ELF on Linux.

The main point with packages is that a package is a libary that can be loaded at runtime with some compiler magic. It is not important whether this is called DLL or SO or what binary format that library consists of.

Regards,
Sven

10.01.2011 13:05, пишет:

But only for all processors, not for all existing combinations of processor and OS.

Am 10.01.2011 11:16, schrieb

The rtl package is OS specific anyways and as soon as you depend on the rtl package (an fpc package will always depend on it), you need to build your package for any FPC version, processor and OS combination.

10.01.2011 13:19, Florian Klaempfl пишет:

I have a few drafts of another design
http://wiki.freepascal.org/User:AlexVinS/Packages

You should re-read Florian's email, and *fully* understand the consequences.

Your proposal requires that we emulate all OSes on all other OSes, because the basic package (rtl or whatever it will be called) always depends on the OS. There is no way around this.

Michael.

10.01.2011 13:50, пишет:

My solution, in short, is that packages should have OS independent interface to RTL built into executable visible to packages as RTL built as c package (with is a bridge to real RTL).

I understood that. Assuming you can make this interface (which I don't believe), your solution is still not realistic:

And how will you make a package that uses a os-specific function OS independent ? (for instance, a package with a control that uses a WinAPI call.)

So a package with the LCL is by definition impossible.

Like I said, your proposal requires that we emulate all OSes on all other OSes.

Michael.

To split some hairs ;)

Level the OS's with an addition (or some additions) and program the universal application on top of that.
This makes the application lighter AND portable.

Regards / Cees (who has no practical experience with fpc-pascal/Lazarus/Freepascal i.e. a lurker)

Like I said, your proposal requires that we emulate all OSes on all other OSes.

Or create a VM layer that levels the OS level differences - but again, why do this? Creating such a VM is probably a magnitude of complexity over just using the tools already available and requiring a recompilation to target each platform.

Looking from the top (the user), is that not what is done now?

All applications have the same user-interface for different OS's. So by definition what is in-between is part application and part 'normalisation' of the OS.

Regards / Cees (who has no practical experience with fpc-pascal/Lazarus/Freepascal i.e. a lurker)

10.01.2011 14:09, пишет:

Of course nohow. But this package is not portable by desing and could be simply loaded, load needed OS libraries and execute.

So a package with the LCL is by definition impossible.

LCL itself is by design crossplatform. And if component developers don't use OS and widget set functions directly the are crossplatform to. Widget sets (and any platform dependend code) should be isolated in separate packages, LCL application load widget set package at startup and use it. It is only impossible to switch widget with packages Application can use ANY widget set.

In our previous episode, said:

The main problem is the Windows target where a library is more like a self-contained exe. On Linux (and probably Mac OS), libraries already work as packages are intended to work.

Could you explain this?

In our previous episode, said:

But I want packages to be binary portable between OS (on target processor architecture)

That's effectively not possible with all OSes providing such emulation ABI.

Esperanto for OSes so to speak.

In our previous episode, Sven Barth said:

The main point with packages is that a package is a libary that can be loaded at runtime with some compiler magic.

IMHO that is only secondary.

The primary point is that a set of units (+mainprogram) can be divided over multiple binaries transparently. (without changes to the code)

In our previous episode, Matt Emson said:

And that is still assuming you don't need priviliged instructions or calls to do so.

10.01.2011 15:31, Marco van de Voort пишет:

Does "inefficiently" meansperformance decreaseis this case? Is it significant?

In our previous episode, said:

(1) I don't know where I use the word "inefficiently".

(2) to answer the question anyway, the efficiency depends on the difference between the target platforms. Wine (windows on FreeBSD/Linux) is gigantic and noticably slower, but Linux binaries on FreeBSD are next to native. (the whole emulation is 60k and can load Linux dynlinker, libraries etc)

A Delphi DLL on windows is roughly equivalent to a exe. A fully contained binary, no dependencies.

We currently 'emulate' this on linux, in that we stuff the whole rtl and whatnot in the library if one is created.

However, normally, on linux, libraries are a means of spreading code over several libraries, i.e. if you create a library, you don't put libc in it, you link to libc. This concept is closer to packages.

Michael.

If you refer to the word "effectively" stated by Marco, it probably meant "in effect", not "efficiently" (although such a solution would not be efficient either as mentioned by Marco's response). As an example, your proposal would imply that no external calls whatsoever would be possible (including calls to external libraries due to them using different calling conventions, etc.).

Tomas

In our previous episode, said:

DLL's can perfectly fine depend on other binaries.

I assume you mean linker namespaces here.

We currently 'emulate' this on linux, in that we stuff the whole rtl and whatnot in the library if one is created.

You can also emulate it the other way around by having Windows DLLs export all symbols.

However, normally, on linux, libraries are a means of spreading code over several libraries, i.e. if you create a library, you don't put libc in it, you link to libc. This concept is closer to packages.

Systems with linker namespaces allow you to avoid painful problems with duplicate symbols in complex linking situations. These don't happen as quickly in Linux because everything is centrally organized from the distribution (at the price of fragmentation, versioning and the missing of a long time stable binary abi)

But IMHO neither has anything to do with packages.

By the way despite the fact that static linking to external libraries will be impossible, but dynamic linking could be implemented. Anyway I understand that platform independence of code imposes certain restrictions and that restrictions impact on application design. F.e. interface to external platform dependent systems should be isolated in separate package (as I wrote about LCL and its widget sets). But I think it makes design better.

Alex

What you are proposing is something similar to Google Native Client (http://code.google.com/p/nativeclient/ ), except that they also added code verification because it's intended for running native applications downloaded from websites.

That is not directly related to packages, but rather to a completely new run time environment with its own ABI, object format etc. The result of having such an environment is indeed that you can have compiled binary code that is portable across different operating systems, but that's outside the scope of adding packages support to FPC.

You first have to define such a platform (or you can take the Google Native Code platform), then add RTL and compiler support to FPC for that platform, and subsequently any program or package compiled for that platform by FPC will run on it. Reinventing such a platform and embedding it in FPC and its run time library falls outside the scope of the FPC project.

Jonas

10.01.2011 17:40, Jonas Maebe пишет:

Interesting link. Thanks.

That is not directly related to packages, but rather to a completely new run time environment with its own ABI, object format etc.

object format is the main question as for now.

The result of having such an environment is indeed that you can have compiled binary code that is portable across different operating systems, but that's outside the scope of adding packages support to FPC.

OK

Yes, my implementation of packages planed to be just a platform from the compiler side. So then "native" support of packages will be added in FPC.

I think he means a hybrid of the old 'overlay' loadable code.

and I also think he means something like:
- if you want to have a "package" for a 386 cpu, then you only have to compile it once
- you can "link" in your code regardless of os, as long as it is a 386 cpu with the same code

ofcourse I cannot agree more on the 'use what the os gives you'. ???.dll/lib???.so

what I don't understand is the remark on "windows dll is a pe executable and linux .so's are how packages are supposed to be". Anybody can clarify that statement?

kind regards,
Dimitri Smits