Hi,

I was thinking about extended RTTI, and important RTTI.pas module (good start point with attributes is waiting 3 years). Seems that the only real reason why we don't have yet RTTI.pas in trunk is incompatibility of generated type info (as Jonas said: TAttributeData vs TAttrData, no TAttrEntry).

Current situation seems like big lock for many developers and development. I see 3 solutions:

1. TAttributeData with additional compiler hinting directive "experimental"
2. Moving TAttributeData into implementation section (and related functions accessible through alias&special *.inc for RTTI.pas). Only usage of attributes through RTTI.pas would be allowed.
3. Not merged.

Point 1&2 is fair enough (especially 1). We can change RTTI layout in the future (when Invoke will be implemented). I think that speed of development of RTTI.pas will increase (btw. management operators are perfect for TValue).

For point 3 I am forced to add RTTI branch into NewPascal project which is dangerous for both projects and for me personally. Today I need to read ridiculous messages (for example from Blaise in "Closures / anonymous methods" topic) about "no respect" for my person ^^ because I have Open Source supporting project (as Blaise said: I am schismatic which create unnecessary forks, splitting the community and resources). For me NP is start point for many patches merged back into FPC, libraries and motivation for improvements and for discussions with core team about improvements. *Sooner or later someone will send agent 47 to eliminate me ;) so please rethink decision about RTTI branch*.

Maybe RTTI branch has some hidden flaw?

The Delphi TypInfo unit also exposes the attributes.

Michael.

2016-12-06 17:46 GMT+01:00 Michael Van Canneyt :

The Delphi TypInfo unit also exposes the attributes.

My message was unclear - I mean that we can hide incompatible part of TypInfo. Type TAttributeData is located in TypInfo (sooner or later that part of TypInfo will be corrected - but IIRC for that "Invoke" function is needed). RTTI.pas is rather fine (need to check).

or second option:

compiler hinting directive "experimental" for TypInfo.TAttributeData. Good temporary deal for all (I think)...

anyway any option is much better perspective than ignoring RTTI.pas for long long long time...

We won't need Invoke() function for attributes, since Joost's branch generates a function that calls the attribute class' constructor with the provided values and returns the created instance. This has the advantage that it's a) faster and b) works on all supported platforms whereas Invoke() will not be supported on all platforms at the beginning and additionally requires the parameter RTTI to be in place (basically what's introduced with the Interface RTTI branch).

I've just looked through Joost's branch. One of the biggest problems is the unit list and the list of types contained in each unit. Since this is only introduced in the branch and the introduction of dynamic packages completely shakes that up (I want to be able to look up types of packages that were dynamically loaded after all ;) ). So if we agree that for now TRttiContext either is removed or its methods return Nil and attribute support is also removed for now then we can indeed move the RTTI unit rather unceremoniously to trunk (I'll need to rework Joost's branch anyway).
Since we now also support generic functions we can also add the missing functions of TValue ;) (patches welcome once we've indeed merged it)

Regards,
Sven

2016-12-06 22:49 GMT+01:00 Sven Barth :

Yes I really like all of mentioned advantages!... at the beginning I was a little confused how it works without Invoke in RTTI.pas. Indeed good idea, but if we really like to have Delphi clone of attributes (with logic and typinfo data) then Invoke is needed.

Sure, very welcome. TValue is good for beginning (would be cool to use TValue with MO). RTTI.pas might be good start point for early implementation of "Invoke" for few platforms.

Most Delphi code probably uses the RTTI unit anyway for this. And when playing with the RTTI one needs to keep track of FPC vs. Delphi anyway cause we'll never be 100% compatible (e.g. the Interface RTTI branch does not keep in mind parameters that might be split across multiple locations (like two registers), so that needs to be extended compared to Delphi).
However I think that we'll add the Delphi compatible TAttrEntry, but with an additonal AttrCreate field before the AttrCtor field. The former will then contain a pointer to the generated "fast invoke" function while AttrCtor, ArgLen and ArgData will be Delphi compatible.

Okay then, maybe I'll manage that this week (no promise though ;) ). I personally think that removing TRttiContext for now is the better choice, so people will know not to use it while they might expect valid values in the other case...

I don't know whether an early implementation of Invoke() would be a good idea already as it would need a heavy rewrite once parameter location support is available in the RTTI...

Regards,
Sven

I am not sure this is possible. The structures exposed in the typinfo interfacee mimic the info the compiler generates in the binary.

If the compiler generates different info than Delphi, you cannot retrieve it in a delphi-compatible way.

Trying to generate the exact same information as Delphi is IMHO not the right path.

What is needed is an API that gives you the info contained in the internal structures.

No-one is ignoring RTII. I have contacted Joost on multiple occasions to merge his work into trunk. I get a lot of questions about it. But I cannot do the work myself. If I could, I would. I am waiting for the attributes since a long time.

Michael.

Don't forget that there is some working code of interface "Invoke" and "fake" in mORMot.pas:
- working with FPC_HAS_EXTENDEDINTERFACERTTI or manual-registered RTTI,
- x86 and x64 support on Windows and Linux ABI
- ARM 32-bit and 64-bit (preliminary) support
- well tested under Windows, good coverage under Linux, preliminary testing under *BSD.

"Invoke" is indeed only half the way.
Creating "fake" interfaces are also needed for any modern

It is not using TValue, but currently JSON serialization, working with by-representation transfer (e.g. objects are transmitted by field values, not class instance).
It was design to be extended to support a TValue-like input and output.

It currently lacks some less common targets (CPU & ABI). But could be used as a starting point.

Arnaud

is

it in a delphi-compatible way.

Trying to generate the exact same information as Delphi is IMHO not the

right path.

What is needed is an API that gives you the info contained in the

internal structures.

That's what the RTTI unit is there for. It provides a higher level API to the type information than typinfo does.

> or second option: > > compiler hinting directive "experimental" for TypInfo.TAttributeData.

Good

I've now added that to the list of my near term projects, together with interface RTTI.

Regards,
Sven

2016-12-06 23:39 GMT+01:00 Sven Barth :

100% agree I don't have problem with a little Delphi incompatible RTTI, the bigger picture is more important than small details, but quote from Jonas Maebe ([fpc-devel] Attributes
thread): "some people wanted it to be Delphi compatible, other people did not mind if it was not. Both categories of people were in the core team."

mostly redundant info, but probably best what we can get.

I am waiting :P. Sure, feel free to remove TRttiContext. IMO almost empty TRttiContext has not many sense yet.

I don't know whether an early implementation of Invoke() would be a good idea already as it would need a heavy rewrite once parameter location support is available in the RTTI...

what you mean by "heavy rewrite once parameter location"? Invoke is extremely usefully even without detailed RTTI info about parameters, for example for scripting purposes. I don't mean only pascal functions, script is able to call external system API. Very handy.

2016-12-07 8:49 GMT+01:00 Michael Van Canneyt :

I think it is possible but solution proposed by Sven is good enough :). As was mentioned in my previous message to Sven:

100% agree I don't have problem with a little Delphi incompatible RTTI, the bigger picture is more important than small details, but quote from Jonas Maebe ([fpc-devel] Attributes
thread): "some people wanted it to be Delphi compatible, other people did not mind if it was not. Both categories of people were in the core team."

Delphi incompatible info was mentioned as serious problem for merging RTTI branch into trunk.

No-one is ignoring RTII. I have contacted Joost on multiple occasions to merge his work into trunk. I get a lot of questions about it. But I cannot do the work myself. If I could, I would. I am waiting for the attributes since a long time.

Glad to hear that. RTTI together with closures is "must to have" feature. Extended RTTI + closures is the last missing element for many users to escape from Delphi.

I was hoping to avoid the RTTI unit, it has a horrible, Horrible, HORRIBLE interface in my opinion. (probably again some misguided .NET clone attempt)

But no matter:
If the info is there, I can write my own interface which is more to my liking. I don't care about Delphi compatibility in these matters. What matters is that the concept is there, the exact API I will creatre form myself.

Great. Then there is some chance it will happen in the near future. I hope you still have a lot of vacation/workfree days left :-)

Michael.

I am one of the people who do not mind if it is not compatible. As I said:

Trying to generate the exact same information as Delphi is IMHO not the right path.

If the RTTI unit is there, people who want Delphi compatibility can and should use that.

People with a more low-level mindset (such as me) will be able to work around the low-level details.

Michael.

Hmh. That should read 'create for myself', obviously.

Michael.

2016-12-07 9:45 GMT+01:00 Arnaud Bouchez :

Good idea with "mORMotish starting point" (especially that licence is compatible with FPC RTL and we have proven code behind), we can wait for Sven (RTTI.pas module on official trunk). Next we can start on NewPascal new branch "fpc-rtti-invoke". Created directly from FPC trunk mirror ( https://github.com/newpascal/freepascal/tree/freepascal ). It could be merged to NewPascal release directly for tests purposes and for more users and for early access. Finally it could be merged as series of patches to official trunk.

the bigger picture is more important than small details, but quote from Jonas Maebe ([fpc-devel] Attributes

thread): "some people wanted it to be Delphi compatible, other people

did not mind if it was not. Both categories of people were in the core team."

I personally am among those who don't need Delphi compatibility ;)

For now we could leave out the AttrCtor, ArgLen and ArgData fields and only add them if someone really needs them...

extremely usefully even without detailed RTTI info about parameters, for example for scripting purposes. I don't mean only pascal functions, script is able to call external system API. Very handy.

Just checked the declaration of Invoke() and my statement was based on a wrong memory of mine as I had forgotten that it merely takes the calling convention and not the parameter locations. Though it might he necessary to add an Invoke() overload that also takes the parameter locations as this way we'd be able to use Invoke() also with the Amiga syscall functions as there the registers to use are specified as part of the functions declaration.
But aside from that you could indeed already start to implement Invoke(), maybe based on the mORMot code.

Regards,
Sven

attempt)

I can't really comment on that since I had left Delphi before that was introduced.

But no matter: If the info is there, I can write my own interface which is more to my

liking.

I don't care about Delphi compatibility in these matters. What matters is that the concept is there, the exact API I will creatre form myself.

The typinfo unit will definitely expose all the information as well (the RTTI unit will probably simply use the data that typinfo provides).

Regards,
Sven

That's what delphi also does, as far as I can see.

Michael.

Sven Barth wrote

But aside from that you could indeed already start to implement Invoke(), maybe based on the mORMot code.

That code is also in fpc/branches/interfacertti. But as mentioned in earlier threads about this, I think it should be generalised in a way that does not require basically duplicating the compiler's parameter manager for every current and potential future architecture also in the RTL.

A previously proposed way centred around some kind of manager interface (like we have for the heap and widestrings), which would be general enough to support interfacing with libffi (and which hence could be used as a generic fallback for every architecture).

The problem is that I am the one supposed to define this interface (for the RTL manager) and the data structures to support it (i.e., to be generated as RTTI by the compiler), and I really don't care at all for this functionality or the related work. I only intervened in this topic because I don't want to be on the receiving end the next time I add support for a new architecture or OS (or even for existing targets, as the current register/offset approach is insufficient; e.g., it does not support multiple parameter locations for a single parameter).

In fact, the manager approach and/or libffi may not even help. On the one hand, libffi probably does not even need extra information beyond standard RTTI, as it just uses standard ABI-defined entities (8/16/.. bit integer, 32/64/.. bit floating point, vectors, struct with list of fields, pointer, ...) and from that calculates the necessary parameter location information. You would need code in the RTL to convert the standard RTTI to this libffi format though, or do it in the compiler and encode that too in the RTTI.

On the other hand, that would not be enough information for the current approach, since that one relies on the compiler encoding the actual parameter locations in the RTTI and then adding architecture-specific code to the RTL (or currently: program) to interpret these locations. So then you still need to add, maintain and extend the current approach anyway if you want to be able to offer a non-generic solution that does not duplicate the entire parameter manager of the compiler.

Jonas

That's however only the code to generate the required RTTI. Invoke() itself does not yet exist in that branch.

Don't worry, you won't need to do that. Since I'm also in support of the manager architecture as well I can do that, too. Also I'm already aware of the fact that the RTTI generated in the branch is not sufficient for multiple parameter locations; that will be something I'll address when I reintegrate that branch.
Maybe it would be even best to have not a single manager like for threads or widestrings, but instead one for each calling convention (or so) so that we could replace certain calling conventions with internal ones while others continue to use libffi or simply raise ENotSupported if the calling convention can't be used (e.g. oldfpccall) (which would also be the default implementation for new targets).

The RTTI.Invoke() method does not even use the RTTI. It merely has a calling convention as parameter and the parameter values so it indeed would need to basically duplicate the parameter manager of the compiler. It would be the job of the invoke manager however to translate the information provided by the RTTI to its own format. We just need to ensure that we provide sufficient information (e.g. in fact all parameter (sub)locations so that a general call manager could be used)

Regards,
Sven

empty TRttiContext has not many sense yet.

I've got the Rtti unit itself ready, but I still need/want to convert Joost's test to FPC's test framework.
Also I've left the TRttiContext, cause it turned out that it's required for keeping track of created type instances as one doesn't free them after all...

Regards,
Sven

So, the unit is committed. If you extend it, please also extend the FPCUnit test in $fpcdir/packages/rtl-objpas/tests/tests.rtti.pas.

You can compile it like this after a "make all" at the toplevel:

./compiler/ppcx64 -n -Furtl/units/x86_64-linux -viwn -FEwhatever -Fupackages/fcl-fpcunit/units/x86_64-linux
-Fupackages/rtl-objpas/units/x86_64-linux
-Fupackages/fcl-base/units/x86_64-linux
./packages/rtl-objpas/tests/testrunner.rtlobjpas.pp

And then you simply execute the resulting testrunner.rtlobjpas binary. It will print its results on StdOut.

Also when you have extended the unit and provide your changes: please use baby steps, not one big patch to rule them all ;)

Regards,
Sven

2016-12-09 21:53 GMT+01:00 Sven Barth :

Also when you have extended the unit and provide your changes: please use baby steps, not one big patch to rule them all ;)

Thanks for tips for tests. I will start new topic - we need to discuss one critical aspect...