This is a problem reported to me by an IBX user. I have not yet confirmed it (need to find the time to set up the latest development branch of FPC), but the issue looks solid, and is due to the following addition to TObject (copied from the GitLab master branch)

|TObject = class|{$IFDEF SYSTEM_HAS_FEATURE_MONITOR}
strictprivate
_MonitorData : Pointer;
private
functionSetMonitorData(aData,aCheckOld : Pointer):Pointer; inline;
functionGetMonitorData:Pointer; inline;
{$ENDIF}
        ...

Since Firebird 3.0, the Firebird project has provided a file called "Firebird.pas" in order to provide an interface to Pascal (both Delphi and FPC) from a C++ environment. There is an underlying assumption in the interface. That is TObject contains no fields.

The idea is simple enough. There is a single entry point to the Firebird client library with the Pascal signature

function fb_get_master_interface : IMaster; cdecl;

This returns a pointer to a structure that may contain fields but is primarily a list of pointers to cdecl functions. These functions can perform various tasks. In the case of IMaster, they can also return similar structures representing other "interfaces".

On the Pascal side, IMaster is defined as a class with no virtual methods and a single field called "vTable" and this is set to the pointer to the structure returned by fb_get_master_interface. The rest of the class comprises methods used to call functions from the vTable. This makes IMaster look like a normal class and hides the use of function pointers. The code is generated by a program know as "cloop". All Firebird interfaces declared in Firebird.pas follow the same approach.

Given the assumption that TObject defines no fields, it is possible to coerce the pointer returned by fb_get_master_interface to appear as an object "instance" of the IMaster class, and similarly for every other Firebird interface class. This coercion ceases to be valid as soon as a field, such as _MonitorData is added to TObject.

It is probably a valid argument to say that this is Firebird's problem as the TObject definition is an internal data structure. However, this ignores a serious real world problem. A fix or workaround is needed and that has to be in Firebird.pas. It would be a logistic nightmare to try and change the C++ Firebird client DLL/SO.

Also, does anyone know whether a similar problem will arise with Delphi, or is thus just an FPC issue?

One of the following seems to be needed:

a) Rewriting Firebird.pas to set the vTable field explicitly rather than implicitly through coercion - ideally without breaking any user code... (note: no current need to free IMaster etc)

b) Taking a different approach to introducing _MonitorData e.g.

i) Putting in a class helper (is this possible?)

ii) Having a version of TObject (e.g. TUnmonitoredObject) which is the current TObject and making TObject a descendent class introducing _MonitorData. The Firebird classes can then descend from TUnMonitoredObject, either explicitly or as a result of a new compiler directive setting the default class ancestor.

iii) Anything else that works.

I am primarily looking for a solution that will work with IBX, but also one that I can suggest to the Firebird Devs to make generally available.

Tony Whyman

MWA Software

This assumption is not valid in Delphi either. It also has this field, but it is 'hidden' at a not-fixed location.

The real problem is the invalid assumption.

This assumption is invalid for Delphi as well. TObject is not empty there either. The memory layout is simply different.

A fix or workaround is needed and that has to be in Firebird.pas. It would be a logistic nightmare to try and change the C++ Firebird client DLL/SO.

It would be a logistic nightmare in FPC as well.

This field was introduced for Delphi compatibility: The implementation of TMonitor requires this data.

Delphi has this field as well but "hides" the field behind all other actually declared fields.

If you examine the actual memory size allocated for TObject, you'll see that an extra field is actually present.

In FPC this turned out to be not so easy to do without modifying the compiler, so it was decided that the cleanest approach is to actually introduce the field as part of TObject.

The firebase people use a trick to save you some work, based on an invalid assumption. I don't think this is sufficient reason to expect the FPC team to adapt their compiler.

The correct solution is to change the cloop program to produce a pascal record definition with procedural type fields at the correct offsets. It should be perfectly usable without too many changes.

So something like (adapt to actual procedure types)

Type
   RMaster = Record
     init : procedure(xyz:sometype);
     // etc
   end;
   IMaster = ^RMaster;
   // optionally; although it seems this already exists?
   TMaster = class
     intf : IMaster;
     procedure Init(xyz : sometype);
     constructor create;
   end;
function fb_get_master_interface : IMaster; cdecl;
constructor TMaster.Create;
begin
   intf:=fb_get_master_interface;
end;
Procedure TMaster.Init(xyz : sometype);
begin
   intf^.init(xyz);
end;

if you cannot change cloop, you can use fcl-passrc or some other tool to actually parse the definition generated by cloop and change it to a record, possibly with a wrapper class for easier access as in the above.

That should not be difficult to do, and it will work in Delphi as well. The advantage is that it will be more robust against future changes as it makes no assumptions about the layout of objects.

Michael.

Michael,

I believe that the diplomatic answer to your response is that it is "disappointing". This is a serious problem and needs to be discussed.

Interfaces between modules written in different programming languages or even built with different compilers are never easy and depend on having published and stable definitions for each data type and structure. This is a common problem for anyone using Pascal given how so many code libraries are written 'C' or C++. Change can imply breaking many user programs and should not be done lightly and for purely internal reasons.

Both object and class type data structures are published in the Free Pascal Documentation and I do not believe that it is unreasonable to expect that the FPC developers will do their best to maintain stability for these structures as they would for any other data type. You seem to think that assuming that the object data structure is a stable and well known data structure that can be relied upon to be the same in each successive compiler version is "an invalid assumption". If that is the case then can we rely upon the stability of any data type and hence of any interface between FPC and other language environments?

In the case of monitor data, you have said that Delphi have implemented this as a "hidden" field. I presume that you mean that this uses a technique such as a negative offset to the object instance pointer. This would certainly be a good way to add system information to an object without destabilising the data structure for the user. Surely the right way to go.

The Firebird OO API may be the only interface library that uses the existing object layout in order to implement the interface. That would surprise me and there may well be more interfaces that are broken by  FPC 3.3.x.

In the case of the Firebird OO API a complete redesign will be needed to work around your change - requiring a separation of the data blocks exchanged via the interface, and the classes supporting them. Even though a code generator is used, there are still about a hundred classes to consider with some used as callbacks. The Firebird.pas unit itself comprises about 14,000 lines of code and comments. You are asking voluntary programmers to devote hundreds, perhaps thousands of hours of time, to build and test a new interface design purely for the benefit of FPC, when the existing design works with both Delphi and FPC and has done so since the early part of the last decade. Until someone puts in this effort Pascal users of Firebird will be restricted to using the old legacy interface (which has not been updated since Firebird 2.5 and misses many important features), keeping with the old version of the compiler, or just moving to Delphi.

And, before going down this path, perhaps you might like to give a heads up of what other changes are in the pipeline that may knowing break interfaces to other programming environments.

Tony Whyman

Tony Whyman wrote

Michael, I believe that the diplomatic answer to your response is that it is "disappointing". This is a serious problem and needs to be discussed.

Thank you for being diplomatic.

We guarantee backwards compatibility (as much as possible) on the language level and API level, but not on the internal structure of objects.

The only structure that is guaranteed to work with C or C++ is the record.

Hence my proposed solution using a record.

Both object and class type data structures are published in the Free Pascal Documentation

You may wish to re-read the documentation and specifically this:

https://www.freepascal.org/docs-html/current/ref/refsu23.html#x73-970006.3.1

'As of version 3.0 of the compiler, the compiler can re-order the fields in memory if this leads to better alignment and smaller instances. That means that in an instance, the fields do not necessarily appear in the same order as in the declaration.'

So your vTable also is in no way guaranteed to be correct, the above explicitly says that your assumptions are invalid.

In the case of monitor data, you have said that Delphi have implemented this as a "hidden" field. I presume that you mean that this uses a technique such as a negative offset to the object instance pointer.

No, actually behind all other fields.

I don't intend to do work on this, since nowhere was there any guarantee that TObject has no fields. As I wrote, this assumption has no basis and this is even documented. It works purely by chance, not through design.

I have given you a possible way to fix it, which is IMO the only correct and future proof way to implement what you need.

Michael.

Hi,

that is bad news

Now all objects become 8 bytes larger?

That is going to waste a lot of space. FPC was supposed to be memory efficient.

Could this not use a global hashmap with tobject as key?

Cheers,
Benito

In 64-bit, yes. Same size as in Delphi.

Could this not use a global hashmap with tobject as key?

Alas no, too slow. TMonitor is supposed to be faster than a lock object (although I have my doubts if this is achieved even in Delphi). Can't have a hash lookup then.

If you don't want to use TMonitor, you can recompile the RTL with OPT=-dFPC_DISABLE_MONITOR then none of this is included: The FPC rtl itself does not use TMonitor.

Michael.