Hi,

I've implemented an optimization that reorders the instance fields of Delphi-style classes (and only of Delphi-style classes) to minimise memory gaps caused by alignment differences and odd sizes. The effect is the same as when you would change the order of the fields in the source code to achieve this effect.

In general, I think this should be safe since unlike records and TP-style objects, a Delphi class is normally never blockwritten to disk or so. There is a switch to disable this transformation, but I'm wondering whether anyone sees a problem with enabling it by default when -O2 or higher is used.

It works fine with at least the compiler and Lazarus without any ill effects, saving a small amount of memory for both (about 2.5MB on 87MB for the Darwin/x86-64 compiler compiling itslf, and 2.5MB on 62.7MB for a Carbon/i386 Lazarus right after startup and loading some source files).

Jonas

PS: a similar reordering could be performed for local variables, although the implementation would be quite different. Additionally, different kinds of reordering, e.g. targeting cache improvements by putting fields/local variables often used together next to each other, could also be implemented in the future.

Hi,

I've implemented an optimization that reorders the instance fields of Delphi-style classes (and only of Delphi-style classes) to minimise memory gaps caused by alignment differences and odd sizes. The effect is the same as when you would change the order of the fields in the source code to achieve this effect.

In general, I think this should be safe since unlike records and TP-style objects, a Delphi class is normally never blockwritten to disk or so. There is a switch to disable this transformation, but I'm wondering whether anyone sees a problem with enabling it by default when -O2 or higher is used.

It works fine with at least the compiler and Lazarus without any ill effects, saving a small amount of memory for both (about 2.5MB on 87MB for the Darwin/x86-64 compiler compiling itslf, and 2.5MB on 62.7MB for a Carbon/i386 Lazarus right after startup and loading some source files).

Jonas

PS: a similar reordering could be performed for local variables, although the implementation would be quite different. Additionally, different kinds of reordering, e.g. targeting cache improvements by putting fields/local variables often used together next to each other, could also be implemented in the future.

I sometimes need "cracker" classes in MSEide and MSEgui to fix bugs and make extensinsions for FCL classes in order to access private fields (examples TDataset, TParam, TField, TFiler, TReader, TWriter, TComponent). Is it guaranteed that the local definitions have the same layout as the definition in the FCL unit?

Martin

Hmm, up to now I listed in the cracker classes fields up to the last private field I need to access so the cracker classes would not brake by changing or adding successive fields in the original classes. I assume now it is necessary to always list all fields of the original class.

Martin

If you you use the same packrecords setting, yes (I doubt any such fcl units contain packrecords directives though, in which case they'll use the default).

Hmm, up to now I listed in the cracker classes fields up to the last private field I need to access so the cracker classes would not brake by changing or adding successive fields in the original classes. I assume now it is necessary to always list all fields of the original class.

If would you want to use that optimization, then yes. "Working around" the type system of the language is however generally asking for trouble, regardless of what reason you do it for.

Jonas

I don't think this is a good idea.

For example while debugging and looking at the memory in raw this would lead to confusion.

It's already bad that Delphi adds invisible fields to classes so they cannot be simply dumped to disk... (virtual method table pointers ?) this would make it even worse.

Bye,
Skybuck.

Indeed. Maybe we can help by making some methods/properties protected ?

Michael.

You are talking seriously? If so I'll provide a list of private FCL fields I need to access in MSEide+MSEgui.

Thanks Martin

I do not necessarily want to use field order optimization but if the FPC RTL is compiled with it I need to compile my cracker classes with optimization too.

Martin

make

extensinsions for FCL classes in order to access private fields (examples TDataset, TParam, TField, TFiler, TReader, TWriter, TComponent). Is it guaranteed that the local definitions have the same layout as the definition in the FCL unit?

Do you have current examples where you fixed bugs? Maybe these can be fixed at the source which might be better than making them protected (though this can still be done for non-bug fields where you need access to certain members)

Regards,
Sven

The only drawback I see that people might sort fields for better cache usage but this is rarely the case imo. So if the optimization can be turned off, it should be fine.

As this might cause an overall gain by better cache utilization i would keep it on by default. But I think it would be best that it can be disabled per class-definition. If somebody tries to do manual cache optimization it will usually only be for a couple of classes. Not the whole project.

So basically, something like

class MyClass = class
{$OPTIMIZATION OFF}
  private
    FHeavilyUsedValue: Boolean;
    FAlsoHeavilyUsedValue: DWord;
{$OPTIMIZATION DEFAULT}
    NotOftenUsed: Byte;
end;

should leave the ordering of the first values alone, while optimizing the rest. Or at least disable reordering for the whole class.

Nico

Martin Schreiber wrote

I do not necessarily want to use field order optimization but if the FPC RTL is compiled with it I need to compile my cracker classes with optimization too.

That's indeed true.

Jonas

That may actually lead to quite some troubles: if someone recompiles the RTL without optimizations, then your class crackers also have to change that setting. And there's no way to detect how the RTL (or in general: units containing classes you are "cracking") has been compiled in your source code. Adding support for something like that is definitely not a road I want to go down -- even a switch to simply treat all "private" fields as "protected" would be less bad (not that I would consider such a switch desirable in any way; it's like adding a switch to allow calling functions only declared in the implementation of another unit).

On the other hand, regardless of whether or not we would make that transformation the default for the release units, you will always get problems if someone recompiles the FPC units using the opposite setting. So that's mainly an argument against including this feature altogether in the compiler, at least without additional hacks to make "class crackers" a sort of supported feature one way or another (which I would be quite reluctant to do).

This demonstrates a real pain of low level languages: there's almost always at least one way in which someone has figured out a way to break the language, killing a lot of possible optimizations and error checking you would normally be able to do without any problem, unless you want to go through the trouble of annoying at least some users (or, in cases like this one, adding extra complexity to turn that breakage into a supported feature).

I once had a discussion with a person who worked on IBM's C compiler, and he said they couldn't even reorder local variables that do not fit in a register because certain large commercial code bases they had to be able to compile "correctly" explicitly relied on buffer overflows from one local variable into the next according to the source declaration order (and annoying those users was not an option).

I guess there are probably no such FPC-compiled programs yet (*), so maybe now is the time to add that before we also get stuck with that particular piece of ballast.

Jonas

(*) except for a few in the FPC testsuite whose purpose is to verify that certain code in fact does not cause a buffer overflow; but those can be easily changed to use all variables of the same size to prevent reordering

Only the setting active at the end of the class declaration (before the "end") will have effect. The setting can be changed via {$optimization orderfields} and {$optimization noorderfields}, just like all other optimization settings.

Jonas

"Jonas Maebe" wrote

I guess there are probably no such FPC-compiled programs yet (*), so

maybe now is the time to add that before we also get stuck with that particular piece of ballast.

Somehow I have the feeling that we all know what one of your upcoming commits will be :P

Regards,
Sven

MSEgui must work with current FPC release so sometimes it is necessary to make workarounds until the next FPC release with the bugfixes. As soon as possible I remove unneeded cracker classes.
I report all FPC bugs I find apart from the compiler crashes by partial compile where reproducing is difficult, needs the whole MSEide+MSEgui source and fixing probably would need construction of a new unit handling in FPC. Bugs which are Delphi compatible are not reported also. ;-)

Currently I need access to the following private FCL class fields because of MSEide+MSEgui extensions:

TComponent:
- FComponents
- FComponentState
- FFreeNotifies

TCollection:
- FItemClass

THandlestream:
- FHandle

TMemoryStream:
- FCapacity

TWriter:
- FPropPath
- FAncestors

TReader:
- FStream
- FLoaded

TDatabase:
- FConnected

TField:
- FValidating
- FValueBuffer
- FOffset
- FFieldno

TFielddef:
- FFieldno

TParam:
- FBound

TDataset:
- FDatasources

I hope I didn't forget anything. Suggestion:
For all FCL base classes which are used in different toolkits and which can't be forked without breaking precompiled third party components, namely the units classes and db, move *all* private fields and methods to protected and mark them as "Use on your own risk, can be changed everytime! Don't cry afterwards.".
I can not fork the listed components because of compatibility with third party components. On the other had I forked SQLdb completely when the workaround burden became too big.

Thanks, Martin

Martin Schreiber wrote

I report all FPC bugs I find apart from the compiler crashes by partial compile where reproducing is difficult, needs the whole MSEide+MSEgui source and fixing probably would need construction of a new unit handling in FPC.

As the compiler sometimes even breaks when partially recompiling itself (from within Lazarus for example) you really don't need to report this ;)

Regards,
Sven

Skybuck Flying wrote

I don't think this is a good idea. For example while debugging and looking at the memory in raw this would lead to confusion.

By knowing the order of the fields, you still don't know their exact offsets. If you want to know their address, print @classinstance.fieldname

It's already bad that Delphi adds invisible fields to classes so they cannot be simply dumped to disk... (virtual method table pointers ?) this would make it even worse.

If you want to program at an assembler level of abstraction, don't use high level language features.

Jonas

A class model is not good if you continuously need to hack your way around it. That is not to say that I will agree with all you ask, of course.

But please give reasons why you think you need to access these fields. Maybe we can, in some cases, find alternatives to solve your problem(s).

Michael.

Martin Schreiber wrote

Currently I need access to the following private FCL class fields because of MSEide+MSEgui extensions:

I have looked at your list. (I sent my previous mail about this without having seen the list)

At first sight, I see no reason to make any of these fields protected.

For example:
TParam.FBound
Is directly accessible through TParam.Bound, so why on earth would you need to have it protected ??

Can you give some explanations at least of what you're trying to do, maybe we can find workarounds for your problems ?

Michael.

Hi,

Michael Van Canneyt wrote

For example: TParam.FBound Is directly accessible through TParam.Bound, so why on earth would you need to have it protected ??

I haven't worked through his list myself, but I had a quick look at one property I had issues with ages ago.... ComponentState is a read-only property and FComponentState is private. Originally I couldn't set csLoading because of the private field variable, but luckily the FPC team was accommodating by adding a protected virtual Loading() method so that I could set/unset the property. They could have just made ComponentState read-write too, but didn't for some reason. Now Delphi and Lazarus never needed this for some reason due to TWriter/TReader being in the same unit (or something like that), but fpGUI doesn't because its loading of forms work very differently.

Anyway, I'm not saying I agree with his whole list... as your example shows, TParam.Bound is a public read-write property already... my example of ComponentState wasn't, but I only needed csLoading. Maybe Martin needs to set some of the other states too and no related methods exist?

Anyway, as you said, maybe justifying some of those items in his list could hep explain the problems more - and suitable solutions could be found.

Well, hopefully, yes.

Usually these things are private for a reason. But maybe there are better reasons for doing it different, of course then I'd like to hear them first :)

Changing them without any justification whatsoever is definitely not going to happen.

Michael.

Jup, I feared it. ;-)

Hmm, this probably is a mistake from me. Was it writeable in all FPC versions?

Can you give some explanations at least of what you're trying to do, maybe we can find workarounds for your problems ?

Before I started to use cracker classes I searched workarounds for weeks or even months already...
If you want you can grep the MSEide+MSEgui sources for "cracker" in order see what I do with them. TComponent, TWriter, TReader for example because in MSEide+MSEgui one can place additional components in an inserted tframe and combination of inherited frames and inherited forms need special handling. The DB components mainly because MSEgui stores string fields as UnicodeString in datasets and because of the direct data access by index without scrolling. The stream components because in MSEgui there is a single hierarchy for file and memory stream so the more specialized descendants (ttextstream, ttextdatastream) can work on files and on memory.

Martin

Michael Van Canneyt wrote

Usually these things are private for a reason. But maybe there are better reasons for doing it different, of course then I'd like to hear them first :)

Yes, and I just wanted to point out that I some time ago I showed a case where I needed to set ComponentState manually but couldn't... The FPC team can't cover all possible use cases. Not everybody is creating a Delphi clone application or framework, so there could very possibly be a valid reason for doing something different to Delphi or Lazarus.

Changing them without any justification whatsoever is definitely not going to happen.

I fully agree.

IMO it's a matter of component users vs. writers. A component writer needs write access to some properties, which a user only should read. The base class designers do not always see such needs, when they implement everything in a single unit with lower protection boundaries. Some members of TControl and TWinControl should be protected, not private, and such a move won't change anything in user land.

DoDi

-----Original Message-----

I wrote 'at first sight', because without explanation I don't know what you try to achieve.

This I don't know. At least, now it is writable.

2 pairs of eyes always see more than one.

If you want you can grep the MSEide+MSEgui sources for "cracker" in order see what I do with them.

That does not tell me why you do something ?

TComponent, TWriter, TReader for example because in MSEide+MSEgui one can place additional components in an inserted tframe and combination of inherited frames and inherited forms need special handling.

Basically, you want to drop a component on a Form containing a frame, and make it's owner the frame ?

The DB components mainly because MSEgui stores string fields as UnicodeString in datasets and because of the direct data access by index without scrolling.

So you take away the concept of cursor. That is a radical change. This assumes that all records are always in memory ?

The string fields should not necessitate the access of private fields, you can just always create TWideStringField instances ?
Maybe we need a function to decide the class of a string field ?

The stream components because in MSEgui there is a single hierarchy for file and memory stream so the more specialized descendants (ttextstream, ttextdatastream) can work on files and on memory.

I'm sorry, I don't understand what this means ?

Michael.

Delphi-style classes (and only of Delphi-style classes) to minimise memory gaps caused by alignment differences and odd sizes. The effect is the same as when you would change the order of the fields in the source code to achieve this effect.

In general, I think this should be safe since unlike records and TP-style

objects, a Delphi class is normally never blockwritten to disk or so. There is a switch to disable this transformation, but I'm wondering whether anyone sees a problem with enabling it by default when -O2 or higher is used.

It works fine with at least the compiler and Lazarus without any ill

effects, saving a small amount of memory for both (about 2.5MB on 87MB for the Darwin/x86-64 compiler compiling itslf, and 2.5MB on 62.7MB for a Carbon/i386 Lazarus right after startup and loading some source files).

the implementation would be quite different. Additionally, different kinds of reordering, e.g. targeting cache improvements by putting fields/local variables often used together next to each other, could also be implemented in the future.

While reading Skybuck's second mail I got a thought: does this optimization also apply on systems that require proper alignment for memory accesses?

Regards,
Sven

The whole point of classes is to offer abstraction. Again: if you don't want abstraction, don't use data structures that offer an abstraction.

I also wonder how much of an optimization it actually is ? Maybe 0.000001% more performance ?

1) as mentioned in the original mail, the current transformation is implemented for saving memory, not for improving performance
2) if it was done for performance reasons, some people already got up to 34% extra performance by doing exactly that: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.34.4009 (download the cached version of the paper, the original link no longer works)

I rarely inspect the binary equivalent of a class instance, so your supposedly optimization is probably not a big deal, for records that would be a different matter since these are used in all kinds of api's and input/output situations.

As mentioned in the original mail, the transformation would only be applied to classes.

It can be used for that, as long as you don't use high level abstractions. The whole point of abstractions to get rid of any guarantees a far as implementation is concerned, in order to increase portability, programmer productivity and compiler optimization opportunities.

Jonas

Sven Barth wrote

does this optimization also apply on systems that require proper alignment for memory accesses?

Well, yes, otherwise the "optimization" would simply consist of removing all alignment padding and there would be no need to reorder the fields.

Jonas

I guess he meant adding a component with
Owner=Form, Parent=nested frame
As far as I remember TReader (or TWriter I'm not sure) does not support this. Lazarus artificially forbids this too. Internally most code supports this.

Martin, what do you mean with "combination of inherited frames and inherited forms need special handling" ?

[...]

Mattias

I see no reason why TWriter would not support this, because it uses GetChildren to get the list of children ? There must be something else.

Michael.

Skybuck Flying wrote

I also wonder how much of an optimization it actually is ? Maybe 0.000001% more performance ?

Cache related optimizations are VERY hard to measure and depend on overall context and used architecture. But as the L1-cache is one of the most performance critical parts in these days cpus the gains of working with cache friendly structures should not be underestimated.

There a a couple of things that need to be taken into account.

1.) Cacheline Utilization: Packing together multiple smaller items into single (machine-)words allows for better utilization of precious cache space. As L1-DCache is usually only 16-32kbytes these days, every byte counts, because saving a single byte can make a difference between using one or two cache lines, which in return will save memory bandwidth and save you from a cache-miss related stall down the line.

2.) Cacheline Streaming: unless your memory bus is a wide as your cacheline it takes multiple cycles to fetch a whole cacheline. The pipeline has to stall till the data in question arrives. If you have relevant data in the end of a cacheline you will have to wait for the whole transaction to complete. So it makes sense to order fields by occurence of access so your first miss will hopefully only lead to the minimal stall time.

While modern CPUs can circumvent/hide some of the cache miss latency with the help of prefetching, out-of-order execution and Hyperthreading it will still lead to a performance penalty. For CPUs without these features (like most ARM cores) this penalty can become substantial, leading to 100 and more stall cycles for a cache-miss.

Nico

Correct.

Martin, what do you mean with "combination of inherited frames and inherited forms need special handling" ?

I don't remember exactly and I don't want to remember because of too much headache. ;-)
Assume frames which had inserted forms (in MSEgui every form or datamodule can act as TFrame, MSEgui has no specific TFrame component), which where inherited and there are descendants which are inserted into a form which possibly has descendants too...

Martin

The cursor concept still exists, direct access without scrolling is an extension.

This assumes that all records are always in memory ?

In tmsebufdataset yes.

The string fields should not necessitate the access of private fields, you can just always create TWideStringField instances ?

TWideStringField probably can not handle the MSEgui storage of UnicodeString with variable length in datasets.
The code of tmsestringfield:

"
[...]
function tmsebufdataset.getmsestringdata(const sender: tmsestringfield;
               out avalue: msestring): boolean;
var
 po1: pointer;
 int1: integer;
begin
 {$ifdef FPC}{$warnings off}{$endif}
 with tfieldcracker(sender) do begin
 {$ifdef FPC}{$warnings on}{$endif}
  if fvalidating then begin
   result:= (fvaluebuffer  nil) and (foffset and 1 = 0);
   po1:= fvaluebuffer;
  end
  else begin
   result:= getfieldbuffer(sender,po1,int1);
  end;
 end;
 if result then begin
  avalue:= msestring(po1^);
 end
 else begin
  avalue:= '';
 end;
end;
[...]
{ tmsestringfield }
destructor tmsestringfield.destroy;
begin
 if fdsintf  nil then begin
  fdsintf.fielddestroyed(ifieldcomponent(self));
 end;
 inherited;
end;
function tmsestringfield.HasParent: Boolean;
begin
 result:= dataset  nil;
end;
function tmsestringfield.assql: string;
begin
 result:= fieldtosql(self);
end;
function tmsestringfield.getasmsestring: msestring;
begin
 if assigned(fgetmsestringdata) then begin
  fgetmsestringdata(self,result);
 end
 else begin
  result:= fieldgetmsestring(self,fdsintf);
 end;
end;
procedure tmsestringfield.setasnullmsestring(const avalue: msestring);
begin
 if avalue = '' then begin
  clear;
 end
 else begin
  setasmsestring(avalue);
 end;
end;
procedure tmsestringfield.setasmsestring(const avalue: msestring);
begin
 if assigned(fsetmsestringdata) then begin
  fsetmsestringdata(self,avalue);
 end
 else begin
  fieldsetmsestring(avalue,self,fdsintf);
 end;
end;
procedure tmsestringfield.readlookup(reader: treader);
begin
 reader.readboolean;
end;
procedure tmsestringfield.defineproperties(filer: tfiler);
begin
 inherited;
 filer.defineproperty('Lookup',@readlookup,nil,false);
end;
{$ifdef hasaswidestring}
function tmsestringfield.getaswidestring: widestring;
begin
 result:= asmsestring;
end;
procedure tmsestringfield.setaswidestring(const avalue: widestring);
begin
 asmsestring:= avalue;
end;
{$endif}
procedure tmsestringfield.setdsintf(const avalue: idsfieldcontroller);
begin
 fdsintf:= avalue;
end;
function tmsestringfield.getinstance: tfield;
begin
 result:= self;
end;
procedure tmsestringfield.Clear;
begin
 setdata(nil);
end;
function tmsestringfield.oldmsestring(out aisnull: boolean): msestring;
var
 statebefore: tdatasetstate;
begin
 statebefore:= tdataset1(dataset).settempstate(dsoldvalue);
 aisnull:= not getdata(nil);
 result:= getasmsestring;
 tdataset1(dataset).restorestate(statebefore);
end;
function tmsestringfield.oldmsestring: msestring;
var
 bo1: boolean;
begin
 result:= curmsestring(bo1);
end;
function tmsestringfield.curmsestring(out aisnull: boolean): msestring;
var
 statebefore: tdatasetstate;
begin
 statebefore:= tdataset1(dataset).settempstate(dscurvalue);
 aisnull:= not getdata(nil);
 result:= getasmsestring;
 tdataset1(dataset).restorestate(statebefore);
end;
function tmsestringfield.curmsestring: msestring;
var
 bo1: boolean;
begin
 result:= curmsestring(bo1);
end;
procedure tmsestringfield.setismsestring(const getter: getmsestringdataty;
           const setter: setmsestringdataty; const acharacterlength: integer;
           const aisftwidestring: boolean);
begin
 fcharacterlength:= acharacterlength;
 size:= acharacterlength;
 fgetmsestringdata:= getter;
 fsetmsestringdata:= setter;
 fisftwidestring:= aisftwidestring;
end;
{$ifdef integergetdatasize}
function tmsestringfield.GetDataSize: integer;
{$else}
function tmsestringfield.GetDataSize: Word;
{$endif}
begin
 if assigned(fgetmsestringdata) then begin
  result:= sizeof(msestring);
 end
 else begin
  result:= inherited getdatasize;
 end;
end;
function tmsestringfield.GetAsString: string;
begin
 if assigned(fgetmsestringdata) then begin
  result:= getasmsestring;
 end
 else begin
  result:= inherited getasstring;
 end;
end;
function tmsestringfield.GetAsVariant: variant;
var
 mstr1: msestring;
begin
 if assigned(fgetmsestringdata) then begin
  if fgetmsestringdata(self,mstr1) then begin
   result:= mstr1;
  end
  else begin
   result:= null;
  end;
 end
 else begin
  result:= inherited getasvariant;
 end;
end;
procedure tmsestringfield.SetAsString(const AValue: string);
begin
 if assigned(fsetmsestringdata) then begin
  fsetmsestringdata(self,avalue);
 end
 else begin
  inherited;
 end;
end;
procedure tmsestringfield.SetVarValue(const AValue: Variant);
begin
 if assigned(fsetmsestringdata) then begin
  fsetmsestringdata(self,avalue);
 end
 else begin
  inherited;
 end;
end;
function tmsestringfield.asoldsql: string;
begin
 result:= fieldtooldsql(self);
end;
function tmsestringfield.getdefaultexpression: msestring;
begin
 if inherited defaultexpression  fdefaultexpressionbefore then begin
  fdefaultexpressionbefore:= inherited defaultexpression;
  fdefaultexpression:= fdefaultexpressionbefore;
 end;
 result:= fdefaultexpression;
end;
procedure tmsestringfield.setdefaultexpression(const avalue: msestring);
begin
 fdefaultexpression:= avalue;
 try
  fdefaultexpressionbefore:= avalue;
  inherited defaultexpression:= fdefaultexpressionbefore;
 except        //catch conversion exception
  fdefaultexpressionbefore:= '';
  inherited defaultexpression:= '';
 end;
end;
function tmsestringfield.getproviderflags1: providerflags1ty;
begin
 result:= fproviderflags1;
end;
procedure tmsestringfield.change;
begin
 if not (fis_changing in fstate) then begin
  include(fstate,fis_changing);
  try
   inherited;
  finally
   exclude(fstate,fis_changing);
  end;
 end;
end;
procedure tmsestringfield.SetDataset(AValue: TDataset);
begin
 if fieldname = '' then begin
  fieldname:= fieldnamedummy;
  try
   inherited;
  finally
   fieldname:= '';
  end;
 end
 else begin
  inherited;
 end;
end;

"

Maybe we need a function to decide the class of a string field ?

Probably does not help if the descendant can not access private base fields.

Martin

AFAIK inheriting forms with nested frame with nested frame works in Lazarus. Some years ago I fixed a bug which only appeared at 5 level depth.
TReader/Writer does not know about forms/frames/datamodules, so I guess it should work for MSE forms too.
MSEGui suppports nesting longer than Lazarus.
Maybe the FCL was fixed in the meantime?

Mattias

I will look at what can be done here.

So you need to access to

     FValueBuffer : Pointer;
     FValidating : Boolean;

I can make these available as protected properties.

" > Maybe we need a function to decide the class of a string field ? > Probably does not help if the descendant can not access private base fields.

Well, from your code adding the following to the protected section:

   Property ValueBuffer : Pointer Read FValueBuffer;
   Property Validating : Boolean Read FValidating ;

Should be enough to remove the need for the TFieldCracker ?

Michael.

Unfortunately no:
"
procedure tmsebufdataset.checkfreebuffer(const afield: tfield);
begin
 {$ifdef FPC}{$warnings off}{$endif}
 with tfieldcracker(afield) do begin
 {$ifdef FPC}{$warnings on}{$endif}
  if foffset and 2  0 then begin
   freemem(fvaluebuffer);
   fvaluebuffer:= nil;
  end;
 end;
end;
"

Martin

Possible.

Martin

If I understand correctly, you have in the record buffer just the pointer of the (wide)string instead of the actual string data ?

If so, why then didn't you implement the string fields as blobs are implemented in SQLDB (it is basically the same problem), in that case you would not need the internal fields of TField at all ?

If that is not possible, it means you need a mechanism to 'finalize' the record buffer whenever it is re-filled or goes out of scope ?

Michael.

Correct. The UnicodeString pointer. That has not directly to do with the procedure above BTW. This one is to allow changing field values in OnValidate IIRC.

If so, why then didn't you implement the string fields as blobs are implemented in SQLDB (it is basically the same problem), in that case you would not need the internal fields of TField at all ?

Maybe, but what about performance? Another complication is the "updatebuffer" with the "oldvalues".

If that is not possible, it means you need a mechanism to 'finalize' the record buffer whenever it is re-filled or goes out of scope ?

That mechanism is implemented in tmsebufdataset and does not need tfieldcracker IIRC. tmsebufdatset holds a list of record offsets which must be finalized.

Thanks, Martin

"
The whole point of classes is to offer abstraction. Again: if you don't want abstraction, don't use data structures that offer an abstraction. "

I have a different view on programming languages and pascal. It's a tool to help generate code. The abstraction is to prevent tieing to a single instruction set/cpu/computer, but it's always nice if the result/instructions/data field can be compared to the high level code.

I also wonder how much of an optimization it actually is ? Maybe 0.000001% more performance ?

"
1) as mentioned in the original mail, the current transformation is implemented for saving memory, not for improving performance "

This wasn't clear, it only mentions gaps. What kind of gaps ? Apperently you ment to minimize memory size, the opposite could also have been ment in the sense of optimizations to make fields fall on memory boundaries for perhaps increased fetch speed or something else.

Later performance optimization possibilities for the future are mentioned as well.

"
2) if it was done for performance reasons, some people already got up to 34% extra performance by doing exactly that:
http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.34.4009 (download the cached version of the paper, the original link no longer works) "

I scanned over this document and it seems to mention "profiling information", it also mentions "compilers which can then use this profiling information to re-arrange fields".

To me it seems this optimization idea belongs in the realm of "profilers". It should be easy for a programmer to use such a profiling tool and make the necessary changes him/herself instead of complexifieing the compiler.

Perhaps someday these kinds of "complex optimization tricks" might backfire as well, because of different runs of the program or so, though usually there is some redline in a program.

How do you envision this profiling to be done with the free pascal compiler or did you think about "some kind of static offline optimization" ?

If the later how about this:

"
setup code:

access field 2.

access field 2

red loop:

forever access field1 then field2.
"

Without actually running the code and profiling it or analyzing the for loop, the static offline optimization trick would believe field2 is accessed the most while field1 is actually accessed the most first.

So could lead to wrong optimization results.

I rarely inspect the binary equivalent of a class instance, so your supposedly optimization is probably not a big deal, for records that would be a different matter since these are used in all kinds of api's and input/output situations.

"
As mentioned in the original mail, the transformation would only be applied to classes.
"

How is a record not an abstraction and a class is an abstraction, that's kinda weird/inconsistent ?!

"
It can be used for that, as long as you don't use high level abstractions. The whole point of abstractions to get rid of any guarantees a far as implementation is concerned, in order to increase portability, programmer productivity and compiler optimization opportunities. "

How about instead extending the pascal language description and specifieing that the order of the fields in the class and records must be the same in binary as well. This seems nice and constant and might allow some other functionalities in the future.

That is not to say that this supposedly optimization could be done and later then removed if this order extension is introduced.

Finally I do see some merit for free pascal compiler or any other compiler to generate some helpfull (debugging?) / profiling information (?) so that a profiler can report back to the user what source code fields are accessed and in what order... and how often to give some hints or suggestions to the programmer how to make the optimizations him/herself.

Some might even take fun in optimization code, but it can also be boring work, then again, after having done it a few times, perhaps programmers learn from it and become better programmers, instead of being left in the dark. That in itself being a better programmer is also a lovety goal and tools which stimulate that would have my preference then just a blackbox ;) :)

Bye,
Skybuck :)

Thinking about it:

I would allocate the buffer as is, with for all string fields an integer-sized slot in the buffer. The slot contains an index, pointing to a separate array of strings containing N*M*2 strings. (N=number of records, M= Number of string fields per record, factor 2 for old values)

Field value = Element [Index [+1 for old value]] in the array. where the [Index] is stored in the buffer.

The speed performance penalty of this system should be negligable, since you assume all records are in memory anyway.

And: everything can be done without meddling with the internals of TField.

Michael.

Hi,

Gaps between fields of an object instance due to alignment. Typically loading data from unaligned address, i.e. an address that is not evenly divisible by the field size, is slower than otherwise. Also, some CPU architectures even give you an exception if you actually try.

E.g. for the following object

type
  test = class
    b1 : byte;
    d : double;
    b2 : byte;
    q : qword;
  end;

due to above hardware limitations an instance will look as follows in memory (first column indicates offset, disregarding any additional internal data):

0 : b1
1-7 : <empty>
8-15 : d
16 : b2
17-23: <empty>
24-31: q

So for storing 18 bytes of usable data, you use up 32 bytes in memory, i.e. a waste of 44%.
Now imagine your program uses thousands of these records.

Apperently you ment to minimize memory size, the opposite could also have been ment in the sense of optimizations to make fields fall on memory boundaries for perhaps increased fetch speed or something else.

You always want to make fields fall on memory boundaries (i.e. align them) except if you are either really scarce on memory, need a specific layout for i/o purposes or have another reason to do so (e.g. extra padding in multi-threaded code when you want to avoid cache line contention).

But then hopefully you know what you need to take care of and read the manual. There is a way to disable this reordering on a per-class basis.

Later performance optimization possibilities for the future are mentioned as well.

Given that software on reasonably modern hardware is very often memory bound, a decrease in memory footprint often translates into real performance gains.

In the above case, if this "optimization" is applied, the object instance looks as follows (for example, do not know the exact algorithm):

0-7 : d
8-15 : q
16 : b1
17 : b2

I.e. the object now uses exactly 18 bytes of memory.

Imo for a static compiler like fpc, re-arranging fields for pure performance reasons without reasonably accurate profiling information is indeed pure guesswork.
I do not see a problem with automating this process.

However as mentioned above, decreasing memory footprint often also increases performance.

Because unfortunately people are already (mis-)using records by blockwriting/reading them without knowing that this is not portable at all.
Actually such things already break e.g. when moving from 32 to 64 bit processors, or from one cpu architecture to another with different alignment rules, and so on. When 64 bit was new, there have been many questions/issues about exactly that - on this list too.

It looks like a purely pragmatic decision. (And maybe sometime somewhere the Borland people defined it that way for records).

Given that not fixing the order has above mentioned tangible advantages right now, and fixing the layout only some unknown ones in the future, it seems the better choice to not do it. Also because it can still be specified in the future if needed :)

So we agree.

Thomas

Thank you. There are more items in the db.pas list... But I think first we should concentrate on classes.pas because I really don't want to fork it. Forking db.pas is less problematic and I probably prefer it in place of an endless discussion and in my eyes not optimal solutions. With a forked db.pas I can eliminate the many workarounds I already had to implement.

Currently needed crackerclasses by MSEide+MSEgui:

Used by MSEide for different tasks (example: ask for ancestor forms and frames/submodules while loading a form/datamodule, recover in case of an error) and for streaming of frames with additional components/widgets:

TComponent:
- FComponents
- FComponentState
- FFreeNotifies

TWriter:
- FPropPath
- FAncestors

TReader:
- FStream
- FLoaded

Used in order TParams create tmseparam items instead of TParam:

TCollection:
- FItemClass

Used to unify memorystreams/files/pipes/sockets/stringstreams:

THandlestream:
- FHandle

TMemoryStream:
- FCapacity

Why must these fields be private?

Thanks, Martin

Probably can be solved in a forked db.pas

Martin

Descendent classes should not manipulate these except through the right methods such as insert/remove component and FreeNotification/RemoveFreeNotification ? You have the means to manipulate the contents, what is wrong ?

TWriter: - FPropPath - FAncestors

I see no problem in making these available as read-only protected properties.

A collection must always contain the same items. The class is determined when the collection is instantiated.

The handle is fixed throughout the lifetime of the stream. It is set when creating the stream, and closed when destroying the stream.

TMemoryStream: - FCapacity

What is wrong with the r/w property "Capacity" ?

I think you are missing an important point:

You want some radical changes, so I expect you to be the one giving the reasons/motivations for a change. I want to help find solutions, but not at the price of destroying what is in my eyes a correctly constructed class hierarchy.

If you don't give a detailed explanation why you need to manipulate all these private fields outside of the proper methods, then I cannot help you find solutions for the problems you experience, so please: elaborate.

Michael.

Or by 2 other solutions:

* having a global variable ParamsClass : TParamsClass and use that whenever a TParams instance is created.

* Or have a virtual function CreateParams which is used everywhere.

There are always other solutions.

Michael.

I meant it can be solved without you need to change anything.

Martin

If I only could be sure that it is worth the effort...

Martin

Does this mean you're not even going to try ?

Michael.

I'll think about.
Thank you for your time.

Martin

class

Though there is the saying that forks aren't bad, I'd say that the less units forked the better. The Pascal community is already small enough so we should not fragment regarding such base units like db. Also in my opinion the less hacks (that could prevent optimizations) the better.

Regards,
Sven

I also have somethings in fpc rtl/fcl that also don't like or changing would simplify my work. But if we start to change base classes at each developer request we may end with a mess.
Just to be clear, i'm not against all changes. Some specific points can be changed at careful examination (like devs are already doing)

BTW: some time ago you pointed in a fpc list that will not use trunk because of encoding aware strings, so you don't need to worry because the trunk changes won't affect you anyway

Luiz

I don't think moving fields from private to protected in order it is possible to implement the wanted behaviour in my library need careful examination because it does not touch the FPC functionality.

I *never* asked to change *any* functionality in FPC-RTL other than to fix bugs which could be demonstrated by a different behaviour to Delphi 7! And I had no problems to use cracker classes and mad workarounds on my own risk in order not to bother the FPC team with my needs. The problem now is that cracker classes can't be used in future anymore because of the new class field ordering optimisation, so I dared to ask.

BTW: some time ago you pointed in a fpc list that will not use trunk because of encoding aware strings, so you don't need to worry because the trunk changes won't affect you anyway

At the moment.
I hope sometimes in the future the FPC team can say "the dust has settled down now and the Unicode handling in FPC is/will be done so and so". I do not want to adapt MSEgui to a moving target.

BTW: does anybody know how unicode resourcestrings work or are planned to work in FPC trunk?

Martin

"
Gaps between fields of an object instance due to alignment. Typically loading data from unaligned address, i.e. an address that is not evenly divisible by the field size, is slower than otherwise. Also, some CPU architectures even give you an exception if you actually try.

E.g. for the following object

type
  test = class
    b1 : byte;
    d : double;
    b2 : byte;
    q : qword;
  end;

due to above hardware limitations an instance will look as follows in memory (first column indicates offset, disregarding any additional internal data):

0 : b1
1-7 : <empty>
8-15 : d
16 : b2
17-23: <empty>
24-31: q

So for storing 18 bytes of usable data, you use up 32 bytes in memory, i.e. a waste of 44%.
Now imagine your program uses thousands of these records. "

If it was in a record then that's what a "packed record" is for.

Perhaps pascal should be extended with "packed class".

I would like something like that... it makes more sense then some weird/automatic/black box behaviour ;)

"
Imo for a static compiler like fpc, re-arranging fields for pure performance reasons without reasonably accurate profiling information is indeed pure guesswork."
I do not see a problem with automating this process. "

I do see a bit of a problem with it. Computer programs can have many options/variables which could have all kinds of values.

It's probably not practical to try every possible value because this would take to long.

So automating it will be problematic.

"
Because unfortunately people are already (mis-)using records by blockwriting/reading them without knowing that this is not portable at all.
Actually such things already break e.g. when moving from 32 to 64 bit processors, or from one cpu architecture to another with different alignment rules, and so on. When 64 bit was new, there have been many questions/issues about exactly that - on this list too.

It looks like a purely pragmatic decision. (And maybe sometime somewhere the Borland people defined it that way for records). "

Does this apply to "packed records" ? I would hope not.

"
Given that not fixing the order has above mentioned tangible advantages right now, and fixing the layout only some unknown ones in the future, it seems the better choice to not do it. Also because it can still be specified in the future if needed :)

So we agree.
"

Not really, if features/optimizations are implemented which need to be removed in the future then that is again a lot of work to do.

Simply removing stuff/code takes time/energy/work as well.

Bye,
Skybuck.

Unfortunately PACKED is not allowed on all suitable places; there should be packed set

There's the {$packset xxx} directive you can use.

/array/record/

"packed array" and "packed record" both exist.

object/class.

Both respect the current {$packrecords xxx} setting ("class" only since FPC 2.6.0). If you use {$packrecords 1}, there will be no gaps between the class fields and hence nothing will ever be reordered either.

Jonas

The problem now is that cracker classes can't be used in future anymore because of the new class field ordering optimisation, so I dared to ask. ==============
The guys, this point is important !
All the discussion members are right in their arguments (Martin's & the "community" impatience to fixing bugs etc because of intensive using all range of MSEgui facilities- and the need of the FPC team to keep the proper class hierachy ) & and neither You nor Martin have enough time to chase tons of code to justify every smallest piece JUST NOW with hot heads.
So, is it possible to design the new feature in such way that to have an option to proceed using cracker classes ?

2012/7/19, Jonas Maebe :

Jonas Maebe wrote

That may actually lead to quite some troubles: if someone recompiles the RTL without optimizations, then your class crackers also have to change that setting. And there's no way to detect how the RTL (or in general: units containing classes you are "cracking") has been compiled in your source code.

Actually there may be:

The cracking unit, needs both cracker classes (with and without optimization).

If sizeof(TOptimizedCracker)=sizeof(TTheClass) then
If sizeof(TNotOptimizedCracker)=sizeof(TTheClass) then

This will however not work, if reordering happens for performance. It could also not work, if the optimizer reorders fields, even if it does not save space in the end. (not sure if that can happen).

Ivanko B wrote

The problem now is that cracker classes can't be used in future anymore because of the new class field ordering optimisation, so I dared to ask.

....

So, is it possible to design the new feature in such way that to have an option to proceed using cracker classes ?

But the whole discussion comes down to one other simple question. Including the above, the whole discussion is about:

Should FPC provide a way to access private fields from any other code?

The same dilemma in expressed by:

Jonas Maebe wrote

That may actually lead to quite some troubles: if someone recompiles the RTL without optimizations, then your class crackers also have to change that setting. And there's no way to detect how the RTL (or in general: units containing classes you are "cracking") has been compiled in your source code. Adding support for something like that is definitely not a road I want to go down -- even a switch to simply treat all "private" fields as "protected" would be less bad (not that I would consider such a switch desirable in any way; it's like adding a switch to allow calling functions only declared in the implementation of another unit).

In context of the discussion, if such an optimization can go ahead, or if it would break (valid) code. Jonas says the following (or I read it into it):
1) Support for accessing private fields as if they where protected is not an option
2) Yet he discusses the option of holding back class re-ordering optimization.

And 2 is (so far) only for the point of allowing cracker classes (in other words: access to private fields). So in entering the discussion for 2, it seems to me he is in direct contradiction of 1?

Or an option:

{$ifdef nonLazarus}
protected
...
{$else}
private
...
{$endif}

Then MSEgui/FPgui/.. may be compiled with "-DnonLazarus" option

2012/7/21, Ivanko B :

I don't see cracker classes as valid code. If it works, fine, but it is subject to break. The layout of classes is simply opaque else we open a whole can of worms: like fixed vmt layout or whatever.

I don't see cracker classes as valid code.
=============
Then the FPC team should eliminate the need in such crackers - via either disabling (via licencing, prisoning etc) the "impatient" [mainly because of impatient customers] non-mainstreams (non-Lazarus) or meeting needs of the non-mainstreams ( the above "{$ifdef nonLazarus}" etc ).

2012/7/21, Florian Klämpfl :

So you propose we should just remove all privat/protected directives?

No, just reorder the fields so that they can be properly $IFDEFed as protected for nonLAZARUS and left (private) as is otherwise. Sure not every filed but those the non-mainstreams developers ask. This'll allow the non-mainstreams to start fixing right now. Florian, it's a huge headache for Martin Graeme to remember & explain all smallest details "what & when & why" and they can easily introduce bugs if fixing their code in a hurry. As they'll remember & fix their code to "without cracker" they'll be informing the FPC team to delete related $IFDEF so on.

2012/7/21, Florian Klämpfl :

Smth like the below:

type
  TNonCrackableClass = class
private
  fldPrivate1: integer;
  fldPrivate2: styring;
{$ifndef nonLazarus}
  fldPrivate3: real; // for LAZARUS
{$else}
protected
  fldPrivate3: real; // for MSEgui/FPgui
{$endif}
protected
 fldProtected1: string
public
  [..]
end;

2012/7/22, Ivanko B :

Ivanko B wrote

No, just reorder the fields so that they can be properly $IFDEFed as protected for nonLAZARUS and left (private) as is otherwise.

Why should lazarus people have less chances to mess with private fields? Either we make them public for all or for nobody. Of course, then everybody has to take care of the fact that users might mess with these fields.

Why should lazarus people have less chances to mess with private fields? =========
AFAIK, they haven't to use any crackers up to now and it is a normal way for the mainstream - where complier & IDE are maintained by same team so fixing any bugs is much more urgent than for no-mainstream.

Either we make them public for all or for nobody
=========
It 'll break the regular (99% of cases) access rights in classes and may greatly rise number of bugs.

2012/7/22, Florian Klämpfl :

But for very base (not yet specialized) classes - a good idea.

2012/7/22, Ivanko B :

As I wrote earlier:
"
Suggestion:
For all FCL base classes which are used in different toolkits and which can't be forked without breaking precompiled third party components, namely the units classes and db, move *all* private fields and methods to protected and mark them as "Use on your own risk, can be changed everytime! Don't cry afterwards.".
"
The {$ifdef} IvankoB suggests is not ideal because it either needs different precompiled RTL's or the users must compile the RTL for non Lazarus use.

Martin

the users must compile the RTL for non Lazarus use. =============
Really , it needs FPC sources at user's side.

Then a workaround - to maintain protected properties for private variables :

private
 fldPrivate1: integer;
 fldPrivate2: string;
[..]
private
  prorerty Private1: integer read fldPrivate1 write fldPrivate1;
  prorerty Private2: integer read fldPrivate2 write fldPrivate2;
protected
  prorerty Private1relaxed: integer read fldPrivate1 write fldPrivate1;

Then MSEgui/FPgui may rely on Private1relaxed insted of fldPrivate1+cracker.

Once a cracker disabled, all related fldPrivate(N) needing replacement with Private(N)relaxed will be revealed at 1-st compilation.

2012/7/22, Martin Schreiber :

FPC and Lazarus are not maintained by the same team. FPC and lazarus know only that close cooperation is important and brings benefits for both sides in the long term.

so fixing any bugs is much more urgent than for no-mainstream.

What are the numbers of the issues worked around by the crackers?

Like recent Delphi versions allow by extended RTTI? <shudder>

My solution for such problems are local patches to the library source code.

A (global) compiler option "treat private as protected" might be another solution.

Finally class helpers could solve the problem as well, the cleanest solution IMO.

DoDi

ask.

Including the above, the whole discussion is about:

FPC will support extended RTTI sooner or later as well.

Finally class helpers could solve the problem as well, the cleanest

solution IMO.

While they would be the cleanest solution they won't work as they can only go as deep as (strict) protected (which I still not think was a good by Borland as public/published should have been enough...)

Regards,
Sven

Which is exactly why we will not do this.

The base classes expose a well-defined API.
This API is a contract you make with the developers of descendent classes.

Some fields are kept private to ensure that the terms of the contract can be met. Making them public/protected means that the terms of the contract can be broken by Developer A, when the code of developer B depends on the terms being rigorously enforced, and his code can go very wrong.

This is of course not so for all private fields, which is why I ask for reasons, so I can decide for each field what can or cannot be done.

And, if possible, alternative solutions will be presented if they can be found. But for that, I need to know in detail what the problem is in the first place.

Michael.

A (global) compiler option "treat private as protected" might be another solution.
============
Same and less flexible than the {$ifdef nonLazarus} solution.

What are the numbers of the issues worked around by the crackers? ===============
The exact number isn't very important. For instance, Martin, Graeme... always report all bugs found by them but still need the ability to proceed without waiting for the bugs gets fixed [ sometimes because guys like me & customers insist on it].
Forking is bad since squeezes the base of advanced users (bug reveals, good proposals & patches, ..) of the forked feature of mainstream project.

2012/7/22, Sven Barth :

Ivanko B wrote

A (global) compiler option "treat private as protected" might be another solution.

This is no solution but a hack.

"Numbers", not count. I want to know which bugs are worked around by crackers.

That's easy: 0 bugs.

Martin needs the crackers for some mse* features.

Michael.

Sorry, there where bugs in the past and there probably will be in the future.

Martin needs the crackers for some mse* features.

Which otherwise can't be implemented without changes in FPC or FCL. I don't dare to ask for changes so cracker classes were a workaround without to bother FPC team.

Martin

Martin Schreiber wrote

Which otherwise can't be implemented without changes in FPC or FCL. I don't dare to ask for changes so cracker classes were a workaround without to bother FPC team.

Asking for changes to make base classes more flexible is not bothering us. It's a regular part of software development, and in the best case every user of FPC comes out ahead in the end thanks to the resulting changes.

Hacking around the type system that results in certain optimizations becoming impossible is annoying, however. And I don't really see a best case outcome in this case, regardless of what is done in the end.

Jonas

PS: just to make it clear, I haven't committed this optimization yet.

There are always bugs. We fix those as soon as we can.

Florian is talking about bugs you fix with cracker classes.

What bug do you solve with the TField stuff ? None. You work around a limitation, yes. But a limitation is not a bug.

That's because you don't just ask for changes.
You ask for your own solutions to be implemented in FPC.

Alternatives we present are discarded as sub-optimal or not to your liking. Let me recapitulate:

1. You reject my solution for TField.
2. For TCollection.FItemClass I presented an alternative for your problem.
3. TParam.Bound turned out not to be a ,
* 1 case which is not a problem.
* 2 cases where you have been presented with a solution, but which you chose to reject. You can't blame us for that.

As for the TComponent and TStream problem cases, I am still waiting to see what bugs (if any) you want to solve:
You failed to give detailed descriptions so far.

Cooperation works in 2 directions. I'm willing to think about solutions. You must give detailed descriptions of what you think is a problem, and be prepared to accept solutions that are maybe not 100% to your liking.

If you are not prepared to accept such solutions, then I cannot help you.

Michael.

In our previous episode, Ivanko B said:

Requires recompile, and a directive to turn said optimization off would then be better (and not carve up the sources).

Asking for changes to make base classes more flexible is not bothering us. It's a regular part of software development, and in the best case every user of FPC comes out ahead in the end thanks to the resulting changes.

Hacking around the type system that results in certain optimizations becoming impossible is annoying, however. And I don't really see a best case outcome in this case, regardless of what is done in the end. ==============================
Hmm..it sounds like "Relaxing access rights is a usual deal and why shouldn't we do it again to resolve the contradictions - especially if it turns out to be the only relevant ?" :)

2012/7/22, Jonas Maebe :

directive to turn said optimization off
==================
But everyone wants the optimization ! (sure without breaking consequences)

2012/7/22, Ivanko B :

Alternatives we present are discarded as sub-optimal or not to your liking. Let me recapitulate:

1. You reject my solution for TField.
2. For TCollection.FItemClass I presented an alternative for your problem.
3. TParam.Bound turned out not to be a problem at all. ==============
True but it relates to old issues, but there may be future issues (subjects to wait until get fixed) - Martin, Graeme are mainly bothered by them - whether it'll be possible to fix them in 1..2 days (customer etc requirements) and in such manner that not to rebuild FPC at developer's side.

2012/7/22, Ivanko B :

Me too.

I do not ask for my own solutions, I ask to move private FPC class fields and methods of base classes which are used in different toolkits to protected in order to make them more flexible without changing any functionality and because the current workaround with cracker classes may be not possible anymore in future because of upcoming FPC optimisations.

[...]

Sorry Michael, I do not trust that the effort is worth the outcome for me.

Martin

If you find a problem to a bug, you can submit a patch.

Martin has frequently done so, so has Graeme.
I always thank people for bugs they report or even fix.

I do not recall us refusing bug fixes.

What Martin currently does is not asking for bug fixes. He requests fundamental changes to the base classes, which he needs to implement some features in the
particular way that he chose.

That is a completely different story.

I can understand his desire to implement new features, which is why I offer to help thinking of solutions.

In some harmless cases this solution may well be
promoting a field to protected. In others, it will not, for reasons I have explained elsewhere.
I judge that on an individual case basis.

It is up to him to accept or reject the offered solutions.

Michael.

In our previous episode, Martin Schreiber said:

I don't like this as a blanket solution. Individual cases can be discussed, making it a principle means that we have to support even more of the interface (everything that can be reach by hacking the implementation of the moment).

Then IMHO you are on your own in finding solutions. We can be reasoned with, but if people don't bother, there is nothing we can do.

Martin Schreiber wrote

Sorry Michael, I do not trust that the effort is worth the outcome for me.

You're really putting us between a rock and a hard place here. It's quite tempting to retort with "the next time I'll just commit my changes and tell people to deal with it, because I do not trust that the effort of discussing them first is worth the outcome for me".

Of course explaining yourself and discussing costs time, and of course there are never guarantees about the outcome. That holds for everyone. If you shut out yourself from the discussion a priori, you basically force the other people to either a) be the "bad guys" because they don't take your grievances into account (mainly because they don't know enough about them to sensibly think about them), or b) just give up because they don't want to break other people's code like that

And maybe the worst of all: the next time you have a question that requires thinking or discussing, people may be less inclined to spend effort on that because you don't return the same courtesy in the opposite situation.

Jonas

So, there're the following solutions which meet the requirements of optimization & not breaking non-mainstream code & no need to custom build FPC:

1) moving related fields from private to protected or even public (makes the current classes structure more vulnerable)

2) creating protected "brother" properties for related private fields ( less appealing than "1)" = people using them knows what they do )

2012/7/22, Ivanko B :

I couldn't have put it better than Jonas and Marco.

I can only repeat: please explain, we'll try to find solutions.

If you don't trust that, there is nothing more to be said. It works both ways.

Michael.

I did and will do.
The situation here is especial because I must convince Michael for things that is of use for MSEgui only and most likely for no use in Lazarus or contradicts his design principles. I learned that this situation is unwinnable so I used mad workarounds with cracker classes instead. Now I see that even moving private fields to protected is very difficult so I will search for other solutions.
An API compatible MSEgui db.pas and MSEgui specific TComponent, TReader, TWriter and other dependent classes is probably the best. AFAIK TObject is the only class which depends on compiler magic? Are there other classes which can not be duplicated?

Thanks, Martin

Martin Schreiber wrote

The situation here is especial because I must convince Michael for things that is of use for MSEgui only and most likely for no use in Lazarus or contradicts his design principles.

I don't think "Lazarus doesn't need it" would be a valid argument to reject a proposed change. As to the design principles:

I learned that this situation is unwinnable so I used mad workarounds with cracker classes instead. Now I see that even moving private fields to protected is very difficult so I will search for other solutions.

The problem with making private fields more accessible is similar as with supporting cracking classes: it makes it much harder to change implementation aspects in the future without breaking anything. The whole point of encapsulation is to make it easier to change the implementation as necessary/desirable later on. This requires work to find the right encapsulation abstractions (which can relatively easily be changed in case the implementation is hidden, because then implementation changes won't hurt other uses), but it's the only way that is maintainable in the long run.

An API compatible MSEgui db.pas and MSEgui specific TComponent, TReader, TWriter and other dependent classes is probably the best.

I don't think it's the best. I'm also quite sure it will require more work from you in the long run, keeping them up to date and merging bug fixes and new features from FPC back into your fork.

AFAIK TObject is the only class which depends on compiler magic? Are there other classes which can not be duplicated?

Pretty much anything in the system and dialect units (objpas, macpas, objc, iso7185) is/can be(come) tightly bound to the compiler. The rest should be safe.

Jonas

No, as I understood Michael, you have to convince others that what you want can not be done otherwise.
Also, making private fields accessible is not always the only - or best
- solutions. So the alternatives also have to be evaluated.

By discussing this the framework as a whole can only become better and more flexible, for all users. The msegui/lazarus points have nothing to do with it.

Joost.

There are more, like typinfo. In fact you could copy that one, but when the underlying rtti-information generated by the compiler changes...

Joost.

True. Looking more closely, there's also "fpintres", "fpextres", "softfpu" and maybe "variants". Still, it's a finite and not that extensive list.

Jonas

In our previous episode, Martin Schreiber said:

Afaik yes. Exception is in sysutils (magic too ?), tobject is in objpas.

The only thing to watch is the {$M+} around TPersistent.

I doubt it, the experience shows the opposite. :-)

So classes.pp except for TObject is safe?

Martin

Some fields are kept private to ensure that the terms of the contract can be met. Making them public/protected means that the terms of the contract can be broken by Developer A, when the code of developer B depends on the terms being rigorously enforced, and his code can go very wrong.
============
Then additional protected properties exposing private fields can also be a part of these contracts - as a agreed exclusions to the contracts.

2012/7/22, Ivanko B :

Ah, yes. So whole classes unit is independent from compiler. Super. :-)

Martin

I wouldn't bet on it though.

Why not?

Martin

Hopefully with better means to disable it.

Public and published members are accessible already without class helpers. This model would turn class helpers into pure syntactic sugar.

But allowing access to private members would not break Delphi compatibility, Delphi code still would compile with FPC.

DoDi

The base classes expose a well-defined API. This API is a contract you make with the developers of descendent classes.

Which you don't know when designing a base class.

IMO there exist two use cases for (base) classes: by end users and by component writers. Both have different goals and problems, which rarely can be covered by an single set of properties. When an end-user never should be allowed to write into some field, a developer may have reasons to do so.

This is of course not so for all private fields, which is why I ask for reasons, so I can decide for each field what can or cannot be done.

The more I think about these problems, the more I see class helpers as the natural extension of the language, that allows to implement extensions which have not been foreseen by the class designers. At the risk of the implementors and users, of course.

And, if possible, alternative solutions will be presented if they can be found. But for that, I need to know in detail what the problem is in the first place.

See above.

DoDi

There is no guarantee.

Il 22/07/2012 10:39, Michael Van Canneyt ha scritto:

There are excellent reasons not to expose private fields. What must be opaque and what must be visible to end users are designers decisions which involve the capability of providing new implementation without breaking existing code, take into account multiplatform needs, consider the planned improvements, etc. etc.

On the other side there are sometimes good and sound reasons for someone to break this rule, at his own risk, of course. It may be to quickly fix a bug, to provide an extra feature by reusing an existing object instead of rewriting a lot of code, etc. Of course this is a practice not to be recommended in general, but it occurs. And in many cases it is connected to specific needs which aren't of general interest, making the effort of supporting them hardly worthwhile.

IMHO a clean solution can be found at language level: if I declare

TMyclass = class(TWhatever)

then I just get the degree of visibility that the class designer has decided.

But if I declare (just an example, a better keyword can be found) TMyclass = subclass(TWhatever)
then I get full visibility, all the private fields of TWhatever become just protected in TMyclass, and of course, I'm on my own. The designer is not bound to any contract, because I have explicitly decided to run the risk.

This solution would allow mainstream designers to keep fields visibility to what they deem to be a safe level in general, without making it impossible to non mainstream designers to provide features which aren't considered of general interest.

Just my 5 cents.

Giuliano

Then "friend" classes as C++ offers and wait for this feature were implemented before proceeding with the optimization :)

2012/7/22, Giuliano Colla :

Ivanko B wrote

Then "friend" classes as C++ offers and wait for this feature were implemented before proceeding with the optimization :)

I never saw a C++ class pretending to be somebodies friend. iirc friend classes must be defined in the class which elements shall be access. So C++ like friend classes would help exactly nothing (iirc).

In our previous episode, Florian Kl?mpfl said:

Indeed, and in our case those friends list would be in precompiled code.

As far as Guiliano's suggestion goes, then I rather use a cmdline option for that (when compiled with that option it is allows to access private fields), rather than syntax. (but I don't know if the PPU format actually contains enough info to access private fields, so this is probably not easy).

But since it is apparently not worth discussing individual cases over, IMHO that means it is not worth compiler changes either.

Il 22/07/2012 23:15, Marco van de Voort ha scritto:

A command line option would make *all* private fields of *all* objects fully visible. A different class definition would make visible just a few selected classes. It makes quite a difference.

But since it is apparently not worth discussing individual cases over, IMHO that means it is not worth compiler changes either.

There's an issue which isn't individual but general. Free software means having the freedom of using it however you see fit for your application. This applies to fpc itself, and to fpc based tools, like Lazarus, FPGui, MSEide, MSEgui etc. If fpc offered a way to provide a higher degree of freedom, it would be a general improvement of fpc. If it were done properly it could keep all the advantages of stable guaranteed API's with the flexibility required by a large number of individual cases, which, taken one by one, would be almost impossible to cope with. Taken as a group they make a general case, worth considering.

"Martin Schreiber" wrote

On Sunday 22 July 2012 12:53:43 Jonas Maebe wrote: > > I don't think it's the best. I'm also quite sure it will require more

work

> from you in the long run, keeping them up to date and merging bug fixes

and

objc,

> iso7185) is/can be(come) tightly bound to the compiler. The rest should

be

> safe. > So classes.pp except for TObject is safe?

TObject is defined in unit System...

Regards,
Sven

Hi,

I'm sorry Ivanko, but that defeats the whole point of well designed classes and hierarchy. As Michael mentioned - if good reason can be given why a specific private field needs to be accessed, then sure, it should be moved to protected or some property added. The FPC team was accommodating with me in this regard in the past, so I can't see why they would treat Martin any different. Martin simply needs to take them case by case - instead of littering his framework with "cracker" classes, and never raising the issues with the FPC team. Yes, now it is more work for Martin [having having many "cracker" classes implemented], but he should have spoken sooner.

Martin Schreiber wrote

I don't dare to ask for changes so cracker classes were a workaround without to bother FPC team.

I sure have had my uphill battles with the FPC team [and not all turned out as I pleased] - but asking the FPC team for a change must still be the first step.

Yes, Iearned it, thanks. :-)
So classes unit is safe?

Martin

Exactly.

Martin

I hadn't read the other mails yet that also contained an answer... So, sorry for repeating. :)

So classes unit is safe?

As Florian said: currently it is. But if the need arises to introduce compiler dependant code in Classes (for example because of Delphi compatibility) then no one could stop the devs to change this.

In sum it would be best to leave all units in the RTL directory alone... Everything in packages on the other hand ;)

Regards,
Sven

No just you, but anyone using your units also runs that risk. When you add a possibility to the language that makes it easier for people to just use a special keyword to hack around something rather than report a bug/feature request or redesign their code, they will simply use that keyword most of the time because it's the way of the least resistance. This means that as a compiler developer you encourage the creation of code that may break any time the compiler is upgraded.

Users of such code however don't care whether such problems are caused by FPC or by the developers of those units. They just notice that things suddenly break and if the creator of that original code has stopped maintaining it, they're stuck or have to dive in themselves to fix things. We already have a sufficient number of backward incompatibilities with every FPC release without encouraging people to write code that increases the chance of that happening. Not to mention that it wouldn't just happen with FPC-released code, but with any code (users of third party frameworks could do exactly the same thing with that framework code).

Jonas

Hans-Peter Diettrich wrote

The more I think about these problems, the more I see class helpers as the natural extension of the language, that allows to implement extensions which have not been foreseen by the class designers. At the risk of the implementors and users, of course.

I agree with this statement as well.

In our previous episode, Giuliano Colla said:

On a per unit basis maybe, but yes.

A different class definition would make visible just a few selected classes. It makes quite a difference.

Still, anything but syntax. Then directive. (though I still think cmdline is enough)

> But since it is apparently not worth discussing individual cases over, IMHO > that means it is not worth compiler changes either. There's an issue which isn't individual but general. Free software means having the freedom of using it however you see fit for your application.

Yes. But this is not about free software principles, since it applies to the binary releases we create, not the source. If it was merely a source issue, maintaining a set of patches would be easy.

And I'm still not convinced there really is a /general/ problem, as opposed to a handful of crucial fields, and just trying to cut corners with due process by taking the easiest way out.

Sure, there is a lot of sentiment flying around, but that is something different that there really is an acknowledged problem.

This applies to fpc itself, and to fpc based tools, like Lazarus, FPGui, MSEide, MSEgui etc. If fpc offered a way to provide a higher degree of freedom, it would be a general improvement of fpc.

More likely yet another time sync. Since we are talking issues here that do not appear to be worth filing mantis reports for.

If it were done properly it could keep all the advantages of stable guaranteed API's with the flexibility required by a large number of individual cases, which, taken one by one, would be almost impossible to cope with. Taken as a group they make a general case, worth considering.

Not entirely the same. It would be if the people using these inofficial interfaces only for their own purposes. But this is for projects that pass it on. IOW people will start to base stuff that is based on stuff that is based on volatile interfaces.

Hi,

Ian Macintosh wrote

the design concept to me. What Graeme is saying is 100% correct, but also the wrong way to design imho.

I don't see it like that. I have developed thousands of classes, and huge frameworks. I often find a new case where I need access to something I haven't thought of before. This is exactly the case here. The FPC team is basing lots of design work on Delphi code, but FPC is also used quite differently to Delphi. So there is no way they could have though of all possible use-cases for each class and every property or field in those classes.

Public is simple, it's part of the contract you have in the class. It is what you should rely on not to change, and I always want to make as much as possible public.

Visibility rules are a "contract". It is easier to add to that contract (make private things protected), that is is to make Public things protected. The latter case will break a lot more code.

Saying that, changes must still be justified to make sure that the OOP design stays good and maintainable.