Hi

At http://blogs.embarcadero.com/abauer/2009/05/29/38888

This could drastically reduce Lazarus executable size.

Best regards.

Doubt that. Just another feature ported from .NET, the same effect may be achieved by changing property accessor to the function which will check the existence of the instance and create it if needed. You then have an extra check for nil at runtime, *but* it guarantees that whatever initialization order won't shoot you in the foot.

Besides, FPC already has the WPO. Has anyone tried it on Lazarus?

Regards,
Sergei

I don't think it is very specific to .NET. We use similar techniques for e.g. cthreads and cwstring under Unix. And I also used it in the fix for http://bugs.freepascal.org/view.php?id=12704

Besides, FPC already has the WPO.

WPO can sometimes help a little in such situations, but not very much.

Has anyone tried it on Lazarus?

Yes, the but the results are not very impressive. The reason is due to the way the LCL is constructed, almost every declared class can theoretically be instantiated. On the other hand, I read that recently the LCL was restructured a bit to improve the situation, so maybe the results would be better now.

Also note that WPO can do little about published methods (since it must assume that these can all be called via RTTI).

Jonas

Jonas Maebe wrote

I don't think it is very specific to .NET. We use similar techniques for e.g. cthreads and cwstring under Unix. And I also used it in the fix for http://bugs.freepascal.org/view.php?id=12704

So will be this added to the fpc?

Yes, we made a refactoring and as result empty form + button application weights 200Kb less now. Can you check, maybe now WPO results can be more impressive?

Best regards,
Paul Ishenin.

In our previous episode, Jonas Maebe said:

That is explicit, we add a unit. The above article, code only links in when you actually use it. There is a gradual difference.

IMHO it is a pity it works on class and not unit level though.

Note that future Delphi versions are rumoured to up the RTTI. There is not much detail knowledge about that yet though, so I hope it will be very local.

Op Sun, 31 May 2009, schreef Jonas Maebe:

Also not that many references through initialization are just one part of the equation. Many code is referenced through resource files, i.e. if a class X can appear in a resource file, the code to construct the class must be in the executable, even though class X will not actually be in the resource file.

One area that particulary demonstrates this is the Tform, it has a lot of fields that can optionally contain a reference to another class. Even when you set all these fields to nil and never construct such a class, there code will end up in the executable, because the executable will only learn that the field is set to nil when it processes the resource file at runtime.

These two aspects (unit initialisation and resource files) make both Delphi and Lazarus executables grow quickly, the LCL more because it has more initialization references than the VCL.

Daniël

I don't know. The way we used that sort of technique currently is manually (either using unit plugging, or referencing data only the first time a related function is called, so regular smart linking can leave it out).

Also, I bet that in spite of the big warning in that blog post saying:

"Beware of cycles! If you have several classes with class constructors within the same unit and they all refer to one-another, the ordering may be indeterminate and you may end up with strange results or even crashes."

that people will still insist that we reverse-engineer and implement exactly the same indeterminate/undocumented behaviour as Delphi has (just like when you have cycles in unit dependencies).

Ok.

Jonas

Paul Ishenin wrote

Jonas Maebe wrote: > Yes, we made a refactoring and as result empty form + button > application weights 200Kb less now. Can you check, maybe now WPO > results can be more impressive?

Still quite minimal results. But unless I'm misremembering, it seems that the size of the application itself has grown a lot. An empty form with a button now results in an application of 1.8MB (Carbon/i386). I thought it was closer to 1.1MB the last time that I tried this (October last year).

Here's where the size comes from (with smart linking and stripping, no debug information):

__TEXT __DATA __OBJC others total
1171456 655360 0 44932 1871748

Jonas

Maybe it was less in size because it did not contain application icon? Icon is rather big resource.

Maybe for you as compiler developer this numbers are meanful but it is not so for me. How can we interprete them?

Best regards,
Paul Ishenin.

In our previous episode, Paul Ishenin said:

text = generated code
data = initialized data.
I assume "others" are the resources as OS X sees them (which might not necessarily be equal to Lazarus resources)

I did a quick test, and for windows, an application became slightly less than 200k fatter compared to january 2008. (from 1250 to 1424k or so)

So if Jonas memory is correct, it might have a significant OS X specific factor in it.

Removing the icon removes only about 100KB. Also, why is it included in the program binary itself? At least on Mac OS X the system never looks there for an icon, afaik.

__TEXT = code
__DATA = data
(others = various kinds of required information for the program loader and dynamic linker)

Jonas

Jonas Maebe schrieb:
situations, but not very much.

Maybe the IDE can feedback some info about used classes and methods?

If it could say that a number of classes definitely are not instantiated, that could indeed help.

Jonas

In our previous episode, Florian Klaempfl said:

In time, a pascal analyser would be a good thing to have. I think analysis of smartlinking is more something for an analyzer than the compiler itself (though the compiler could generate certain datafiles obviously).

I was thinking about such things due to the recompilation bug, searching for a way to get _easily_ insight into the uses struccture.

This won't help lazarus. For lazarus, also the contents of the lfms is important.

In our previous episode, Florian Klaempfl said:

Yeah, I got it when I saw Jonas' reply. The IDE could list all visual components in forms in a file and hand it to the compiler. And a "custom" list could allow additional streaming types (for users code)

Marco van de Voort wrote

Yeah, I got it when I saw Jonas' reply. The IDE could list all visual components in forms in a file and hand it to the compiler. And a "custom" list could allow additional streaming types (for users code)

Either the IDE has to be able to provide a list of all instantiated class types in a subtree of the inheritance tree, or it has to provide a list of class that certainly are not instantiated. I'm not sure how it could do either though, since a programmer could always instantiate stuff directly in his code (unless there are classes reserved for internal Lazarus use).

Simply a list of certainly instantiated types without any guarantees about other classes would not be useful. The wpo infrastructure can already extract that information from the source. Although it can only know that these types "may" be instantiated, knowing it for sure is not going to change the results of the analysis or generated code (since it has to be pessimistic).

Jonas

In our previous episode, Jonas Maebe said:

In case of enabling WPO he would have to register them, and the IDE would fold them into the list from the dfms. Not ideal, but everything comes at a price.

If the registration would have to be full-subtree or not, depends on how much info the IDE has to figure this out for itself.

Non listed classes could raise an exception on instantiation.

All components are already sortof reported :)
In order to get streamed, each component must have a member variable in the owning form.
However what is needed is a list of properties touched in the lfm (setting control.caption or control.font.color for instance).

Marc

Unfortunately, thanks to RTTI this isn't enough.

Because any published property can be changed to any value, even:
- if it is not referred in the code
- if it is not in any LFM
The code could read some kind of .ini file (or worse ask the user to input a property name and value (ouch)) and use RTTI to find and set the property.

As a result, any code that can be triggered by a puplished property must be included.
This means:
- drag and dock manager (since they are enabled by setting a boolean property)
- probably a lot of image code
- many others

The only way round this, is to break compatibility with the VCL. If drag/dock (just a random picked example) were components, that must be put on the form or otherwise can not be used, then they would not need to be included.

With graphics a lot of image-format readers are registered, if the unit is used. Again the issue is to find if they could be used in some automated way...

Martin

[snip]

but... The IDE cannot know of any of those cases either, so it cannot generate the list.

Marc

I've now added support for removing not called virtual methods (similar to the {$ifdef vtentry} code in the compiler, but fixed so it also works for the lcl/lazarus). It removes a bit more, but still not very much. Here are the file sizes of iterative compilations using wpo of the above mentioned test program and the lcl (so no recompilation of the rtl or fpc packages) until nothing changes anymore:

$ ls -l button-?b
-rwxr-xr-x 1 jmaebe admin 1711656 Jun 5 17:10 button-1b -rwxr-xr-x 1 jmaebe admin 1687020 Jun 5 17:10 button-2b -rwxr-xr-x 1 jmaebe admin 1657604 Jun 5 17:10 button-3b -rwxr-xr-x 1 jmaebe admin 1653428 Jun 5 17:10 button-4b -rwxr-xr-x 1 jmaebe admin 1649280 Jun 5 17:10 button-5b -rwxr-xr-x 1 jmaebe admin 1648928 Jun 5 17:10 button-6b -rwxr-xr-x 1 jmaebe admin 1648928 Jun 5 17:10 button-7b -rwxr-xr-x 1 jmaebe admin 1648928 Jun 5 17:10 button-8b -rwxr-xr-x 1 jmaebe admin 1648924 Jun 5 17:10 button-9b

All programs are stripped. As you can see, wpo can now remove about 61kb (before, wpo gained 8kb). Still not very much. The reason that you need several iterations to get all possible savings, is that the compiler only records calls to virtual methods from code that is included in the final program. So removing a method may cause some other methods to be no longer from anywhere, etc.

It should be possible to do this during one run by keeping the information per method instead of per unit, but that would increase the ppu size quite a bit.

For lazarus itself, it's:
$ ls -l lazarus-?b
-rwxr-xr-x 1 jmaebe admin 9181256 Jun 5 17:13 lazarus-1b -rwxr-xr-x 1 jmaebe admin 9070132 Jun 5 17:13 lazarus-2b -rwxr-xr-x 1 jmaebe admin 9040948 Jun 5 17:13 lazarus-3b -rwxr-xr-x 1 jmaebe admin 9032640 Jun 5 17:13 lazarus-4b -rwxr-xr-x 1 jmaebe admin 9028436 Jun 5 17:13 lazarus-5b -rwxr-xr-x 1 jmaebe admin 9023984 Jun 5 17:13 lazarus-6b

So savings of about 153kb (before, wpo gained about 70kb).

It's not yet committed, because there are still some compiler crashes when recompiling units after my changes, similar to http://bugs.freepascal.org/view.php?id=13323

Jonas

FPC does not strip empty methods, functions, initialization/finalization sections. Does wpo strips them? How difficult is to teach compiler to remove them during compilation?

It's not yet committed, because there are still some compiler crashes when recompiling units after my changes, similar to http://bugs.freepascal.org/view.php?id=13323

Good that you care this time.

Best regards,
Paul Ishenin.

No.

How difficult is to teach compiler to remove them during compilation?

I don't know, I have not looked into it. I don't think it's really trivial though.

Jonas

Op Fri, 5 Jun 2009, schreef Paul Ishenin:

How difficult is to teach compiler to remove them during compilation?

I sense a hope that somehow the compiler developments will ultimately be able to turn large Lazarus executables into small ones. It is not realistic.

The WPO features Jonas is working on are very promising, and I think he is doing something unique, I know no competing compiler that does it.

But....

The compiler will never be automatically able to remove PNG readers, XML parsers and so on, because the LCL deliberately is designed to make executables handle such data.

What is resonable to expect is that due to advances into making the compiler more clever, occasionally small gains will be achieved.

Daniël

No, I expect just another 60Kb size decrease for a form+button application.

Best regards,
Paul Ishenin.

I think you guys are doing a great job. From 93MB with debugging info down to 9mb is quite a big step. Especially since 2mb are icons and images.

Mattias

Mattias Gaertner wrote

I think you guys are doing a great job. From 93MB with debugging info down to 9mb is quite a big step. Especially since 2mb are icons and images.

I don't think that it's useful to include debugging information in any comparisons. Especially since stabs is known for its enormous sizes (that was one of the reasons why the DWARF designers introduced many features specifically geared at reducing the debug info size).

Jonas

Daniël Mantione wrote

The WPO features Jonas is working on are very promising, and I think he is doing something unique, I know no competing compiler that does it.

I know that method devirtualisation is done at least in JVMs from IBM (and probably also in Sun's), and I believe that ARM's C++ compiler can remove unreachable virtual methods (since for ARM, size is everything).

Jonas