One thing that has always gottem to me when using Delphi and now Lazarus is the tedious edit, compile, debug cycle. It has been growing worse of late that now I am beginning to dread the whole process any time I have to make some changes, especially when they are just minor GUI changes.

I have to wait for what is now approaching 60Mb of debugging enabled exes to be generated every time before debugging can start and I'm afraid I am beginning to show signs of an ADD. I tend to approach development with what I can do in the mean time whilst stuff is cooking.

The problem is the is duration too long to be staring at the screen, and too short to give something else your attention before coming back to it.

I guess for compiled languages ObjectPascal isn't bad, I believe it is even worse for C++ programmers.

I may have to get a hex core CPU with 24Gb of RAM, and the fastest SSDs to make the process tolerable. Some hosting providers are doing them for circa ?100 a month.

Quite simply I am thinking of a way to create the application in parts, so that the parts containing the new functionality can be developed separately as standalone programs, perhaps even console programs.

This raises the question of how procedure parameters and results can be passed, and these are the options that have initially come to mind.

1. Find a way to automatically translate simple variables, record structures, or objects into text and pass them via the command line or pipes. Perhaps JSON, XML or simple name-value pairs can be used here. Designing the parameters as TPersistent objects that are streamed into text form may be a better option here.

2. Rather than using the command line, make them network services and pass the parameters that way, or may make them RESTful/XML-RPC with all the new webby stuff( I am getting into some of that stuff now)

3. Design them as DLLs initially - I suspect that will bring along a different set of problems.

Once they are well tested I can incorporate them into the main executables.

I suspect that others have come across this problem before, with ObjectPascal and perhaps other languages and have found a way to mitigate the problem.

If I have to do business as usual, will using stuff like interfaces help?

I really need some help here or else I may have to resort to PHP(gasp!!) or even Java (bigger gasp!! shock!! horror!) or Zen (not the programming language)

I'm of to stackoverflow and reddit for some more relief now.

....

Yes splitting your project into parts is a good idea. Quite simple use packages.

Then you still have one application, you still can pass data around in the normal pascal way.

Each package can be tested with a small simple runner application, or even better a fully automated testcase.
Testcase are especially useful, if you can extract the functional layer from the visual layer.

So instead of writing something big into TForm.SaveDocumentButtonClick, you just have "TheDocument.Save" in there
=> all functionality goes into a none visual object TheDocument (or whatever it is in your case).
- this object takes instruction, if the user clicks something
- it can be asked what to display.
- it can be given callbacks, if it requires to notify the visual layer. for example it can have "OnTitleChange" => which is linked to a method changing the forms Caption (Title)
....

Martin

Why do you think this is different from any other language ?

I did quite a lot of web programming lately. JSON, Javascript and stuff. None of that went any faster than FPC/Lazarus, in fact, they went lots slower.

Same with PHP. Worse even, because in PHP, you only notice coding errors when the interpreter actually gets there. It may go unnoticed for a long time. Javascript is even worse. Give me compiled languages any day.

In my experience, the compile cycle is the least time consuming. A couple of seconds, at most. No big deal.

Setting up test data, working through the login screen and several other screens to get to the screen I'm actually testing: this is the chore of debugging in 99% of the cases.

If you want to speed things up:

One thing which I've learned is that Test-Driven development makes the debugging easier. FPCUnit (or name any other testsuite) makes the debugging process faster:
1. You clearly separate business from GUI logic.
Once business logic is done, I'm pretty confident the code is OK.
2. The unit test apps compile quick and run quickly;

    As soon as they've run, I know the cause of the failures;
    There is seldom any need to actually debug.

What you may want to do is try and avoid windows when working with Lazarus. Lazarus/FPC still works significantly faster on Linux.

Michael.

On Sat, Nov 27, 2010 at 1:48 AM, Michael Van Canneyt <

Michael, is it known why it's still so? I mean, apart from os-specific bindings, general i386 code should compile identically (loops, searches and so on). Also linker for Windows is rewritten some time ago as long as I remember to be fast enough.

Max
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20101127/8e3b34a8/attachment.html>

I have no idea;
I work almost exclusively with Linux for Lazarus/FPC. On the rare cases I compile something for windows, I notice that the same project still compiles less fast. The IDE itself is a good testcase there :-)

But Florian or Jonas should be able to answer this question more precisely.

Michael.

At least the startup time on Windows is much slower as on Linux, because on Windows (at least on NT) you have multiple API layers while on Linux there's mostly only the C library and with Free Pascal not even that.

On Windows you have the Win32 API libraries that either can do stuff themselves (good), call functions from ntdll to do the job (a bit less good) or even call the Win32 subsystem server (IPC!) which is - compared to the other two - a very slow process.

These IPC calls also happen on application startup, because the kernel is informing the Win32 subsystem that there is a new process (and a new thread - the main thread) in the system that it has to take care of. Also the API that is provided by the Win32 subsystem carries around much overhead out of compatibility reasons which slows down at least process startup (initialization of TLS, console information (if any), thread information, window information) as well.

That's why the introduction of the internal linker reduced compile time on Windows that much: one less process to execute :D

Regards,
Sven

Thanks for the reply, Martin.

Are there some code examples which use this approach, especially the package based one?

Will it also reduce the size of the debugging exe or speed up its creation?

Sometimes I see only one file compiled, then it goes to the linking phase, but it doesn't seem to reduce the exe size or the speed of its creation.

It will speed up its creation, cause if you didn't touch a unit in a package it isn't recompiled. The size of the EXE won't be changed (see below).

Sometimes I see only one file compiled, then it goes to the linking phase, but it doesn't seem to reduce the exe size or the speed of its creation.

That's because only one file needs to be compiled.

It does speed up the compilation (a little bit), cause the compiler only needs to load the ppus not to recompile their units. And it would be a very bad sign if that would reduce the EXE size, cause that means that code would not be compiled in that otherwise would be in there.

Regards,
Sven

I don't know of any examples, but I guess there must be. It's the same as what Michael Van Canneyt said: "separate the business and gui logic". (you can google this phrase)

In the simplest case:
create a new class (in it's own package) that reflects your form, that has a method for every method that currently exists in your form. Then move all the code (and remove all reference to the form itself). All values are transferred to begin with, all actions applied at the end, or by callback.
Of course such a separation will have much better results if you refactor the code more than just copy it in a new class.

Will it also reduce the size of the debugging exe or speed up its creation?

The Size stays the same, therefore also the speed. But that doesn't matter, because you will create this exe far less often. (could be up to 90% less often)

Let me give you an example. I work on SynEdit.

If I changed something then in order to test it I had to rebuild SynEdit and the IDE. And as you know you need to test often before everything works fine. So I had to rebuild very very often, that has a big impact. SynEdit already was a package. So I was spared the step of creating one.

I created a test case though. It's still very small, but it helps. If I change something, I run t in the testcase. So all I do is rebuild the package SynEdit (that will only rebuild the files I changed, so I compile 3 or 4 files on average). And compile the testcase. that is a very small and simple application, and is done very very fast too.

I could have just thrown a SynEdit on an empty form and compile this. That would be as fast as the testcase. Then I could try out every feature by hand.
But as the amount of features grow (and as each can break on every change), I would have to test more and more scenarios. Doing so by hand is increasingly time consuming.

The testcase does it fully automated. It does not only test the new feature, it alerts me if I break another one.

The GUI itself, the visual interaction, te mouseclick, and canvas drawing, are things that are hard to test in automated testcase (possible but harder).

But what I test, is that the data on which all those actions are done, is correctly calculated, and provided. And that reduces the amount of work I have to do on those GUI/frontend code.

Sometimes I see only one file compiled, then it goes to the linking phase, but it doesn't seem to reduce the exe size or the speed of its creation.

See above. the size of the final exe and the compilation time for that does not matter.

If I change SynEdit, I use the testcase. I may have to recompile th 3 or 4 units affected by my change several dozend times, and rebuild the testcase (another handful units that are recompiled). But thats about one dozend unts, and one small test app. The big IDE I build maybe 2 or 3 times, wher as without the testcase I would have build it several dozen times.

The trick is to find code that can be tested easily, and to extract it into it's on package, and classes.

Martin

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20101127/f0dc37b7/attachment-0001.html>

Sven, it all really matters, but on modern computers with enough memory installed all these hd-related difference become irrelevant on the next usage of the same files/executables/dlls, since they all go to the system cache. Lazarus IDE starts in less than a second on my win7 notebook after the first run/exit.

I did also some checking with compiling. Inside the Lazrus and if I did the same from command-line with ppc386, the compile time for lazarus.lpi is 5.4 seconds if a did not any single change and a little more ( 5.7 ms) (the numbers from ppc386). If I did a change to one of the pascal files. gdb start is about 3 seconds, but I just checked, this is because of the anti-virus was probably investigating the new running exe file (if I switch AV off, the program appears almost instantly). Michael mentioned "a couple of seconds at most" and he is known to be working on IDE :), so if on his linux machine ide compiles/links in less than two seconds, then it really can be a noticeable difference for a comparatively big project (although we can not comparing without comparing hardware configurations)

Max
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20101127/4732e24c/attachment-0001.html>

While some of the post was about process startup times (which depend on more than just HD times), there are things about HD access times, that do matter. I do not know how much they affect the compiler, but...

An example: Somewhere in the last 12 month there was an issue in Lazarus, where the IDE would be unresponsive for several second (sometimes 10 to 15 seconds). Very annoying if you are in the process of editing something.

The reason was that the IDE would check the last-modification-times of all dependency files on disk (several 1000, maybe 10000 files). Calling the system to ask for the last-modification time that often (even with all/most data cached by the OS) would take that long on Windows, while under Linux it wouldn't even take a single second...

Similar issues can affect compilation times too.

Martin

Calling the system to ask for the last-modification time that often (even with all/most data cached by the OS) would take that long on Windows, while under Linux it wouldn't even take a single second...

But how does it come that there can be such a difference doing nearly the same things on Linux and Windows? I can't believe that Windows is *such* a bad design. They all cook with water I think.

It would be interesting to see a comparison on the same filesystem. E.g. fat32 or ext2 (using ext2ifs). NTFS is a bad example because it is implemented on Linux using a user file system driver (fuse), which might influence the performance test.

@Martin: How does the IDE ask for the last modification time (on Linux and on Windows)? I can then crawl through the source code (for Windows using ReactOS) to see what both OSes are basically doing.

Regards,
Sven

I don't remember, I can't even find the bug at the moment.

It was somewhere in codetools. It was invoked when you tried to do implementation-jump. I vaguely remember Codetool was not caching this info, so potentially asking several times for the same file.. A cache was added, and things go better.

Who said slow linking performance has anything to do with the file system ?

if I remember correctly, then it has more to do with the huge number of symbols in the windows unit which are by default smarlinked. Since all programs depend on it, all programs compile slow.

But like I said, the compiler people should check this.

Michael.

This would be a useless comparison in so many ways. ext2 drivers performs _way_ worse under windows, due to code quality. there just aren't that many people interested in a high performance ext* driver for windows, understandably. Also, the features of the filesystems are so different, you can't even compare them. fat and ntfs are stuck in the dark ages compared to ext*.

There is an in-kernel NTFS implementation, but it's no good for writing.

Henry

That's why I suggested to do the test on the same filesystem. That should (ideally) reduce differences in the filesystem itself and show the latency from the get-last-modification-time call down to the driver and back.

While I agree that FAT today is no good any longer (except for small usb keys and such ^^), I'd like to know your reasons why you think NTFS is "stuck in the dark ages" as well.

There is an in-kernel NTFS implementation, but it's no good for writing.

Hm... it might be enough for a test like this, because we only want to retrieve the last modification time, not change it. But here might apply the same as for the Windows ext2 driver: there might not be enough people interested in a high performance read only in-kernel NTFS driver while there is the more feature complete NTFS-3G one.

Regards,
Sven

Sven Barth wrote

While I agree that FAT today is no good any longer (except for small usb keys and such ^^), I'd like to know your reasons why you think NTFS is "stuck in the dark ages" as well.

My comment was probably unfair, since ntfs has had some updates bolted onto it over the years. Either way, this is not the right place to discuss this, so please disregard my statement.

NTFS does not have an official spec, which makes things very hard.

Henry

Henry Vermaak wrote

Also, the features of the filesystems are so different, you can't even compare them. fat and ntfs are stuck in the dark ages compared to ext*.

Modern NTFS implementations have some really nice and advanced features none of the ext* has: snapshoting (saves file server admins a lot of time :)) and transactions. I guess especially snapshotting makes ntfs slow: CoW semantics simply requires a lot of coping of files and fragments probably the disc. I made a similiar experience with btrfs (whsich has CoW semantics by default as well) on linux, working with fpc on a btrfs partition made fpc really slow. I didn't test further by turning off CoW for btrfs, but it might be reason.

One thing which might help on ntfs: turn off atime, see e.g. http://oreilly.com/pub/a/windows/2005/02/08/NTFS_Hacks.html

Florian, the reading/writing of files seems to be ok under Windows. But the file access functions are much slower than under Linux (file exists, date, open, close).
fpc already caches these attributes, but of course it has to read them every time it starts. It reads over 80 directories. Maybe someone can check how much time this costs under windows.

BTW, some file systems like zfs under Linux have snapshots too, and the file access is still as fast as ext*.

One thing which might help on ntfs: turn off atime, see e.g. http://oreilly.com/pub/a/windows/2005/02/08/NTFS_Hacks.html

;)

Mattias

I checked these things some years ago but at this time this wasn't a problem. As Sven already mentioned, the only real bottleneck on windows is process startup time.

Besides turning off atime, another important things on windows are: never ever fill a hard disk more than approx. 80%. Filling a partition more than 87.5% will probably fragment the MFT and this makes a system really slow.

But after the first compilation, this is void anyways, all files are in the disk cache if the system has sufficient memory.

BTW, some file systems like zfs under Linux have snapshots too, and the file access is still as fast as ext*.

Of course, it might depend on the FS where it has its weakness: recently I unpacked gcc sources on an xfs partition: it took several minutes, on ext3 this takes only a few seconds.

On Sat, Nov 27, 2010 at 10:31 PM, Mattias Gaertner <

Mattias, it's challenging issue indeed. I might do some research, it's also interesting to read the discussion that already was about the the file scanning on Windows.

Two observations that come to mind:
- If one looks at any directory scanning utility in Windows, he or she will notice that usually it doesn't consume much cpu, at least first scanning. That's because all file operations are made with wait objects. So somewhere in the kernel, the request is made to read something (part of file, directory and so on) and the process that requested it just waits for a signal not consuming as a indirect measurement the more percentage of total time cpu spend (can be queried with GetProcessTimes) comparing to total time passed the less file system shortcoming affected the compilation. Is it possible for ppc386 (maybe with a special option), report with final message ("..compiled successfully") the total time + (user+kernel) time. If the difference is significant most of the time, we have to do something.
btw, the same logic can be applied for example for fragmentation of sqlite databases. The less cpu spent on the query, the more the database is fragmented internally

- When modifying a file, windows also changes the corresponding attribute of the parent directory (and only direct parent). Since fpc looks only to directories it knows, it could bypass scanning the files themselves, only parent directories. I sometimes use this trick to find a file I worked with recently in Far Manager since this allow me to work one level up in my searching saving some time.

Max
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20101127/d43e7054/attachment.html>

Hmm... it might be interesting to compile FPC for pure Native NT. Then we could test what the Win32 subsystem adds to the startup time :D

[OT]Can I cross compile FPC for another target using the Makefile? Does FPC use units (outside of compiler) that aren't part of the RTL?[/OT]

Except when you send the system to standby or hibernation... then you sometimes need to wait longer then after a cold start until the compiler starts compiling ^^ (because the disk cache is empty again?)

Wow O.o I should test something like this on my computer... (ext4, xfs and ntfs)

Regards,
Sven

Sven Barth wrote

[OT]Can I cross compile FPC for another target using the Makefile?

e.g.
make clean all OS_TARGET=linux
?

Does FPC use units (outside of compiler) that aren't part of the RTL?[/OT]

No.

Wow O.o I should test something like this on my computer... (ext4, xfs and ntfs)

I think the problem is always the history of a fs, what other files are on the disk etc.

Hello Frank,

On 26/11/2010 20:56, Frank Church wrote:
...

The FPC/Lazarus way is really to use components as the way to break down the overall complexity of your application, and structure your components into packages, several components working together going into one package.
The dependency graph of your set of packages should be acyclic (no cycles), where the more stable components and interfaces are the most used, and the newer things are at the end of the dependency graph.

Traditionally, Delphi has supported dynamic package loading allowing plug-ins to be added dynamically. FPC doesn't do that yet, but making DLL's is still possible if you need that option. Compiling your packages statically into your application does the trick nicely otherwise.

There are many books about making components for Delphi, those should be useful to you, especially those describing how to write non-GUI components (domain objects) as well as linking GUIs to domain objects.

Additionally, Peter Coad has a nice list of patterns for components that could be useful. It's in his book "UML in color with Java", but very readable. The UML tool, ModelMaker, now implements Coad's color pattern and can help with your design. It parses most of FPC/Lazarus code.

My own site www.tcoq.org has a few non-visual components that may provide examples, especially the Excel interface unit. You've got an example of how to independently test a component.

I hope this helps,
Best regards,
Thierry

Would this create a compiler that can run on Linux if I do this on Windows? That's what I'm unsure about.

Nice :)

I can't really comment on that, because I've not much experience with filesystems and their performance... :(

Regards,
Sven

Is there some automated way to include DLL:s into the main application exe file after one has finished the debugging?

The DLL defines entry points for the function calls so by finding the corresponding use entry point in the exe file it should in principle be possible to merge the dll binaries into the exe.

But I have never seen any such utility/discussion...

2010/11/28 Bo Berglund :

Packages DLL's.

Vincent

On Sun, Nov 28, 2010 at 3:35 PM, Vincent Snijders <

Vincent, although logically you're right, technically delphi packages are dlls. And I suppose when finally there's a dynamic package support in fpc, there is no other way to implement them, at least on Windows.

I tried to search for the principle Bo suggested. Actually every utility that claims just uses temporary extraction of internally packed dlls. From the technical point of view, there are challenges that should be solved when implementing (for example embedding loading and unloading routines of the dll), but I think that nobody did this also because of possible legal problems. The reason when one usually wants this is when he has no sources or other way to link it statically, that is the case when you use a third party library. If there's a way to make it an unbreakable part of the main exe, I'm sure there will be many complains from the authors/companies of dlls, not mentioning LGPL violation.

But when the packages are dynamic finally in Lazarus, such trick will make no sense since one always can have them separately on the development stage (for time saving when linking) and for the install exe, link them together using fpc.

Max
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20101128/7dc4f6c0/attachment.html>

Delphi packages are always dlls.
But Lazarus packages can be
- dlls
- OR (eventually) dlls plus some sugar for shared VMTs and mem
- OR a set of units
- OR just a set of source files

That's why you can split the compilation process of big projects in smaller chunks under Lazarus.

Before this thread goes into the wrong direction, please read http://wiki.lazarus.freepascal.org/shared_library

Mattias

Mattias Gaertner wrote

Before this thread goes into the wrong direction, please read http://wiki.lazarus.freepascal.org/shared_library

Thanks Mattias,
I did not think far enough, just simply thinking of DDL:s providing some standalone functions that would return a result.

But of course a DLL can contain much more, like one or more forms which could also use objects and more. The memory manager alone would break my idea, I guess...

So please drop this on the floor.....

Op 2010-11-26 21:56, Frank Church het geskryf:

One thing that has always gottem to me when using Delphi and now Lazarus is the tedious edit, compile, debug cycle. It has been growing worse of late that now I am beginning to dread the whole process any time I have to make some changes, especially when they are just minor GUI changes.

You are describing the classic case where I would recommend TDD (Test Driven Development). It completely removes the "guessing game" from programming. By that I mean, you don't have to guess what the new change will affect or break. Simply make the change, compile the test suite and run it. If you get a red bar, you know what it affected and broke. And because you just made that code change, you know exactly what code you need to work with to fix the issue.

I highly recommend you read the book "Test Driven Development by Example - by Kent Beck". It's a thin book, so quick to read, but full of very useful information on how to get started with TDD and how the development cycle works.

TDD is a shift in thinking about how to write a program, but once you have experienced how great it is, you will not consider writing any program without writing unit tests first.

Regards,
- Graeme -

Sven Barth wrote

That's why the introduction of the internal linker reduced compile time on Windows that much: one less process to execute :D

The binary startup time matters for building via make files.

I doubt if it matters for compiling _apps_ in lazarus. Once compiler, once linker, each 100ms, together 200ms, clearly not the major component.

The internal linker is simply faster. Smartlinking lazarus on windows could take 10 _minutes_ with the external linker on a not that old (Core 2 1.7Ghz) machine, eating 1.5GB in the process. Now reduced to 30s and 400MB. (on the same machine when I last tested it, 2.2.x days)

In my experience, Windows is relatively slow in enumerating directories. If you reduce the number of search dirs, the compile time will considerably improve.

That's partially also because the segments with the debug info are not even loaded.

It will. You can even add CPU targets. I generated an entire lazarus for Mac OS X/PowerPC on Windows.

But there are caveats:
- you must have suitable crossbinutils on Windows. Some are available on ftp in the /pub/fpc/contrib and /pub/fpc/contrib/cross directory
- non windows systems typically need any needed libraries on the compiling system. Simple programs for Linux and FreeBSD excluded.

The buildfaq contains a lot of background info on this.

Not units, but on e.g. BeOS and OS X, it might use libraries. (like libc). The rtl of these interfaces libc.

I have confirmed that by myself yesterday as well. :D

As my NativeNT target is quite similar to Win32 (in terms what the compiler needs, not in terms of the RTL ^^), I don't need any crossbinutils or libraries.

For me it was important to know whether I need to implement some stuff in packages (fcl-base, etc) as well or whether RTL is sufficient.

I was already able to successfully compile FPC for NativeNT, but for profiling the process start up time I'll need to implement ExecuteProcess and let me say that Win32's CreateProcess is a big beast to translate ^^

(also I'll need to test StdIO as well...)

Regards,
Sven