Perhaps nobody noticed that in the past days the Pascal syntax has changed a bit. This leads to wrong behaviour of the code tools :-(

Now unit properties can follow an "var" block, while this was flagged with an error: expected ":" after "property".

The usage of an function name instead of the Result variable has been restricted. For some reason Lazarus now rejects "Result" in code completion, even if it is accepted by the compiler.

It also seems that Lazarus has problems with $IF, even simple expressions are not evaluated properly, so that the wrong branch is handled as enabled/disabled.

I had no look at the concrete changes, but if somebody needs assistance in updating the tools, I'm willing to figure out what should be updated.

BTW, what about a compiler package, installable into the IDE? I'm just redesigning the compiler (OO_rewrite branch), but the future of this branch is yet unknown.

DoDi

Can you give an example or better: the definition?

The usage of an function name instead of the Result variable has been restricted. For some reason Lazarus now rejects "Result" in code completion, even if it is accepted by the compiler.

Can you give an example or better: the definition?

It also seems that Lazarus has problems with $IF, even simple expressions are not evaluated properly, so that the wrong branch is handled as enabled/disabled.

Can you give an example?

Remarks:
-You don't need the OO rewrite for this. But the OO rewrite could help to install cross compilers for other processors and to allow to reset the compiler.
-it requires changes to the options, the gui, the cache. -I plan to rewrite the part that executes external tools after the next release. Registering another compiler is planned. Conclusion: let's wait with this after the oo rewrite.

Mattias

I would be happier myself, if I had a valid definition :-(

Let's postpone this one, until I can be sure that it is not related to my current work on the compiler. A duplicate function of the same name, in different units, and the wrong use of a function result instead of function invocation (recursion) has caused much confusion in the last days. I'm also confused by the compiler versions, now existing on my test system (trunk and branch version).

A simple example:

function GetProp: byte;
var
   test: integer;
property
   prop1: byte read GetProp;
   prop2: byte read GetProp;

If this is handled properly for you, I'm sorry for this wrong item.

This happened to me in a property getter function, i.e. in the GetProp implementation in above example. So it may be related to the above property problem.

Commandline option: -dMyConst:=1 (may work with local $DEFINE as well)

Code: {$IF MyConst>1} ...

DoDi

Right, it's not a requirement, but it can simplify and speed up the use of the compiler.

-it requires changes to the options, the gui, the cache.

In detail the compiler options could/should be passed directly to the compiler, not in form of commandline options (string). Also the time consuming unit search could be reduced or eliminated, when the IDE and compiler share the same file/directory cache. And not to forget the handling of compiler messages, that can go immediately into the IDE messages window.

-I plan to rewrite the part that executes external tools after the next release. Registering another compiler is planned. Conclusion: let's wait with this after the oo rewrite.

ACK. But I found your ideas very instructive, with regards to what functionality should be moved into dedicated and exchangable classes/objects :-)

Do you already have more concrete ideas, what in detail should be encapsulated, WRT later IDE integration of the compiler?

DoDi

Local properties? I didn't know that this is allowed. And my four days old fpc 2.5.1 gives a strange error:

Error: Resourcestrings can be only static or global

If this is handled properly for you, I'm sorry for this wrong item.

The compiler can not handle it. I guess it is correct for the codetools to not support local properties.

I don't know. The above does not compile.

I implemented it.

Code: {$IF MyConst>1} ...

This works now.

Mattias

Compiling must work the same with an installed compiler and on command line. It's hard to test all combinations, so better not use any shortcuts.

Also the time consuming unit search could be reduced or eliminated, when the IDE and compiler share the same file/directory cache.

Yes.
Although I guess they need different structures. But the compiler could load files/directories from the IDE caches. That should speed up a lot, especially under windows.

And not to forget the handling of compiler messages, that can go immediately into the IDE messages window.

They should not go "immediately". They need to be categorized, filtered, improved and buffered. And when code changes the output is adapted.
I want to rewrite that part in the next release.

I don't have concrete ideas yet.
I have a long list of goals, including translation, multithreading and different compilers.

Mattias

> A simple example: >

interface

implementation
function GetProp: byte;
begin
   Result := 0; //whatever you like. check code completion for "Result"
end;

Local properties? I didn't know that this is allowed.

Above snippet should be in the *interface* part of a unit, see above.

Sorry for the confusion :-(

And my four days old fpc 2.5.1 gives a strange error: Error: Resourcestrings can be only static or global

That's definitely a bug. If one of the FPC developers lurks:

procedure psub.property_dec, line 673: parser_e_resourcstring_only_sg is wrong message. A new message parser_e_property_... is required.

Please try again.

Pretty fast, thanks :-)

DoDi

Make all compiler options objects (or records with IDs) in the IDE, and let the (future) compiler handle them.

But you are right in so far, as the possible speedup may not be worth such efforts around commandline parsing in general.

We only have to agree about a common base class...

They can be intercepted in the Message() subroutine(s). This part (verbose.pas...) deserves refactoring anyhow, in a threaded compiler. Then every unit-parser thread must maintain its own message queue, else the resulting log would be garbled. But see below...

I'm not sure whether parser threads and parallel compilation really will speed up the compiler. But a compiler in a background thread would be nice for the IDE.

DoDi

Zitat von Hans-Peter Diettrich :

Fixed.

function GetProp: byte;
begin
   Result := 0; //whatever you like. check code completion for "Result"
end;

It works here.

Mattias

Zitat von Hans-Peter Diettrich :

There is a difference between the stored options and the real options. Most options are stored as name, value pair. The real options are computed on demand. The compiler only needs the real options. The computed options are concatenated as command line string. Theoretically it can produce another output, but the compiler has everything to parse command line options, the speedup is not measurable. I think we should not add extra formats without need.

But you are right in so far, as the possible speedup may not be worth such efforts around commandline parsing in general.

I have never seen the command line handling in the profiler. Let's first concentrate on what really takes time: file access.

Some kind of virtual file system is also needed for the version control addons.

The goal is that the IDE should not slow down, even if compiling with -va. I'm not sure how much parallel compilation speed up things in daily work, but I see some potential. Using the IDE file/directory caches will probably help more in daily work.

Mattias

We should have kind of property editors for the options, so that the user can pick valid values from a list - this is where the GUI may be affected. Once the options are known by option descriptor and value index, these values can be picked from and stored in the data structures imported from the compiler. This would not necessarily make any speedup, but could simplify the additional coding in the IDE.

Let's first concentrate on what really takes time: file access.

WRT to the recent namespace discussion, IMO file access can be reduced down to direct access of every (source) file. With project and package files (does FPC already have packages?) the location of every used unit should/could be known in advance, eliminating the need for an unit search the output path stored in the project files as well, and the location of the ppu/o files would be known as well.

All stored directory informations should allow for macros, so that e.g. the target/machine specific directories can be exchanged, as is required (currently) in building the compilers.

On newer (64 bit) systems memory mapped files can be used, so that every file has to be mapped only once and, thanks to the huge address space, can stay mapped until the IDE is closed. This would eliminate all file searches in the integrated compiler. Some re-mapping may be required when extending source files while editing, but this could be reduced by sufficiently large file mappings for all such files. Eventually unused parts are eliminated by the OS RAM paging.

DoDi

What additional coding?

I doubt that, because for backwards compatibility fpc has still to search in the paths.

Correct me if I'm wrong: memory mapped files is a more direct access to the OS file cache. You can omit the copy on loading and the copy-back on writing.
The IDE loads/saves files as whole with one TFileStream.Read/Write call.

I tried to measure the time of the copy:
The code browser loads under Linux fpc 2.5.1 about 87 MB of sources and takes on this machine about 4.5 seconds.
I doubled the copy on load and it is still about 4.5 seconds. I put the copy on load in a loop for 10 times and it is still about 4.5 seconds.
That's why I doubt that memory mapped files will give a noticeable speed increase.

Mattias

Mattias Gaertner wrote

the OS file cache

The OS cache already does a great Job:

root at t40:/home/bernd\ $ sync && echo 3 > /proc/sys/vm/drop_caches

bernd at t40:~/lazsvn/lazarus/trunk\ $ time make ide [...]
real 1m12.491s
user 0m8.653s
sys 0m2.348s

bernd at t40:~/lazsvn/lazarus/trunk\ $ time make ide [...]
real 0m6.127s
user 0m4.752s
sys 0m0.916s

1:12 with a cold cache versus 0:06 when the cache is warm is an impressive difference.

There exists a small tool for linux that tries to speed up file access with an extremely pragmatic but very effective approach by simply permanently monitoring which files are usually read most often by all the user's applications and then as a daemon running on idle priority permanently read all those files in intervals of a few minutes which will ensure that they are always all in the OS file cache.

Lazarus could adopt this idea and could after opening a project during idle time and in reasonable intervals read every file that would also be read by the compiler and linker. Maybe in a certain order that makes sense: first only the directory listings to immediately have all file timestamps, then all compiled units that would be needed (because most source will not have changed since it was compiled the last time) and then after this is done read all source files belonging to the project.

For the IDE this will at most save 20% of the needed memory. For normal projects it will be probably less than 10%. I didn't heard complains that lazarus needs too much memory in the last years, especially not on 64bit systems.

It heavily depends on file access pattern. Make your own tests.

Mattias

For the compiler options, that are not yet supported. And for the options of your additional compilers, that may have different textual encoding.

[file search]

I doubt that, because for backwards compatibility fpc has still to search in the paths.

Only in the commandline version (fpc.exe).

That duplicate install suggests a shared library to me (.dll,.so), that can be used by both Lazarus and a commandline front-end...

Correct me if I'm wrong: memory mapped files is a more direct access to the OS file cache. You can omit the copy on loading and the copy-back on writing.

Yes.

The IDE loads/saves files as whole with one TFileStream.Read/Write call.

The related file buffers are subject to swapping, i.e. they may have to be written to the swap file. MMFs deserve no such writes, except for changed parts, since they can be reloaded from the original disk file or cache.

The compiler is very concerned with memory usage, and uses small file buffers for all files. When file handles are restricted resources, on legacy platforms, the files are closed and re-opened frequently. AFAIR buffer size of more than 64 KB did not bring any noticeable speedup, most probably because most files are smaller.

I have no practical experience with MMFs myself, but users reported factors of 3 in reading files (Delphi/Win32). At least it would be worth a try.

DoDi