Should we use CRT or VIDEO unit? What's the state of Free Vision? We plan to port our WMS (Warehouse Management System) module from C to Pascal (to use our persistence framework). It has a few colors, a numbered menu and a few scrollable lists. The module usually is run thru Putty from mobile computers like MC-3090 (Windows CE + WiFi).

I can't speak for Free Vision (the only real world reference application seems to be the IDE and there's also a testapp in the $fpc/packages/fv/examples directory), but I can say a bit regarding CRT and VIDEO:

VIDEO seems to be targeted for "fullscreen" applications, because you basically work on a buffer that represents the complete screen. You won't be able to use UTF8 or Unicode characters and are thus restricted to what the unit gives you, but it tries to somehow map the block drawing characters into the extended ANSI area (> #127). Free Vision is based upon the VIDEO unit.

CRT allows you to only print to a part of the screen and will also allow you to write UTF8 characters as it's basically just a redirected Writeln that somehow injects the changes of fonts and the cursor movement. You should also be able to generate a fullscreen application though.

So it depends on what you want to achieve. You'd need to test whether those units work on Win CE at all though, because I can't help you here (I was never able to get the console on my Windows Mobile running :( )

Regards,
Sven

I find the choice of going for a console app quite curious. Wouldn't it be easier to write a standard GUI app using the LCL?

A standard GUI app would require the (dozens of) warehouse operators to use the "stylus" heavily. Today they use it only to start the PUTTY app (that opens a saved SSH session) and yet they manage to often damage the screen (!). Also, with a console app that runs via SSH we can easily run it from desktop PC (no need to recompile, just open the SSH). Some processes do not require high mobility and a PC + barcode scanner is way cheaper than MC-3090 handhelds.

We run it in 40x17 mode in the 3-inch small screens. The operators usually type 2-digit numbers in the menu to access the features. They call them "screens", like "run screen 52 for branch 17". After that they usually walk/drive around the warehouse, read barcodes and move boxes. Here's a screenshot:

2011/8/11 Felipe Monteiro de Carvalho

Felipe Monteiro de Carvalho wrote

I find the choice of going for a console app quite curious.

+1

And this is not the first message (in recent days) where the developer is considering console app over gui app. Weird, but I guess there must be a niche market out there we don't know about. :)

Personally, if I had to go console app, I would probably choose something like FreeVision at this point. I think you get quite a bit of functionality for free. Though I haven't written a long running console app (this excludes simple utility apps) in decades.

Sven Barth wrote

I can't speak for Free Vision (the only real world reference application seems to be the IDE and there's also a testapp in the $fpc/packages/fv/examples directory),

Isn't the FPC installer also written with Free Vision - I remember seeing something like that years ago.

Which installer are you talking about? At least the default Linux (or even Unix?) installer is a shell script. But you could try your newly aquired regular expression skills and search for "fv" in the fpc source directory :D (though using the search with "complete word only" would work as well...)

Regards,
Sven

Ok... that search won't help much... better try something like "app" as a complete word, then you'll find (besides many "garbage") the DOS and OS/2 installer for FPC in $fpc/installer/install.pas :D

Regards,
Sven

Sven Barth wrote

Which installer are you talking about? At least the default Linux (or even Unix?) installer is a shell script.

I didn't need regex for this. Simple browsed my FPC src directory. There is a 'installer' folder with an executable 'installer' or for windows 'installer.exe'

See attached screenshot. I guess you can only run the installer once, or something. :)

I tried to catch two flies at once... I wanted to find the installer (fly 1) and any other application in FPC directory that uses Free Vision (fly 2). So far there are only 3:
- example application
- IDE
- installer

Regards,
Sven

In the warehouse environment, with ruggedized handhelds, it's easier to use the keyboard than a stylus.
http://images.google.com/search?tbm=isch&q=warehouse+worker&gbv=2&oq=warehouse+worker

2011/8/11 Graeme Geldenhuys

+5

i have numerous apps that are being "upgraded" from DOS console apps and there's really no reason for them to have to incorporate all of the fluff'n'stuff of graphics... especially when doing so would require that the older boxen that are being used must be upgraded to the li'l monsters of today's apparent world and all for nothing more than graphical stuffs...

i'm still rather appalled that lazarus takes 75Meg on one machine and 125Meg on another for a debug environment... i'm not sure about the size difference since both should be identical other than one on w2k and the other on vista... but still...

Personally, if I had to go console app, I would probably choose something like FreeVision at this point. I think you get quite a bit of functionality for free. Though I haven't written a long running console app (this excludes simple utility apps) in decades.

i just found out about freevision earlier today in this thread... never used it or any of the similar stuff in TP/BP back in the day... everything i've written has been straight forward and minimal... no OOPs stuff other than message base management stuffs i decided to try many years back... even today, i still code my own replacement routines for stuff that it too large or cumbersome...

this switchover to the Lazarus RAD environment and FPC is really taking me on a beating :(

waldo kitty wrote

i have numerous apps that are being "upgraded" from DOS console apps and there's really no reason for them to have to incorporate all of the fluff'n'stuff of graphics...

OK, Alexsander raise a valid point. You have to cater for the environment where your application will be used. This makes sense.

i'm still rather appalled that lazarus takes 75Meg on one machine and 125Meg on another for a debug environment... i'm not sure about the size difference since

I also have a huge gripe against bloated software we see today. Today's software (well most of them) are just not as efficient has they used to be. Plus they make your hardware upgrade pretty useless in no time at all. Years ago you might have waited for a app to load or process something... with todays computers at a magnitude of 800x faster than the 386 days, you still wait for apps to load or process something! Nothing much (from a user experience) has changed.

On the flip side, there are some software that still perform very well today. eg: One of my projects loaded in Lazarus takes up about 100-150MB of memory. That same project loaded in MSEide takes up 9-15MB of memory!!! In each case the project is loaded, a complete recompile of the project is done, and a debug session (project run from inside the ide to activate debugging) has been done. How do you account for that difference in memory usage? MSEide is doing something right.

Lets just hope and pray that Lazarus doesn't one day get the "performance" of Eclipse IDE. That is probably the slowest and most memory hungry IDE I have every come across. :-/

i just found out about freevision earlier today in this thread... never used it or any of the similar stuff in TP/BP back in the day...

I've used Borlands version back in my TP days, but I haven't played with FreeVision either. I'm curious to find out how you design forms etc. I can't even remember what I did in my TP days. :) I think Free Vision is covered in the FPC docs though.

Regards,
- Graeme -

My HTC Wildfire (the cheapest HTC Android phone when I bought it) has 384 MB RAM

So I can only conclude that 150 MB of RAM is not nearly a big deal =D It could even fit on my phone!

Felipe Monteiro de Carvalho wrote

So I can only conclude that 150 MB of RAM is not nearly a big deal =D It could even fit on my phone!

The test project I used as an example is a new (and small project) we started recently, and already Lazarus uses such a lot of memory. Loading some of our much larger projects, the memory usage jumps considerably.

Irrespective of what device or hardware you have, it doesn't explain away why Lazarus is such a memory hog compared to the same project loaded in MSEide.

In both IDE's I used fpGUI based projects, so that cuts out a lot of the IDE's built in GUI designer, Object Inspector etc... So I feel my comparison is not unreasonable - I used the same features in both IDE's. If Lazarus doesn't watch out, it will soon become yet another bloated software product. Maybe it's time some "new features" get put on hold, and some optimizations are implemented instead. This is a normal process of any software development team. Maybe just the thing required for that looming 1.0 release. Just a thought. If MSEide can be light on memory, very fast and still have plenty of features, it should be possible with Lazarus too.

Regards,
- Graeme -

You're doing something wrong ;-) (ok, I know it was an exaggeration) While debugging another Lazarus instance, the host Lazarus has 62MB VM (56MB RSS), with a dozen open files and some forms. And I wouldn't be surprised if more than half of it is occupied by CodeTools tree.
gdb unfortunately takes 136MB.

Lets just hope and pray that Lazarus doesn't one day get the "performance" of Eclipse IDE. That is probably the slowest and most memory hungry IDE I have every come across. :-/

Netbeans and JBuilder don't fare much better...

Best regards,
Fl?vio

That depends. Does MSEIDE offer the same features ?

Are the MSEIDE Codetools on par with the Lazarus ones ? (and not just the codetools you use, but the complete set of features)

Michael.

While Video is usable standalone, the KVM units ( keyboard - Video - mouse) are mainly meant as the Free Vision backend.

Crt is pure legacy, and has many quirks and idiosyncracies.

Free Vision is the most complete solution, but its programming model is very old, it uses TP object and shortstrings, but pchars here and there to work around shortstrings and it in general doesn't take advantage of any newer FPC/Delphi features (mostly the more automated streaming system (RTTI, properties) ansistrings, and in general cleaning out design cobwebs from when apps still had to work on 20MHz 386s.

For minor stuff you can take any of them, as long you test them for your terminal types. I would use Video over Crt. (see the lister demo).

FV is also an option, but for the more complex (more than a few screens) apps.

A few attempts have been made (by Florian and me) to update FV stepwise (with at least ansistrings), but such projects quickly expanded in scope, and still would deliver a patched old and limping framework.

For real serious work, one would have to redo TV on a more Delphi/VCL basis, since then maintainability is king, and the limitations of TV get annoying over time.

Personally I like (TUI) console apps for server maintenance and configuration apps. They work quick, and work over a SSH connection, without much additional configuration. While this is a _better_ solution one has to ask if it is worth maintaining experience with two frameworks (GUI+TUI) for, specially if they are so different as LCL and TV.

Alexander seems to have a clear specialized reason though, where TUI connects to the pecularities of a certain devices interface. No discussion there.

That depends. Does MSEIDE offer the same features ?

I'm not arguing feature vs feature. Each IDE has there own unique features, and both have many features in common too. I simply tested the features in common. eg: Loading and managing a project, debugging an app, code navigation via Ctrl+Click, syntax highlighting enabled with the same color scheme, Search, Search & Replace, various project option changes etc. All features both IDE's support. But using those common features, Lazarus uses 5-10 times more memory? That doesn't sound right to me.

And as I said, that's with a small project. Using much larger projects, Lazarus IDE memory usage jumps considerably by the end of a working day. MSEide I on the other hand hardly changes its memory usage. In fact, I don't think I have ever seen MSEide go over 25MB memory in normal usage for a whole day.

I just tested this again. I have a Quad Core system with 4GB memory, running Ubuntu 10.04 64-bit. Using Lazarus 0.9.30.1 (latest stable version). Opening 208 units in one folder with Lazarus took around 15 seconds to complete. Opening that same 208 units in MSEide took around 1 second to complete. Then doing a complete recompile of all units.

memory usage after the above:
Lazarus IDE (gtk2): 145MB
MSEide: 31MB

The same task, but a huge difference in memory usage!

Yes, having 4GB of memory makes this hardly a concern. But keep on thinking like that, and soon Lazarus will perform like Java's Eclipse IDE, requiring a minimal of 1GB just to edit text. Any software project needs to pay attention to performance on a regular basis. This weeds out sloppy code. This is all I wanted to bring to light. Lazarus memory usage seems to jump a lot with each new release. Maybe with the preparation for a 1.0 release this trend can be reversed.

Regards,
- Graeme -

Graeme Geldenhuys wrote

Lets just hope and pray that Lazarus doesn't one day get the "performance" of Eclipse IDE. That is probably the slowest and most memory hungry IDE I have every come across. :-/

I normally don't use such acronyms, but here it fits:

+1 (for both points ^^)

Regards,
Sven

Alexsander Rosa wrote

A standard GUI app would require the (dozens of) warehouse operators to use the "stylus" heavily.

??? A GUI application can easily react on keystrokes....

-Michael

...
in fact I once did a GUI application (in Delphi) that the user (a person watching what goes on in a subway station) handles "blind" via the keyboard. No mouse attached to that PC.

-Michael

Where are you seeing these numbers? I guess you know you have to look at RSS, right?
(In case anyone is interested, nn Linux event library mappings is accounted in VM. GTK has huge mappings but doesn't use nearly as much memory actually).

-Fl?vio

Fl?vio Etrusco wrote

Where are you seeing these numbers? I guess you know you have to look at RSS, right?

gnome-system-monitor
Then look under the "Processes" tab, there is a Memory column.

alternatively I can use 'top -p <process_id>
Then take then do a bit of math: RES - SHR = memory usage This comes to the same result as shown in gnome-system-monitor.

Regards,
- Graeme -

OMG. Then we really need to discover what the gtk2 glue is doing so wrong.

Thanks,
Fl?vio

Fl?vio Etrusco wrote

OMG. Then we really need to discover what the gtk2 glue is doing so wrong.

It's not just LCL-GTK2, the same thing happens with Lazarus IDE compiled for LCL-Qt4 too. Just tested. Opening a blank project, Lazarus IDE (using LCL-Qt4) uses 69MB already. Open a 100 units and the memory usage jumps to 140MB. That's not ever working with the IDE yet, so imaging what it would be at the end of a normal working day.

Regards,
- Graeme -

Graeme Geldenhuys wrote

memory usage after the above: Lazarus IDE (gtk2): 145MB MSEide: 31MB

Now to show how Lazarus is increasing in memory usage per release. Here I loaded the same project and same amount of units under an old Lazarus version v0.9.29 (r27743). Just loading the project with 5 open units, memory usage is 44MB. Open the 208 the total of 208 units, and memory usage is just 77MB - compared to the above 145MB of a 0.9.30.1 build.

Regards,
- Graeme -

But what exactly are you comparing?

the base memory footprint of Lazarus vs MSEide (without loaded project), or the amount of memory needed to open a project (similar project)?

Comparing MSEIde against Lazarus. Lazarus includes codetools, a lot of info about the RTL units is always loaded (afaik), but you do get a benefit out of that.
Sure not all the memory maybe codetools, there are many other features that use the memory. but memory is rather cheap nowadays? (BTW, how is the memory usage, if you strip all packages from Lazarus that can be stripped?)

Both projects have completely different targets, and that result in different requirements.

Martin wrote

But what exactly are you comparing?

Lazarus (old) vs Lazarus (new)
MSEide vs any Lazarus version

At the moment I'm just comparing memory consumption doing the same tasks in each test case.

But as I also list, opening 208 units, the speed in MSEide is near instant compared to Lazarus IDE (any version of Lazarus, old or new).

But my main concern is by how much the memory consumption is growing in Lazarus. Previous release compared to current release. And beginning of the day to the end of a working day (without shutting down lazarus IDe once).

that use the memory. but memory is rather cheap nowadays?

So that's your answer. There is no issue even though memory usage double between each release?

Regards,
- Graeme -

Graeme Geldenhuys wrote

But as I also list, opening 208 units, the speed in MSEide is near instant compared to Lazarus IDE (any version of Lazarus, old or new).

15 secs for 200 units are NOT a bad time. And btw, older lazarus was slower, there was recent work.

Part of this time (and very minor amount of memory 100kb for every 25.000 (actually I believe 100kb for every 50.000) lines, go into functionality that is broken in MSEide.
Try horizentol scroll in MSE with lines longer than (afaik) 2000 chars. Yes, maybe not really needed to edit pascal code. But an editor should be able to deal with that.

Another bit of time is used by codetools. MSEide doesn't have them. It is every users choice to decide if they need them, and then to use MSEide or Lazarus.

I haven't compared the highlighter for pascal. But it may be similar.

> that use the memory. but memory is rather cheap nowadays? So that's your answer. There is no issue even though memory usage double between each release?

IMHO, this is a simplified view. And also remember 0.9.30 was a big release, after a long time...

Without knowing where this memory went, there is no telling how memory usage will look in future. It may well be, that some big amount went into a feature, that is well worth it, and that is of a kind that one can be sure, it will not happen with every release.

I am not having the time now to look into it.

But certain functionality, comes at certain cost. Lazarus priors the functionality over the cost (that does not mean Lazarus ignores the cost, but it accept certain costs, for certain features). you can choose to use it or not.

I am certain that if you can find ways to save that memory, without cutting functionality, then this will not be disregarded (It doesn't mean it will be accepted immediately, and without feedback or adaptations required....)

It's not a bad time. It's a _terrible_ time.

hcv at technical09:~/source/freepascal/latest/packages/univint/src$ time gvim "+qa" -fp *
450 files to edit

real 0m3.433s
user 0m3.110s
sys 0m0.208s

This is with syntax highlighting, everything opened in tabs. I wonder how fast lazarus will open all the files in univint...

Henry

I don't have a copy of gvim.

I can't compare the quality and amount of features. How well the highlighting deals with the various fpc constructs, if it provides folding, if it provides codetool like pascal context sensitive tools, ....

Yes there are many editrors that are faster, (and many that are slower, but that's now excuse). But many of the comparisations are apples and bananas, they compare 2 products, that only share a minority of features (though this minority may just be the part with the highest visibility)

Mind: "not bad" => that does neither mean "excellent", nor say it is on the top ranks. As I said there had been improvements (in the last 12 month). There is room for more, sure.
But looking at the overall:
- opening several 100 files is rare. (yeah some people only open, never close, and next time they reload they open all of them again => but how often to you edit e4ach of 200 files on one work day?)
- Yet it is possible, and unless you need to restart the ide several times an hour AND load all the files each times, it is not a major issue (it may be to a very small minority of people
So from this, there is no major priority in requiring to speed this up. There is no denial that it would be nice, but no terribly high requirement either.

Compiler development ;) I do almost all fpc development work with the textmode ide :)

Martin wrote

- opening several 100 files is rare. (yeah some people only open, never close, and next time they reload they open all of them again => but how often to you edit e4ach of 200 files on one work day?)

You are completely missing the point: This is a benchmark. It points out that we are at least an order of a magnitude slower than editors with similar capability. As Graeme showed, we've become slower, so if we don't watch it, this kind of attitude will end us up with a sluggish, bloated ide.

Henry

On a slower system (seconds rounded):
lazarus *
real 1m57
user 1m39
sys 0m1
geany *
real 2m3
user 0m52
sys 0m1
gvim -fp *
real 0m17
user 0m1
sys 0m0

I compared also with geany because it has the reputation of being small and fast....

Ludo

Ok, but you miss the point too, you picked the wrong part of my reply out of context:

part1, related to benchmarking:
- are we comparing apples with apples, apples with bananas?
- are the compared products indeed providing similar quality/functionality?

part2, related to the above quote.
- that was about every day use. And while it may not be top of the list, it is not a major issue in ever day use.

As for paying attention. Several bits of work where undertaken in the last 12 month.

as for apples and bananas:

gvim does only open about 10 tabs !!!! not one tab per file. At least my installation. I can not open more tabs, or access any other than the 10 tabs. I can from the buffer menu change the content of each tab, but that's still only 10 tabs.

gvim (i just installed) does not have horizontal scrollbar, and if it wraps line it does not update the vertical scrollbar (maybe by design, but that's a terrible design, though one that saves a lot of time, hence good for the benchmark.....)

the highlighting appears to be erroneous and far less complete "public" in a class is highlighted, "protected" is not "deprecated" is not hl.
I was unable to find out, if ANY hl is context sensitive at all; but for example "nested comments" are not correctly highlighted. Overall it appears that it is a simple "word based" highlighting, versus a content sensitive highlighing in Lazarus (apples and bananas)

has gvim any codetool like stuff???

....

Are we comparing apples and apples. OTherwise what kind of benchmark is this?

Btw Lazarus took about 27 secs for the 450 files. I am not saying that couldn't be improved, but if you want to benchmark. benchmark more fairly

And btw, if I save the 400 files as part of a project, and re-open, it goes in 16 secs. (As lazarus defers some of the work to idle time later)

0.9.30 or trunk?
Can you test trunk? I believe some of the improvements to opening files where made in trunk only after 0930 was out.

on my system geanny took 57 secs (all together).

laz trunk was faster (see other mail)
27 secs (450 files from command line)
16 secs (450 files from project session)

2011/8/12 Graeme Geldenhuys :

> that use the memory. but memory is rather cheap nowadays? So that's your answer. There is no issue even though memory usage double between each release?

Another reason to have as little releases as possible. ;-)

Vincent

OMG. Then we really need to discover what the gtk2 glue is doing so wrong.

I noticed a few weeks ago that, in at at least one unit, we don't appear to free'ing g_list's consistently. I queried this in the forums with no response - as I stated in that post I have no idea if the glib has a garbage collection style scheme, so I don't know how significant this is.

http://forum.lazarus.freepascal.org/index.php/topic,14149.0.html

This won't, of course, lead to an immediately overly large memory usage, so I doubt this is the main culprit, but I suspect it doesn't help...

Mike
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20110813/be901536/attachment.html>

> OMG. Then we really need to discover what the gtk2 glue is doing so

wrong.

It's not just LCL-GTK2, the same thing happens with Lazarus IDE compiled for LCL-Qt4 too. Just tested. Opening a blank project, Lazarus IDE (using LCL-Qt4) uses 69MB already. Open a 100 units and the memory usage jumps to 140MB.

Does the memory usage lower when you then close the 100 units? Trying to eliminate leak as a possibility.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20110813/cc65b0db/attachment.html>

That is trunk (svn 31134)

The system I used to test is a laptop = slow disk. So the IO part for lazarus and geany are larger but roughly equal. Hence the smaller performance difference on my system, but in favor of lazarus.

Ludo

That obviously explains the bulk of the time difference. However, I think this is something we can learn from. Does lazarus have to load (+ highlight
+ ...) the files that are in tabs that are not even on the screen? Why not fe. load in the editor only the files in tabs initially visible (after startup/load project) and the others when tab actually opened (not when scrolling the tabs, but when selecting a tab). Similarly, unchanged files in editor tabs not opened since, say an hour, could be unloaded from the editor and free up some memory. In practice, on low memory system, these pages are already swapped to disk. A "find in open files" will have to open them all, obviously, but that could be threated as a variant of "find in directory" which doesn't load them in the editor.

Ludo

Just ran a valgrind --massif --pages-as-heap=yes ./lazarus. Lazarus QT on kubuntu with 2G of memory, opening a small project (7 files). Without being an expert in the massif tool here is what it boils down to: Peak usage 165M divided into:
89M from dl-load
47M from /lib/dl.so which is the lazarus program itself (141M on disk )
37M from the various grapics initialisation routines loading the QT libs
2M glib libraries loaded from QT
remainder below ms_print's threshold.
20M KIconLoader (KPixmapCache) triggered by Qstyle::standardIcon from CONTROLS_TWINCONTROL_$_CREATEWND
20M SYSTEM_SYSOSALLOC <--- fpc
memory management
8M SYSTEM_GETMEM_FIXED from 90 places
2M SYSTEM_GETMEM_VAR in IMGLIST_TCUSTOMIMAGELIST
2M SYSTEM_GETMEM_VAR in CODECACHE_TCODECACHE <--- here's codetools
8M from several places all below ms_print's threshold. 17M unknown
8M Qthread (allocatestack in pthread_create) triggered by LCLINTF_SETFOCUS 2M QFontEngine
6M allocated with brk/sbrk from several places all below ms_print's threshold.

So, 50+% is program, resources and libraries. Codetools, which has been pointed to frequently, is peanuts. 20M in PixmapCache is enormuous. 17M unknown is a mistery. It's there from the first to the last massif sample at exactly the same size. The last 6M is interesting in that it continuously grows, compiling a project made it double but it never decreases, even when total memory usage dropped to 123M. This could be linked to the debugger having loaded symbols, etc.

If somebody is interested I can attach the complete massif and/or ms_print output.

Ludo

You have to set tabpagemax (i set mine to 500 for the test).

gvim (i just installed) does not have horizontal scrollbar, and if it wraps line it does not update the vertical scrollbar (maybe by design, but that's a terrible design, though one that saves a lot of time, hence good for the benchmark.....)

Of course there is a setting for this, too (wrap).

You have to make sure that you've set the highlighting for delphi or fpc. In either case, the highlighting is trivial. If you're taking ten times longer to open the file, then you're doing it wrong.

It's quite funny that you're trying to prove that lazarus is somehow more powerful than vim. It's the opposite way around. But that's not what I'm here to discuss. My point was that a really powerful editor is taking orders of a magnitude less time to open files than lazarus, so there is much scope for improvement.

Henry

That actually happens.

That explains the difference, between loading 450 files from command line (27 secs) and the same 450 files as part of "project1 session" (16 secs)
It appears, that the "lock" responsible for this, only occurs during opening the project (which is the common way in lazarus anyway), but not during loading files from command line.

Lazarus defers the scan for highlight+folding (one and the same) as well as finding the longest line (required for horiz scrollbar). Both of them are done once lazarus is fully operational, within idle, or if the tab is selected.
I realize for laptop user, the "idle operation" is undesired...

I do not know if Codetool scans are deferred however. (but afaik, codetools do scan on request)

I do believe (comparing with gvim and MSEide) that simply opening 450 tabs (even without content) may take some deal of the time (I have not tested that). It would explain at least some of the difference: gvim not opening tabs / MSEide does not use the widgetset, hence it doesn't need to allocate handles for each tab (I guess)

Again 450 files, is a benchmark only usage. Comparing against editors that skip the actions of the benchmark is no use. Comparing with geany, we actually look decent. And I don't even know if geany has something like codetools...

Similarly, unchanged files in editor tabs not opened since, say an hour, could be unloaded from the editor and free up some memory. In practice, on low memory system, these pages are already swapped to disk.

Which means unloading them, would actually be contra-productive. As you would access the memory to free it, and have to swap it in from disk. If the memory isn't touched then it want hurt anyone.

The same for the 450 univint files:

Peak usage 288M (450 files use 20M on disk):
89M from dl-load
47M from /lib/dl.so which is the lazarus program itself (141M on disk )
37M from the various grapics initialisation routines loading the QT libs
2M glib libraries loaded from QT
remainder below ms_print's threshold.
20M KIconLoader (KPixmapCache) triggered by Qstyle::standardIcon from CONTROLS_TWINCONTROL_$_CREATEWND
107M SYSTEM_SYSOSALLOC <--- fpc
memory management
35M SYSTEM_SYSGETMEM_FIXED
15M SOURCEEDITOR
11M from 167 places below threshold
6M from CLASSES_TCOLLECTION ADD COLLECTIONITEM (from SOURCEEDITOR_TSOURCEEDITOR)
various smaller blocks in SYNEDIT, EDITOROPTIONS, SYSUTILS, ...
70M SYSTEM_SYSGETMEM_VAR
30M from CODECACHE_TCODECACHE
6M from ZINFLATE_INFLATEINIT called from
GRAPHICS_CREATEBITMAPFROMLAZARUSRESOURCE
17M from LCONVENCODING_ISO_8859_1TOUTF8 from
CODECACHE_TCODECACHE DECODELOADED
3M from IMGLIST_TCUSTOMIMAGELIST
3M from CODECACHE_TCODEBUFFER LOADFROMFILE
1M from GRAPHICS_TCUSTOMICON LOADFROMRESOURCEHANDLE 1M from GRAPHICS_CREATEBITMAPFROMLAZARUSRESOURCE
remainder from several places
17M unknown
25M Qthread (allocatestack in pthread_create)
8M triggered by LCLINTF_SETFOCUS
8M by clone.S
9M by g_main_context_iteration
15M GIMAGE:paintEngine from synedit
2M QFontEngine
2M KMimeType from QFileDialog
8M allocated with brk/sbrk
3M Qimage_paintEngine from LCLINTF_DELETEOBJECT <---- ???? remainder from several places all below ms_print's threshold.

Bad news is that peak memory is reached after closing all files and opening a new small project (2 files in editor).

Ludo

Henry Vermaak wrote

You have to make sure that you've set the highlighting for delphi or fpc. In either case, the highlighting is trivial.

At least considering nested comments etc., this is not true :) I never used vim, but all fpc highlighting I saw so far in unix editors was far from perfect, it was trivial, indeed, but wrong :)

I haven't looked at changing the highlighter. No time to find out how, or even how to install....

but I added the max tab.

Now gvim takes 10 seconds
Lazarus 16 secs

That is not "orders of magnitude" ?

Even if you look at the 27 secs when using command line instead of project. This can be fixed in Lazarus, by applying 2 lines of code => once the proper code block has been found.

As for my comparison of the two. I am not trying to compare the overall power of the two.
Most of the functionality only uses resources, if it is triggered. So gvim may have a lot of editor commands lazarus does not have, they do not affect loading.

Again, I do not know gvim well enough. and I haven't analysed where the time in Lazarus really goes.
It may be that Lazarus is slower due to some design flaw, without any benefit provided... But it is equally possible, that some functionality in Lazarus, that gvim does not have) uses this time for a very good cause. e.g code tools scanning the files, may take some time (if it isn't deferred, and assuming gvim doe s not have a codetool equivalent).

-------------
Anyway, back to my first statement.

the time lazarus requires to load a huge amount of files is as I said "not bad". I never said excellent or top of the list. "not bad" of course means that you will be able to find individual editors that are faster. But if you look at a range of editors, you will equally find those that a slower. "geanny" took double (or even tripple) the time.

I did tests on windows half a year ago (don't remember the editors) But severla editors where slower or similar in speed than Lazarus. That would set Lazarus to be "average", and to me "average" is "not bad"

---------
As for "room for improvment":
- I never said there wasn't
- I said, that speed improvement (for loading) would not have to be not top priority
I stand by that. It already is average. That means while it does not top benchmarks, it serves well enough for normal daily usage.

-------------------
btw:

gvim 10 secs
Lazarus 16 or 27 secs
emacs: 20 secs
geany 57 secs
gedit 82 secs

Reopening the same 405 files (saved it as a project) ticks on 30M every time reopened... Closing files just leaves a lot of stuff which isn't re-used when re-opening the same files. Closing lazarus cleans up everything since no big leaks are reported.

Ludo

Vincent Snijders wrote

Another reason to have as little releases as possible. ;-)

LOL. The funniest comment I have seen in a while. :)

Henry Vermaak wrote

similar capability. ?As Graeme showed, we've become slower, so if we don't watch it, this kind of attitude will end us up with a sluggish, bloated ide.

Thank you. Somebody gets what I'm trying to get across. The problem is not not serious yet, but Lazarus is getting slower.

2011/8/13 Michael Thompson :

Does the memory usage lower when you then close the 100 units? ? Trying to eliminate leak as a possibility.

Closing all but one file, memory usage dropped by a whopping 2MB. I left the system idle for about 10 minutes while having a coffee brake. Memory usage was still the same, so there doesn't seem to be some delayed garage collection. :-/

Here again MSEide frees up resources nicely when I close files.

Are you sure?
Have you compared speed to 0.9.28? And which actions have you used for checking speed?

I know of several things that in 2010/2011 have had huge speed improvments. Some of them have been after 0.9,30. But some of them were between 0.9.28 and 0.9.30.
Things that speed up opening projects with many files have been improved before 0.9.30 and after.

At least the textmode IDE is then tested on a regular base ^^ (I personally prefer Lazarus for almost everything except simple test projects: for that I use nano :P )

Regards,
Sven

Martin wrote

Lazarus defers the scan for highlight+folding (one and the same) as well as finding the longest line (required for horiz scrollbar). Both of them are done once lazarus is fully operational, within idle, or if the tab is selected.

Would multi-threading help here - instead of idle timer process (which probably limits all the work to a single processor? Especially on today's multi-core CPU's? eg: create a thread pool of say 4 or 8 threads. Then hand each task required per tab (highlighting, folding, codetools processing etc) to a thread as they become available in the pool. Maximize the usage of available processors/cores, that is what they are there for.

Just a thought.

tabs / MSEide does not use the widgetset, hence it doesn't need to allocate handles for each tab (I guess)

I'm not sure how MSEide handles or allocates tabs, but I do know (from speaking to Martin) that MSEide only has one editor component instance. As you swap tabs, the buffer content of the editor component changes to that file.

I have no idea how Lazarus handles this. Would you mind explaining, I'm just curious.

actually look decent. And I don't even know if geany has something like codetools...

I hear a lot of talk about CodeTools. What exactly does it do? I'm just asking, because if it's about generating code etc, MSEide seems to be doing a great job too. Code-Completion, code templates, syntax highlighting, code navigation etc are all possible and working well in MSEide.

So I guess my question is, what does CodeTools do, and what does SynEdit (the editor component in Lararus) do?

Martin wrote

Are you sure? Have you compared speed to 0.9.28? And which actions have you used for checking speed?

Yet, at least between a 0.9.29 binary I had lying around, and 0.9.30.1

In the days of 0.9.26 and 0.9.28 I used the GTK1 interface. At that point LCL-GTK1 was considerably faster than the LCL-GTK2 - in the editor, dialogs, and general screen updates.

I know LCL-GTK2 has come a long way since then.

When I say slower, in this test case I can not the following.

- Loading time of all tabs (eg: when I tested with 100 or 200 units)
- tab / editor screen update. The editor was "blank" for the duration of loading
those units. MSEide was instant with near zero screen flicker.

My other concern was memory consumption. I know I have no clue as to what Lazarus IDE everything does, but I can say that for the generate usage of loading a project with some 5-15 units open, code navigating here and there etc.. the memory consumption of Lazarus is by magnitudes more than MSEide. Regarding what I was test or using in the editor, both IDE's had the same functionality.

I guess yes, though not urgent.
I guess the IDE will probably at some point start getting there, and the above task can be included.

The speed up at project loading would be minimal, probably only noticeable if you have several source-editors open (and therefore more than one file visible).
Yet it requires synchronisation. When a user switches tabs to an editor, that hasn't been processed, then this editor removes it's scanning from the idle handler, and finishes it from whatever point has been reached (sources can be partially scanned).

Well the start up speed could differ, as files could be opened and loaded in parallel.

currently, opens all files at startup
- creates a tab and an editor for it, and assigns the text to the lines
- codetool checks for resources, need to load a form?
- not sure if codetool needs anything else.

When all files are open, one tab will be activated. The editor for this file will perform a highlight scan. Also calculate the longest line, for the horiz scrollbar.

After that the IDE is ready and work can start.
Other editors will then start scanning on-idle

So in startup, only one editor (per open source-edit window) will be activated, and perform the scanning. This would not benefit much from threading. The file loading from disk itself, might benefit.

There is one editor instantiated for each tab. This does of course create extra handles, but I do not think that you will save much memory if you have only one instance. All the data hold by the editor, would have to be stored somewhere and swapped in and out as sources where changed.

Sure SynEdits could be changed to share one handle, and reduce the amount of handles needed. But I am not sure that will save much(timing wise, maybe memory wise, if there is a bitmap allocated for the synedit.

IIRC the handle is only allocated, once the tab is first time switched too.

Currently unused tabs do allocate memory for the highlight scan results etc. As this is added on idle.
That could be changed to be scanned only, once you switch to the tab. It would lead to a tiny delay on first time switching to a tab (not noticeable for normal files, but files with 10k or 100k lines might be noticeable).
In that context, I would be more worried about battery usage on a laptop. (which suffers from the on-idle work).

I am not the expert on codetools. And I do not know if it takes any of that time.

Apparently MSEide then must have it's own sort of codetools.

Codetools does the scanning of the source, to provide the navigation, completion, hints, integration of form resources....

So I guess my question is, what does CodeTools do, and what does SynEdit (the editor component in Lararus) do?

The only knowledge about pascal, that synedit has, is the highlighting, and with this the code folding, and divider-draw lines. All other pascal related work is codetools.

Question: do you have "Open designer on open unit" preference enabled?

-Fl?vio

Both GTK2 ? For the opeing of a project with *m any* files?

But well yes, if you had a 0.9.29, that already had the loading optimizations (and was faster than 0.9.28) then that could be.

Obviously the speed isn't constant. New features get added, the may take time while loading. The idea can not be, to keep the time always constant, or getting faster only. Then many features could never be added (your elastic tabs, if ever done, might mean a slow down... / though that falls under the deferred-to-idle scans).

In Laz trunk there where some speed improvements for project loading again (IIRC)

In the days of 0.9.26 and 0.9.28 I used the GTK1 interface. At that point LCL-GTK1 was considerably faster than the LCL-GTK2 - in the editor, dialogs, and general screen updates.

interesting. Gtk1 does not have support for propper scrolling. So scrolling in synedit under GTK1 is considerable slower. Though the scroll support under gtk2 was only added quite late. So for a long time that may have been true.

Otherwhise I can not tell much about gtk2. But then one would need to have some fundamental info about the difference of speed of the 2 GTKs themself.

You mean between gtk2 and gtk1? Well I can not speak about that.

- tab / editor screen update. The editor was "blank" for the duration of loading

The "blank" is partly intentional. Older lazarus did add tab by tab, updating tabs that wouldn't even be displayed. there was no point in that. That change was made in the IDE, affecting all widgetsets.

those units. MSEide was instant with near zero screen flicker.

Very interesting. I just added the 450 univint files to a project in mseide (though under windows, while all other tests where done under linux)

It takes 15 seconds to open (so yes it is a little faster than lazarus, but not such massively)

My version may be a few month old....

All I know is, when I open the ide project (mseide ide project / whatever the projectfile is called) itself, and then try to open file, it always prompts me for missing form files or similar...

Then again, maybe my config got screed along the way...

as for it codetools equivalent: Some thinks don't jump, but may be my config. I found a link I just clicked, which took over 2 secs before the mse-ide actually jumped (at least it showed a "busy" mouse pointer)

Let me just stress out: I do not mean in any way to say, that this is in anyway something bad about mse-ide. I only observe that some of the statements made before are not reproducible n my system. MSEide does require time too (memory I have not analysed.)

Martin wrote

Well the start up speed could differ, as files could be opened and loaded in parallel.

Yes, this is what I was thinking too. Plus the GUI could be more "snappy" (updated quicker). As you stated, if I had to use the 400 units example again, currently Lazarus would create 400 tabs and 400 synedit instances, then only does the ide get chance to update the GUI for the current active tab. This is when I see the "blank" editor window for a few seconds while the tabs are being created - making the user experience much worse.

Being multi-threaded, the GUI could update the active tab almost instantly, while the other threads create the extra tabs and synedit instances in parallel.

Apparently MSEide then must have it's own sort of codetools.

I guess so. Probably just the terminology used is different.

The only knowledge about pascal, that synedit has, is the highlighting, and with this the code folding, and divider-draw lines. All other pascal related work is codetools.

OK, thanks for explaining the differences and what each one does.

There was a missing beginupdate()/endupdate() so the load time was O(n?). Should be faster soon. ;-)

Martin

Well one thing, I noted is that MSEide does not seem to deal with nested comments, that opens the question how much context the remaining highlighting takes into account.
The more context is needed, the more time and memory it will cost.

There were plenty of reports (some by you IIRC Graeme) like "deprecated" being highlighted in the wrong place. That was fixed. But of course it requires to analyse the source much more detailed. (yet it is still fast)

Well, I think we're getting off the track here, since opening hundreds of files isn't a realistic/important use-case, but anyway...

I don't think multi-threading is the desired approach. IMO ideally we should have only one TSynEdit per window, and just switch "state" when switching tabs.
Also, parsing should be delayed to until the editor (or the TODO window, etc) is displayed.
BTW does anybody know how much memory gtk2 uses because of double-buffering?

-Fl?vio

Since you are on it. Same issue, if you open the 450 files from the open-file dialog (with multi select)

Additional, if you do open them like that (into an otherwhise empty project) some files stay without highlighting. They will be, if you close and re-open the project

Watching the process, it appears, that when opeing a project, MSE-ide does not load all the files for which it creates tabs? At least sysutils does not show any disk activity to that extend.
While if I open them via the file-open (into an empty project) then it appears they are loaded?

If the first assumption is true, then there is little point in comparing mse ide and Lazarus. Lazarus is designed to open them. (e.g It needs to know if the file refers to a form/lfm // I don't know what else may require an immediate load of the file).
However that is a fixed difference. That will not get worse with each lazarus version (which was the expressed concern, that started this discussion)

The default pascal.sdef file is "Delphi compatible". Please change in apps/ide/syntaxdefs/pascal.sdef
"
scope comment1 comment
endtokens
'}'
"
to
"
scope comment1 comment
calltokens
'{' comment1
endtokens
'}'
"
for nested comments.

The more context is needed, the more time and memory it will cost.

MSEide does not store context for syntax highlighting after the source has been colored. The editor grid stores "richstringty": "
richstringty = record

  text: msestring;
  format: formatinfoarty;
  flags: richflagsty;

end;
"
"format" stores the color and font info of the string. The parser for code navigation is independent from syntaxhighlighting.

Martin

Martin Schreiber wrote

The default pascal.sdef file is "Delphi compatible". Please change in apps/ide/syntaxdefs/pascal.sdef " scope comment1 comment endtokens '}' " to " scope comment1 comment calltokens '{' comment1 endtokens '}' " for nested comments.

Thanks, that worked

Well in a way that stores the info too. I assume you store the formatinfoarty for the begin of each line?

In that case it is pretty similar to SynEdit

The parser for code navigation is independent from syntaxhighlighting.

Same in Lazarus

Martin Schreiber wrote

Correct. After the fix of the missing beginupdate()/endupdate() MSEide should open the project and create the 400 editor tabs in 2..3 seconds.

Yes, I have the update, less than 1 sec.

Correct.

If the first assumption is true, then there is little point in comparing mse ide and Lazarus.

Correct. After the fix of the missing beginupdate()/endupdate() MSEide should open the project and create the 400 editor tabs in 2..3 seconds.

Martin

The source code is stored in a trichstringdatalist which is an array of richstringty, an array item is the text of a grid row.

In that case it is pretty similar to SynEdit

Probably.

Martin

2011/8/13 Fl?vio Etrusco :

Question: do you have "Open designer on open unit" preference enabled?

I have no idea. But it is most likely not relevant either. My projects are all fpGUI based, and that's the projects I used to test Lazarus with. Those projects don't use *.lfm files like LCL apps do. To Lazarus, those units are just source code.

Martin wrote

> > Yet, at least between a 0.9.29 binary I had lying around, and 0.9.30.1 Both GTK2 ? For the opeing of a project with *m any* files?

Yes, the old 0.9.29 and new 0.9.30.1 versions of Lazarus was both compiled against GTK2 widgetset.

Obviously the speed isn't constant. New features get added, the may take time while loading.

New features don't always need to mean slower and more memory hungry. Just look at Mozilla Firefox project. The last few versions came with new features, but also with speed improvements and less memory hungry. So it is possible. :)

or getting faster only. Then many features could never be added (your elastic tabs, if ever done, might mean a slow down... / though that falls under the deferred-to-idle scans).

I've seen implementations of Elastic Tabstops (ET) that are slow and others that are fast. There are different algorithms and implementations to implementing ET. jEdit is quit fast at everything (especially for a Java desktop app). They are introducing ET support in their next version. I am very curious to see how fast that will be.

In Laz trunk there where some speed improvements for project loading again

Excellent. I haven't really tried Trunk much since the 0.9.30 release.

> ?- Loading time of all tabs (eg: when I tested with 100 or 200 units) You mean between gtk2 and gtk1?

No, between Lazarus IDE and MSEide - how long it takes to create 100 or 200 tabs.

Can't you have one tab as active, and add more tabs without changing the active tab? Does LCL always set focus (active tab) to the newly created tab sheet? That's what it looks like to me (when I tested Friday at work).

Very interesting. ?I just added the 450 univint files to a project in mseide (though under windows, while all other tests where done under linux)

I only tested under Linux, so can't comment on the Windows version.

It takes 15 seconds to open (so yes it is a little faster than lazarus, but not such massively)

Mine was a huge difference. I use the latest revision of MSEide. But I don't follow to closely what Martin adds or changes to MSEide&MSEgui. Just like with Lazarus IDE, I use MSEide as a stand-alone product (editor) for my fpGUI based projects.

All I know is, when I open the ide project (mseide ide project / whatever the projectfile is called) itself, and then try to open file, it always prompts me for missing form files or similar...

I remember I used to get those too, but don't any more. I think it was related to search paths in the "Project Options | Debugger tab". Not 100% sure.

as for it codetools equivalent: ?Some thinks don't jump, but may be my config.

This works differently to Lazarus IDE. Lazarus always adds all code (I think), even from FPC src directories. With MSEide you can tell it what code it should parse to allow navigation of code. I think this is what speeds MSEide up as well - by default it has less work to do. This is setup under "Project Options | Debugger tab | Source Directories tab". I have project templates that are already pre-configured for what I want or need.

anyway something bad about mse-ide. I only observe that some of the statements made before are not reproducible n my system.

That's understandable. MSEide has very different settings to Lazarus IDE (just like any other IDE would have too). It took me a while to find my way around MSEide. But now I am pretty efficient in using both Lazarus IDE and MSEide. I often switch between the two at work - sometimes even for the same project. At home I only run MSEide because I have a aging Intel P4 laptop which is 8 years old, and MSEide is just faster on it than Lazarus and uses much less memory. I don't even run Gnome on my laptop any more. :)

Graeme Geldenhuys wrote

Closing all but one file, memory usage dropped by a whopping 2MB. I left the system idle for about 10 minutes while having a coffee brake. Memory usage was still the same, so there doesn't seem to be some delayed garage collection. :-/

How do you measure memory usage? Maybe it is the heapmanager, not lazarus that has a problem. (and then you are not that far off with the delayed garbage collection)

Martin wrote

Well one thing, I noted is that MSEide does not seem to deal with nested comments,

Florian also mentioned something like that. First, nested comments give me compiler warnings, so I never do that. I listen to the FPC compiler. :) And if I must, my outer comments then use (* .... *) style commenting, which works fine in both Lazarus IDE and MSEide.

So could you give a code example of what you mean?

There were plenty of reports (some by you IIRC Graeme) like "deprecated" being highlighted in the wrong place.

That seems to work fine in my copy of MSEide, and my syntax highlighting scheme. I don't use default syntax highlighting schemes in Lazarus or MSEide. Attached is two examples in MSEide where deprecated is correctly highlighted.

I can't remember the example I reported in Lazarus - I'll search the Mantis reports.

Graeme Geldenhuys wrote

I have a aging Intel P4 laptop which is 8 years old, and MSEide is just faster on it than Lazarus and uses much less memory. I don't even run Gnome on my laptop any more. :)

[Off Topic]
I run Lazarus on a 11 year old 700 Mhz Celeron with 512 Mb using both Windows (WinME) and Linux (Suse 10.0)/GTK2.
It's definately not as fast as Delphi 3, but it has outgrown it.

Then again, I only program for my hobby, so I don't mind (although in Linux GTK2 dragging components etc. in the formdesigner is way slower than win32).
[/Off Topic]

Bart

2011/8/13 Fl?vio Etrusco :

I don't think multi-threading is the desired approach.

Why not, it works for my applications I develop. It makes the UI much more "snappy" to the end-user as well. All new PC's and millions of existing PC's already have multi-core CPU's so why not use those features. Why limit yourself to a single threaded application?

IMO ideally we should have only one TSynEdit per window, and just switch "state" when switching tabs.

If I understood Martin correctly, Lazarus IDE still does processing on the other files, so a delay is still to be expected because it is single threaded, and the "active tab" is only set and processed at the _end_ of processing all the opened files. By processing I mean: creating a tab sheet, creating a synedit instance, open and load the file. Other processing like scanning the unit by codetools is done only when the IDE is idle.

Also, parsing should be delayed to until the editor (or the TODO window, etc) is displayed.

I understood that this is already how Lazarus IDE works.

BTW does anybody know how much memory gtk2 uses because of double-buffering?

MSEide also does double buffering as far as I know (who doesn't these days), and it still uses much less memory.

Don't get me wrong. Yes I mentioned the delay in the tabs when loading 100+ units. But that is not my major concern, because that is not normal daily usage. Though a benchmark is know not to be normal daily usage. My main concern is how greedy Lazarus IDE is becoming with memory.

OT: at least say (fireforx) say so. For my way of using it, it hasn't changed a bit...

Anyway, I did not say, that all new feature mean a slow down. Some may do however. Even if they are fast in them self, if additional work needs to be performed, it will take time or memory or both.

That does not conflict, with the ability to speed up other existent code. As I equally told you have happened:
- since 0.9.28: All highlighter scanning except the visible tab, have moved to idle time, instead of start up time
- after 0.9.30: project management of open files has a hash or tree structure, instead of linear search.

Both of them where very noticable speedups. I do not know if part of the gained speed was lost to other additions since 0.9.28. But if you whis you can always get yourself a 0.9.28 and do the test.

Other speed ups, unrelated to project loading have also been made. One that comes to mind was codetools checking for files modified on disk => which could stop the entire IDE for half a minute, because filefunctions on windows seem to be magnitude slower than linux.

So as far as I can see speed is watched, where it has an impact on normal use cases.
But I am not sure how high the priority is of getting speed to win a benchmark (there is plenty other, better stuff to be done)

I didn't say that they are necessarily slow.

I did say they will mean a slow down => that may be a very minor slow down, but they will need an amount of time gretaer than zero to work. And with a very high likelihood, they will need an amount of time greater than the current tab implementation. So the will mean a slow down

>> - Loading time of all tabs (eg: when I tested with 100 or 200 units) > You mean between gtk2 and gtk1? No, between Lazarus IDE and MSEide - how long it takes to create 100 or 200 tabs.

As found out in another msg. MSEide only creates the tabs, but doesn't load the files.

I don't know if loading the files can even be omitted. After all the IDE must know if they contain a {$R *.lfm}, in order to decide if a form needs to be loaded in the designer.

Sure you could save it in the session, and hope that never an update from svn/git does change a file to suddenly have a form... But I am not sure someone is going to rewrite the entire project management/loading, only to win a benchmark.

I don't know enough about that, and it may depend on the widgetset too. maybe windows, or gtk are forcing it?

But even if, then if you allowed the user to start editing, and all the sudden new forms open in the designer.... not useful at all and again: I doubt it is worth the amount of work it requires. It certainly isn't to me.

I've got a for (or maybe five) year old computer, with a Intel Dual core 6600 2.4ghz CPU. It takes 16 seconds for 450 files. I don't know if anyone has such a number of files open in real life, you can't maintain them anyway, you never find a file through the tabs, and neither through any menu.

Martin wrote

ide and Lazarus. Lazarus is designed to open them. (e.g It needs to know if the file refers to a form/lfm // I don't know what else may require an immediate load of the file).

This might sound silly, but Why? If the project opens lots of units, those are just source code units. Forms should only come into play when the developers presses F12 to show the Form for a specific unit.

Or am I not understanding Lazarus's form designer or codetools correctly? Are you saying that if Lazarus opens a project with 50 units that are actually the source code of form declarations, then Lazarus will also display 50 forms when loading that project?

Those are examples of it in the right place.
the reported trouble was for something like this

procedure Deprecated;
procedure Foo(deprecated: Boolean);
type
   deprecated = boolean;
var
   a: deprecated;

etc.

MSE-ide appears to highlight quite some of them.

Marco van de Voort wrote

How do you measure memory usage?

Two ways:

1) 'gnome-system-monitor', look in the Processes tab, there is a Memory column.

2) Use 'top -p <processid>. Subtract the SHR value from the RES value.

What is what I mean by "memory usage".

Martin wrote

So as far as I can see speed is watched, where it has an impact on normal use cases.

OK, thanks.

Maybe all that is required is: pagecontrol.beginupdate ... pagecontrol.endupdate. ;-)

Then the control should only need to redraw once, and not per tab.

I don't know if anyone has such a number of files open in real life, you can't maintain them anyway, you never find a file through the tabs, and neither through any menu.

Martin does in MSEide, and I am starting to use it in a similar fashion. Open those units in MSEide. Then press Ctrl+E. A form will popup with a grid which is searchable by just starting to type, or navigating quickly with the arrow keys.

I used to put all my units inside a "project inspector" like tree structure. But having hundreds of units makes navigation no faster even if they are sorted alphabetically. You just can't see enough items at a glance. Martin's idea in MSEide works quite well.

And yes, you are correct. Navigation in Lazarus with that amount of units is totally useless. Even the popup menu when you right click on the tab, or the Window menu is just to large, and scrolling that menu takes forever (in GTK2).

Threading may archive a speed up here. Because simple you have more CPU most OS will be able to supply files faster if
requested in parallel.

But all the above work, must be finished, before the IDE can hand over to the user. See my other mails. If you let the user work in one tab, while other tabs are still loading files, then designer forms may still would really interrupt everyone's work-flow....

> Also, parsing should be delayed to until the editor (or the TODO > window, etc) is displayed. I understood that this is already how Lazarus IDE works.

At least for highlighting.

codetools only Mattias can answer. But I believe that some parts of code tools only scan on demand, so it may well be. Yet finding a {$R *lfm} can not be deferred.

Don't get me wrong. Yes I mentioned the delay in the tabs when loading 100+ units. But that is not my major concern, because that is not normal daily usage.

+1 glad we agree

Though a benchmark is know not to be normal daily usage. My main concern is how greedy Lazarus IDE is becoming with memory.

Well then someone needs to find out where it goes....

Just testing on windows.
- freshly starting lazarus, automatically re-opens last project: 12 tabs in 2 source-edit windows => 49.8 MB memory
- adding the 450 univint files => 205.1 MB
the files contain 16.7 MB actual data
Also noting that drag and drop opening of multiply files is slower, as there is an Begin/EndUpdate missing
- yes closing pages doesn't close free much memory at all Not even closing the project

It goes down to 201.9 MB
Interestingly it hasn't even freed the 16.7 MB of raw data. So part of this must be the memory management. Why? Because:

I know for fact that the synedits get destroyed. At least last time I worked on it, the did get destroyed, so unless there is a real huge new bug....
If the SynEdits get destroyed, then the textbuffers get destroyed too, including the strings containing the text (one string per line, and I am pretty sure no other part of the IDE has a ref count on any of them.)

So the memory is probably available, but due to fragmentation, and small pieces still being allocated, it can't be returned to the OS.

if the source was open, yes. (afaik), maybe there is an option...

I don't know for sure, if the project would remember, if the form was closed, but the source open.

But assuming that normally form and source are open together: If you had 50 fomrs open when you closed the project, then you can have them open again, or not?

Otherwise, you probably wouldn't have had the units open either.

If you open a project , that is already done.
The page control draws only at the very end (on windows) I don't know if gtk supports that...

If you load files by other means, the lock is not always applied. Obviously work that needs to be done

I don't think multi-threading is the desired approach. IMO ideally we should have only one TSynEdit per window, and just switch "state" when switching tabs.

Then we need new tabs, because the current implementation allocates a page per tab.

DoDi

Threading may archive a speed up here. Because simple you have more CPU most OS will be able to supply files faster if requested in parallel.

I doubt that, because concurrent file reads tend to increase disk seeks.

I know for fact that the synedits get destroyed. At least last time I worked on it, the did get destroyed, so unless there is a real huge new bug....

The source files may be needed for other purposes, so that they stay in the file pool.

If the SynEdits get destroyed, then the textbuffers get destroyed too, including the strings containing the text (one string per line, and I am pretty sure no other part of the IDE has a ref count on any of them.)

Hmmm.

So the memory is probably available, but due to fragmentation, and small pieces still being allocated, it can't be returned to the OS.

Why return memory to a nowadays system?

DoDi

In MSEide the *.pas and *.mfm (the MSEgui *.lfm equivalent) files are independent. A MSEgui form can be loaded by reading the *.mfm file alone. MSEide does not need to parse the library source for MSEgui form handling, all needed information is the RTTI compiled into MSEide . This were MSEide+MSEgui design decisions at the start of the project in order to allow to build a fast, stable and versatile developmnent environment.

Martin

Correct. MSEide uses a very simple syntax highlighing method and does not parse the code in deep.

Martin

An additional method is to double click on the most used editor tabs in order to move them on the top of the tab list. Changed files are marked with a "*" and easily visible, see attachment. The limit is at about 100-200 "open" files. 400 are too much because the Ctrl+E grid needs scrolling.

Martin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: editortabs.png
Type: image/png
Size: 16360 bytes
Desc: not available
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20110814/38bfb018/attachment-0001.png>

Clearly you haven't looked at the valgrind massif data I posted earlier. They clearly show that a lot of data is staying in memory. It'll require a lot of digging to figure out exactly where the problems are. One I found out is that synedit creates an image list with gutter symbols for every page! That adds up to 3 M for the 450 file, which apparenltly aren't released when closing the page. An other one is in zlib when reading lfm resources (there aren't any in the 450 tes files afaik). Another few meg's hanging around.

Ludo

Maybe you could create a small IDE package that provides an interface to the GetHeapStatus (
http://www.freepascal.org/docs-html/rtl/system/getheapstatus.html ) and GetFPCHeapStatus (
http://www.freepascal.org/docs-html/rtl/system/getfpcheapstatus.html ) functions by e.g. a menu entry to check whether it's just the heap manager that didn't release the memory chunks yet.

Regards,
Sven

Sorry, no I didn't. Also i do not know enough about fpc memory management (and maybe it even depends on the memory manager.

Does fpc hand calls for memory directly through to the OS? I thought fpc does allocate some meory, and then return slices of this when the app requires it? the heap trace for example definetly keeps some memory around, as it is checksummed, and later tested for modifications by invalid memory access.

Actually, I thing my figures where on a Laz compiled with heap trace => so memory would not be freed for real.

Does valgrind know about the fpc internal mem handling

They clearly show that a lot of data is staying in memory. It'll require a lot of digging to figure out exactly where the problems are. One I found out is that synedit creates an image list with gutter symbols for every page!

Well there is a synedit for every page, every synedit is acting completely alone currently. (except for sharing the highlighter.)

Well yes, actually, the highlighter keeps some memory during the entire IDE lifetime. However, as long as you keep opening pascal files most of the structures in this memory are re-used.

That adds up to 3 M for the 450 file, which apparenltly aren't released when

Hm, strange, not sure which one you refer too.
BookmarOptions has an image list (the digits for bookmarks), but the image list is actually a single instance created by SourceEditorMarks)

I haven't gone through all the rest of code, but I just checked by debugging the IDE. SynEdit itself definitely gets destroyed when the page is closed. And I have not yet seen a memory leak in regard to this...

closing the page. An other one is in zlib when reading lfm resources (there aren't any in the 450 tes files afaik). Another few meg's hanging around.

is zlib used on windows? my figures where on windows?

anyway: we have 200 MB - 50 that were already allocated = 150

16.7 MB real text, but split into sting by line. I haven't counted the lines, but they a line is avg 20 bytes => 800K lines, each of them a string (4 bytes pointer, mem alloc overhead, len => at least 12 bytes) another 8 MB, synEdit needs other per line info, for highlight and similar things. estimate another 8 MB

Codetool may keep separate copies of the text or parts of it, as it uses a different way of storage, another up to 16 MB (+ mem manager overhead)

So far that makes 50 MB + 3MB image lists => still a lot missing.

I am not sure if we look at any bitmap memory for synedits paint area. SynEdits handle are only created when you switch to the tab => before this, I would not expect bitmaps to be allocated? But I do not know for sure. The memory however is gone, before I ever switch a tab, that is only one synedit got a Handle allocated.

-----------
Now doing the same test without heaptrace

- Lazarus just started 36.8 MB
- added the 450 univint files 137.8 MB / significantly less than the 200 that I had with heap trc
- but yet no memory freed (to OS) when closing project

Whil the lack of freeing memory may in parts be caused by Lazarus, (do not know), it is not entirely caused by Lazarus.
There is definetely some memory freed. And this freed mem is not going back to the OS for a reason outside the control of Lazarus.

I do not know, if valgrind would detect such mem as freed or not.

Valgrind massif looks at the heap and can't distinguish memory released but kept by the memory manager or memory not released. For massif it is allocated by the program under test and reports where it was allocated.

Here is a chunck of the valgrind output I spent some time on and which I'm referring to (this is lazarus Qt):

| ->05.44% (15,728,640B) 0x43F2854: _int_malloc (malloc.c:3226) | | ->05.44% (15,728,640B) 0x43F3F9A: malloc (malloc.c:3660) | | ->02.54% (7,340,032B) 0x582B62E: ??? (in /usr/lib/libQtGui.so.4.6.2) | | | ->02.54% (7,340,032B) 0x582BE64: ??? (in /usr/lib/libQtGui.so.4.6.2) | | | ->02.54% (7,340,032B) 0x57068D9: QImage::paintEngine() const (in /usr/lib/libQtGui.so.4.6.2)
| | | ->01.45% (4,194,304B) 0x57A598C: QPainter::begin(QPaintDevice*) (in /usr/lib/libQtGui.so.4.6.2)
| | | | ->01.45% (4,194,304B) 0x57A6306:
QPainter::QPainter(QPaintDevice*) (in /usr/lib/libQtGui.so.4.6.2) | | | | ->01.45% (4,194,304B) 0x4175932: QPainter_create2 (qpainter_c.cpp:24)
| | | | ->01.45% (4,194,304B) 0x828E22C: QTINT_TQTWIDGETSET_$__SELECTOBJECT$HDC$HGDIOBJ$$HGDIOBJ (./qt/qtwinapi.inc:4773)
| | | | ->01.45% (4,194,304B) 0x81C0206: LCLINTF_SELECTOBJECT$HDC$HGDIOBJ$$HGDIOBJ (./include/winapi.inc:886) | | | | ->01.45% (4,194,304B) 0x81CBFA7: GRAPHICS_TBITMAPCANVAS_$__CREATEHANDLE (./include/bitmapcanvas.inc:57) | | | | ->01.45% (4,194,304B) 0x81D410F: GRAPHICS_TCANVAS_$__REQUIREDSTATE$TCANVASSTATE (./include/canvas.inc:1609) | | | | ->01.09% (3,145,728B) 0x81D2E8B: GRAPHICS_TCANVAS_$__FILLRECT$TRECT (./include/canvas.inc:929) | | | | | ->01.09% (3,145,728B) 0x81D2EF3: GRAPHICS_TCANVAS_$__FILLRECT$LONGINT$LONGINT$LONGINT$LONGINT (./include/canvas.inc:939)
| | | | | ->01.09% (3,145,728B) 0x868A582: SYNGUTTERCODEFOLDING_TSYNGUTTERCODEFOLDING_$_INITPOPUPIMAGELIST_NEWIMG (synguttercodefolding.pp:193)
| | | | | ->01.09% (3,145,728B) 0x868A394: SYNGUTTERCODEFOLDING_TSYNGUTTERCODEFOLDING_$__INITPOPUPIMAGELIST (synguttercodefolding.pp:201)
| | | | | ->01.09% (3,145,728B) 0x868AE94: SYNGUTTERCODEFOLDING_TSYNGUTTERCODEFOLDING_$__CREATE$TCOMPONENT$$TSYNGUTTERC ODEFOLDING (synguttercodefolding.pp:302)
| | | | | ->01.09% (3,145,728B) 0x87CACF7: SOURCESYNEDITOR_TIDESYNGUTTER_$__CREATEDEFAULTGUTTERPARTS (sourcesyneditor.pas:564)
| | | | | ->01.09% (3,145,728B) 0x8687B10: SYNGUTTER_TSYNGUTTER_$__CREATE$TSYNEDITBASE$TSYNGUTTERSIDE$THETEXTDRAWER$$TS YNGUTTER (syngutter.pp:109)
| | | | | ->01.09% (3,145,728B) 0x87C9A35: SOURCESYNEDITOR_TIDESYNEDITOR_$__CREATEGUTTER$TSYNEDITBASE$TSYNGUTTERSIDE$TH ETEXTDRAWER$$TSYNGUTTER (sourcesyneditor.pas:219)
| | | | | ->01.09% (3,145,728B) 0x863C124: SYNEDIT_TCUSTOMSYNEDIT_$__CREATE$TCOMPONENT$$TCUSTOMSYNEDIT (synedit.pp:1671)
| | | | | ->01.09% (3,145,728B) 0x87C9AD0: SOURCESYNEDITOR_TIDESYNEDITOR_$__CREATE$TCOMPONENT$$TIDESYNEDITOR (sourcesyneditor.pas:224)
| | | | | ->01.09% (3,145,728B) 0x84BDD70: SOURCEEDITOR_TSOURCEEDITOR_$__CREATEEDITOR$TCOMPONENT$TWINCONTROL (sourceeditor.pp:3996)
| | | | | ->01.09% (3,145,728B) 0x84B8F57:
SOURCEEDITOR_TSOURCEEDITOR_$__CREATE$TCOMPONENT$TWINCONTROL$TSOURCEEDITOR$$T SOURCEEDITOR (sourceeditor.pp:2435)
| | | | | ->01.09% (3,145,728B) 0x84C3F4A:
SOURCEEDITOR_TSOURCENOTEBOOK_$__NEWSE$LONGINT$LONGINT$TSOURCEEDITOR$ANSISTRI NG$$TSOURCEEDITOR (sourceeditor.pp:5801)
| | | | | ->01.09% (3,145,728B) 0x84C66AA:
SOURCEEDITOR_TSOURCENOTEBOOK_$__NEWFILE$ANSISTRING$TCODEBUFFER$BOOLEAN$TSOUR CEEDITOR$$TSOURCEEDITOR (sourceeditor.pp:6552)
| | | | | ->01.09% (3,145,728B) 0x81454A1:
MAIN_TMAINIDE_$__DOOPENFILEINSOURCEEDITOR$TUNITEDITORINFO$LONGINT$LONGINT$TO PENFLAGS$$TMODALRESULT (main.pp:8385)
| | | | | ->01.09% (3,145,728B) 0x814801D: MAIN_TMAINIDE_$__DOOPENEDITORFILE$crc4B94F828 (main.pp:9403)
| | | | | ->01.09% (3,145,728B) 0x8147565:
MAIN_TMAINIDE_$__DOOPENEDITORFILE$ANSISTRING$LONGINT$LONGINT$TOPENFLAGS$$TMO DALRESULT (main.pp:9123)

This are the gutter symbols being created 450 times. Now is this the QT handle not being released correctly? That is possible but there are too many of these. Like I said before, 30M is lost opening the 450 files. Everytime they are reopened. Probabaly some stuctures are reused, but 30M isn't. BTW I wrote a small testapp that simulates creating and releasing small bitmaps, pretty much similar to INITPOPUPIMAGELIST_NEWIMG and made it create a few 100000 images in an imagelist. After freeing the imagelist, all the memory was released to the OS. Does fragmentation of memory cause the fpc memory manager not to return all memory to the OS? That is a possibility.

Another example?

100.00% (288,964,608B) (page allocation syscalls) mmap/mremap/brk, --alloc-fns, etc.
->37.20% (107,483,136B) 0x80C7DD3: SYSTEM_FPSYSCALL$LONGINT$LONGINT$$LONGINT (in /home/ludo/lazarus/lazarus)
| ->37.20% (107,483,136B) 0x80C86DF:
SYSTEM_FPMMAP$POINTER$LONGWORD$LONGINT$LONGINT$LONGINT$INT64$$POINTER (in /home/ludo/lazarus/lazarus)
| ->37.20% (107,483,136B) 0x80DE2EE: SYSTEM_SYSOSALLOC$LONGWORD$$POINTER (in /home/ludo/lazarus/lazarus)
| ->13.06% (37,748,736B) 0x80DEFB7:
SYSTEM_ALLOC_OSCHUNK$PFREELISTS$LONGWORD$LONGWORD$$POINTER (in /home/ludo/lazarus/lazarus)
| | ->13.06% (37,748,736B) 0x80DF396:
SYSTEM_SYSGETMEM_VAR$LONGWORD$$POINTER (in /home/ludo/lazarus/lazarus) | | ->10.52% (30,408,704B) 0x8382C3A:
LCONVENCODING_ISO_8859_1TOUTF8$ANSISTRING$$ANSISTRING (lconvencoding.pas:4619)
| | | ->10.52% (30,408,704B) 0x8389C25:
LCONVENCODING_CONVERTENCODING$ANSISTRING$ANSISTRING$ANSISTRING$$ANSISTRING (lconvencoding.pas:6389)
| | | ->10.52% (30,408,704B) 0x81593E7:
MAIN_TMAINIDE_$__ONCODEBUFFERDECODELOADED$TCODEBUFFER$ANSISTRING$ANSISTRING$ ANSISTRING$ANSISTRING (main.pp:14590)
| | | ->10.52% (30,408,704B) 0x8368709:
CODECACHE_TCODECACHE_$__DECODELOADED$TCODEBUFFER$ANSISTRING$ANSISTRING$ANSIS TRING$ANSISTRING (codecache.pas:784)
| | | ->10.52% (30,408,704B) 0x8369C80: CODECACHE_TCODEBUFFER_$__DECODELOADED$ANSISTRING$ANSISTRING$ANSISTRING$ANSIS TRING (codecache.pas:1153)
| | | ->10.52% (30,408,704B) 0x836D3D9: SOURCELOG_TSOURCELOG_$__LOADFROMFILE$ANSISTRING$$BOOLEAN (sourcelog.pas:825) | | | ->09.07% (26,214,400B) 0x83697B0: CODECACHE_TCODEBUFFER_$__LOADFROMFILE$ANSISTRING$$BOOLEAN (codecache.pas:1045)
| | | | ->09.07% (26,214,400B) 0x83671DF: CODECACHE_TCODECACHE_$__LOADFILE$ANSISTRING$$TCODEBUFFER (codecache.pas:347) | | | | ->09.07% (26,214,400B) 0x83A63E1: CODETOOLMANAGER_TCODETOOLMANAGER_$__LOADFILE$ANSISTRING$BOOLEAN$BOOLEAN$$TCO DEBUFFER (codetoolmanager.pas:1109)
| | | | ->09.07% (26,214,400B) 0x85A0B18: DIALOGPROCS_LOADCODEBUFFER$TCODEBUFFER$ANSISTRING$TLOADBUFFERFLAGS$BOOLEAN$$ TMODALRESULT (dialogprocs.pas:224)
| | | | ->09.07% (26,214,400B) 0x813E5B7: MAIN_TMAINIDE_$__DOOPENUNKNOWNFILE$ANSISTRING$TOPENFLAGS$TUNITINFO$BOOLEAN$$ TMODALRESULT (main.pp:6395)
| | | | ->09.07% (26,214,400B) 0x8147F58: MAIN_TMAINIDE_$__DOOPENEDITORFILE$crc4B94F828 (main.pp:9380) | | | | ->09.07% (26,214,400B) 0x8147565: MAIN_TMAINIDE_$__DOOPENEDITORFILE$ANSISTRING$LONGINT$LONGINT$TOPENFLAGS$$TMO DALRESULT (main.pp:9123)
| | | | ->09.07% (26,214,400B) 0x812EBE3: MAIN_TMAINIDE_$__SETUPSTARTPROJECT (main.pp:2253)
| | | | ->09.07% (26,214,400B) 0x812B951: MAIN_TMAINIDE_$__STARTIDE (main.pp:1456)
| | | | ->09.07% (26,214,400B) 0x80C7AFF: main (lazarus.pp:113)

That is lconvencoding.pas hanging on to 30M of data. Remember, these figures are taken after closing the 450 files and opening a 2 file project.

is zlib used on windows? my figures where on windows?

With zlib, I mean zinflate etc. Not the dll/so. Here is the chunk:

| ->05.63% (9,175,040B) 0x80DEF8A:
SYSTEM_ALLOC_OSCHUNK$PFREELISTS$LONGWORD$LONGWORD$$POINTER (in /home/ludo/lazarus/lazarus)
| | ->05.63% (9,175,040B) 0x80DF396:
SYSTEM_SYSGETMEM_VAR$LONGWORD$$POINTER (in /home/ludo/lazarus/lazarus) | | ->01.77% (2,883,584B) 0x821F586:
ZINFLATE_INFLATEINIT$Z_STREAM$$SMALLINT (in /home/ludo/lazarus/lazarus) | | | ->01.77% (2,883,584B) 0x8219A1D:
ZSTREAM_TDECOMPRESSIONSTREAM_$__CREATE$TSTREAM$BOOLEAN$$TDECOMPRESSIONSTREAM (in /home/ludo/lazarus/lazarus)
| | | ->01.77% (2,883,584B) 0x8243E46:
FPREADPNG_TFPREADERPNG_$__INTERNALREAD$TSTREAM$TFPCUSTOMIMAGE (in /home/ludo/lazarus/lazarus)
| | | ->01.77% (2,883,584B) 0x8215565:
INTFGRAPHICS_TLAZREADERPNG_$__INTERNALREAD$TSTREAM$TFPCUSTOMIMAGE (intfgraphics.pas:5853)
| | | ->01.77% (2,883,584B) 0x81F7C72: FPIMAGE_TFPCUSTOMIMAGEREADER_$__IMAGEREAD$TSTREAM$TFPCUSTOMIMAGE$$TFPCUSTOMI MAGE (in /home/ludo/lazarus/lazarus)
| | | ->01.77% (2,883,584B) 0x81DAAB1: GRAPHICS_TFPIMAGEBITMAP_$__READSTREAM$TMEMORYSTREAM$LONGINT (./include/fpimagebitmap.inc:146)
| | | ->01.77% (2,883,584B) 0x81C9A5C: GRAPHICS_TRASTERIMAGE_$__LOADFROMSTREAM$TSTREAM$LONGWORD (./include/rasterimage.inc:447)
| | | ->01.77% (2,883,584B) 0x81C9971: GRAPHICS_TRASTERIMAGE_$__LOADFROMSTREAM$TSTREAM
(./include/rasterimage.inc:423)
| | | ->01.77% (2,883,584B) 0x81C40D7: GRAPHICS_CREATEBITMAPFROMLAZARUSRESOURCE$TLAZARUSRESOURCESTREAM$TCUSTOMBITMA PCLASS$$TCUSTOMBITMAP (graphics.pp:2125)
| | | ->01.77% (2,883,584B) 0x81C41B6: GRAPHICS_CREATEBITMAPFROMLAZARUSRESOURCE$ANSISTRING$TCUSTOMBITMAPCLASS$$TCUS TOMBITMAP (graphics.pp:2144)
| | | ->01.77% (2,883,584B) 0x81C4164: GRAPHICS_CREATEBITMAPFROMLAZARUSRESOURCE$ANSISTRING$$TCUSTOMBITMAP (graphics.pp:2135)
| | | ->01.61% (2,621,440B) 0x84BE0E0: SOURCEEDITOR_TSOURCEEDITOR_$__CREATEEDITOR$TCOMPONENT$TWINCONTROL (sourceeditor.pp:4036)
| | | | ->01.61% (2,621,440B) 0x84B8F57: SOURCEEDITOR_TSOURCEEDITOR_$__CREATE$TCOMPONENT$TWINCONTROL$TSOURCEEDITOR$$T SOURCEEDITOR (sourceeditor.pp:2435)
| | | | ->01.61% (2,621,440B) 0x84C3F4A: SOURCEEDITOR_TSOURCENOTEBOOK_$__NEWSE$LONGINT$LONGINT$TSOURCEEDITOR$ANSISTRI NG$$TSOURCEEDITOR (sourceeditor.pp:5801)
| | | | ->01.61% (2,621,440B) 0x84C66AA: SOURCEEDITOR_TSOURCENOTEBOOK_$__NEWFILE$ANSISTRING$TCODEBUFFER$BOOLEAN$TSOUR CEEDITOR$$TSOURCEEDITOR (sourceeditor.pp:6552)
| | | | ->01.61% (2,621,440B) 0x81454A1: MAIN_TMAINIDE_$__DOOPENFILEINSOURCEEDITOR$TUNITEDITORINFO$LONGINT$LONGINT$TO PENFLAGS$$TMODALRESULT (main.pp:8385)
| | | | ->01.61% (2,621,440B) 0x814801D: MAIN_TMAINIDE_$__DOOPENEDITORFILE$crc4B94F828 (main.pp:9403) | | | | ->01.61% (2,621,440B) 0x8147565: MAIN_TMAINIDE_$__DOOPENEDITORFILE$ANSISTRING$LONGINT$LONGINT$TOPENFLAGS$$TMO DALRESULT (main.pp:9123)
| | | | ->01.61% (2,621,440B) 0x812EBE3: MAIN_TMAINIDE_$__SETUPSTARTPROJECT (main.pp:2253) | | | | ->01.61% (2,621,440B) 0x812B951: MAIN_TMAINIDE_$__STARTIDE (main.pp:1456) | | | | ->01.61% (2,621,440B) 0x80C7AFF: main (lazarus.pp:113)

Yes, but not 30M.

I do not know, if valgrind would detect such mem as freed or not.

No, it won't.

If interested, I can forward you the full massif and ms_print output.

Ludo

Ludo Brands wrote

Valgrind massif looks at the heap and can't distinguish memory released but kept by the memory manager or memory not released. For massif it is allocated by the program under test and reports where it was allocated.

So we can't be sure about whta memory is actually kept by the IDE

>> They clearly show that a lot of data is staying in memory. It'll

Sorry, but how does that go with the above statement? Well yes technically the data is still in memory. In memory that the heap manager still keeps, but that to the application has been freed.

>> One I found out is that synedit creates an image list with gutter >> symbols for every page!

...

....

Those are images created for the gutter pop up menu. And There are calls to free the image list when synedit is destroyed.
If QT keeps something in memory, then maybe it's a problem in the QT parts, or qt itself.

About it created per synedit instance. This may not make sense in the IDE, where the images are indeed equal.
But SynEdit can be used in other apps. I don't know if an interface exists, but it well could in future, that would allow individual synedits to use different images.
Sure even then it can be solved, and a singleton default image list could be created...

at the moment it isn't exposed. so if anyone supplies a well written patch...

to me it's not worth the work, as it only applies to artificially created extreme situations. In normal life it does not cause a real problem. => doesn't mean it wouldn't be done, just that it currently is very very very low in priority

I haven't counted the lines in the 450 files. but say we half my estimate 400,000 lines => I guess they will easily be in there.

Each line is a string. Even if the data of the string had a ref-count > 1 (which I am 99.999% sure it has not) alone the "array of string" (the list of pointers) would be 1.6 MB (probably much more, as it is more lines). Add to that other "per line data" that synedit must keep (I know of at least 6 other bytes per line, maybe more) already 3 MB or more.

I have not even those seen returned to the OS => and I have single stepped (debug) through the code that frees the memory. So clearly memory freed by the IDE, is hold in some pool somewhere else. You can start debugging the various memory managers for that, if you wish.

Now once it is known that this does happen, the same may happen for any other of the "unfreed" memory. It may or may not. It is totally unknown. Of all the unfreed memory, when you close the editors, or project anything from 0.1% to 99.9% could be hold by the memory manager.

Sorry wouldn't currently be able to look at it...

Hello Lazarus-List,

Sunday, August 14, 2011, 1:56:31 PM, you wrote:

M> Now once it is known that this does happen, the same may happen for any M> other of the "unfreed" memory. It may or may not. It is totally unknown. M> Of all the unfreed memory, when you close the editors, or project M> anything from 0.1% to 99.9% could be hold by the memory manager.

Why not simply compile with heaptrc and produce a "heaptrc.dumphead;" using a menu option. It would take some time, but large memory blocks would be discovered and divergences with VM should be revealed (memory not returned to the OS by the memory manager).

Martin wrote

Does valgrind know about the fpc internal mem handling

When using valgrind, cmem is used.

If you use such crude tools, better do additional research before making claims about a program. It might be the memory manager or heap fragmentation, not the program.

Marco van de Voort wrote

claims about a program. It might be the memory manager or heap fragmentation, not the program.

Then it's very strange that just Lazarus IDE is showing this behaviour on my system. Other software and tools I have written can allocated lots of memory, but when I free those objects, those "crude memory values I spoke of" drop considerably.

Even MSEide shows this. Start the MSEide, memory usage is 8MB in Gnome System Monitor. Load 400 units and memory jumps to 30MB. Switch between a few tabs, compile the project, run the project. Then do File -> Close All files. Memory usage in 'gnome-system-monitor' drops back down to 7MB. Go figure.

So again, Lazarus IDE is doing something wrong!

But then it would be re-used later on, unless there is a major bug in the memory manager. I repeat myself, but 30M is lost every unload/load cycle of the same 450 files.

Are the circumstances extreme, yes, but the results are extreme also. This is a testcase reproducing in minutes repeatable results. This discussion was started by Graeme I believe who complained about memory usage going up over the day under normal usage.

Ludo

I think the above may have been lost in the rest.

yet, while you certainly have a point. It isn't proof. it could be argued fragmentation based inability to re-use. Though I accept the lower likelihood of it.

Yet at least part of it may be.
I tried the process about 20 times (on windows). The first 4 or 5 times the increase was big, it then however decreased, and at the end there was very little further increase, when adding the files again, and removing them (memory would be higher while they where added, but for each cycle the memory would go down as well)

NOTE: I do not deny the possibility of an issue. But all the symptoms can be explained in other ways too. likelihoods may go either way. But in the end, unless someone can show actual code that does not release the data it allocated (or unnecessarily defers this), it appears that there is nothing that can be proven either way.

The fact that lazarus has a higher base-usage of memory, is not in itself wrong. It is only wrong if it can be shown that the memory is used for none necessary data.

I accept that some people like to keep hundreds of files in an "accessible" or "open" list. And for that purpose people keep this files open in tabs of the editor.
One way for an editor to allow that, is what MSEide does => Do not actually load the files.
But most editors are not designed for this, neither is Lazarus. This is not a wrong design, nor is it a fault. This is at best a feature that isn't implemented. Actually, some PARTS of it are implemented, like the deferred scanning.

No memory manager is ever safe for all memory access patterns.

Ludo Brands wrote

Clearly you haven't looked at the valgrind massif data I posted earlier. They clearly show that a lot of data is staying in memory. It'll require a lot of digging to figure out exactly where the problems are. One I found out is that synedit creates an image list with gutter symbols for every page!

The attached patch creates a shared imagelist

Please test and see what is the gain

PS: this is a quick patch just for test. more cleanup can be done

Luiz
-------------- next part --------------
A non-text attachment was scrubbed...
Name: syngutter.diff
Type: text/x-diff
Size: 8858 bytes
Desc: not available
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20110814/0b45af00/attachment.bin>

would not a leak show up in the debug GUI compilation profile when everything is closed and exited? or does this maybe go deeper and require that FPC be compiled with debugger/heaptrace support?

I'm noticing also a "slow down" in memory consumption after 5-6 unload/load cycles.

I compiled Lazarus with cmem to be sure valgrind massif recognises correctly the memory manager: results are very similar. So the probability that memory isn't returned by the program (vs memory manager) is growing.

So far I have been concentrating on the memory usage before and after closing the 450 files. I'm now concentrating more on what increases when unload/load the 450 files since that would really be the "bad" memory. Interesting finding so far:
- the gutter images don't increase, nor do the codecaches mentioned before. They stay in memory but apparently are reused.
- a big memory chunk that grows is memory allocated by pthread_create called from Qthread::start. The stack trace mentions QFileSystemWatcher::addPath, IBusPlugin. I mention QFileSystemWatcher and IBusPlugin because after a lot of testing, Lazarus failed to start with an error
QInotifyFileSystemWatcherEngine::addPaths: ,inotify_add_watch failed. I had to reboot the system. The threads created are all shortlived (I can't find a lot of threads attached to Lazarus). I read somewhere that pthread releases resources when pthread_detach or pthread_join is called, not after a pthread_exit. This could explain resources held back for threads not running anymore, and nicely released when the program finishes. This is probably Qt specific. But, the same stack trace also shows that CUSTOMFORM_SETWINDOWFOCUS causes 1/3 of the Qthread::start. I'm surprised that loading multiple files in an editor should move focus to every window. Another 1/3 is triggered by TMENU_ITEMCLICK triggering much later a QFileDialog_create. Is that how files are loaded?

I'll look further into this.

Ludo

Thanks, I'll test this later. As mentioned in my previous message, the memory used by the gutter symbols seems not to be growing in different unload/load cycles. My guess is that this will save 3M in the extreme test case we are using. The small chuncks are scattered over the heap which could make it difficult for the memory manager to release the memory to the OS. The impact could be bigger.

Ludo

Well at first thanks for looking so persistent into it, and for trying to establish actual facts rather than just jumping to conclusions.

The gutterimages should be freed. otherwise they would have to show as leak at the exit. or they are a leak in the widgetset, in which case heaptrc may not notice.
Try a small app with just an image list. add images, then destroy the image list. And see if qt leaks.

"codebuffer" if this is a name from the debug symbols, then thta is probably codetools. SynEdit uses a different name

Don't know about any of this. sounds like something is watching for file changes??? codetool does. But last time I checked, it didn't use any OS notification, but rather by just getting the file's modified timestamp itself.
So I wonder why QT foes that.
Also that may be qt specific. But the memory growth happens on other widgetsets too. And While possible, it would be strange, if it was leaking on qt, but mem management on other OS?

I also don't know where the threads are coming from. I do not know if Lazarus does create any of them intentionally (maybe something that was added lately?). I know on w32 for example that the ODS sometimes creates them (e.g for fileopen dialog)

Strange. I wouldn't expect a fileopen dialog.

Maybe if a fileopen-dialog-component is used in an app (as it is in Lazarus) then some initialization happens on application start, rather than on execute?
Try putting a fileopen dialog on the form of an empty project....

Yep, that should pretty much be the case (should...)

The problem in Lazarus (at least SynEdit, but probably others too) is a lot of memory constantly freed and released. also SynEdit allocates tenthousands of small bits (e.g a string per line).

It's a dream of mine to optimize that. eg. to have synedit allocate a bigger chunk, and have synedit use it knowlege about lifetime of data, to organize it better. But there is just to much other important work...

In a previous message I already mentioned that I wrote a small test app exactly doing that: all memory was returned to the OS. Of course the memory wouldn't be scattered as it would be in lazarus.
The fact that the memory used by gutter images doesn't grow after unload/load cycles does indicate that "somebody" holds on to it but reuses it. Remember I use cmem now, so it probably isn't the memory manager. Like I wrote earlier, for the time being I'm focussing on what memory usage grows in these cycles.

Because the massif tool takes some time to run and even more for me to interprete, I started doing the same on a different machine for gtk. Memory growth is smaller, 20M on amd64. Interesting here is that memory growth is located in 2 areas:
- pthread_create, again, invoked from gtk this time with no lazarus code in the stack trace. Stack trace starts with a g_closure_invoke whih indicates an event. Will be difficult to locate the source.
- again the TMENU_ITEMCLICK, similar location as for qt, found on the stacktrace. This time not losing memory in pthread but in codetoolmanager...

Looks like the memory loss is different for the different widgetsets.

One of the problems when interpreting massif data is that they are cumulative. In this case one file open dialog ( the one I used to open the 450 files) triggers 450 DOOPENFILEEDITOR calls. I wrongly interpreted this as many file open dialogs...

Ludo

Still curious what might hold n to it... See my other mail about the strings for the text/line data

The imagelist is a private field, and destroyed.

As for the images itself, I can't see how anything could hold on to them AD re-use them. That would mean to detect, if the same image is added again. Of course if image are loaded from resources, that may be possible, but IIRc the images in question are drawn on the canvas, by synedit => so unless something compares the image pixel by pixel....

just puzzled.

Ludo Brands wrote

So far I have been concentrating on the memory usage before and after closing the 450 files. I'm now concentrating more on what increases when unload/load the 450 files since that would really be the "bad" memory.

So I did another test.

I added at the end of SynEdit. destroy:

   GetMemoryManager(m);
   debugln(['****************** ',m.GetFPCHeapStatus().CurrHeapUsed])

That would show any memory allocated by pascal code. (but not memory that qt or w32 keep for some reason (even though the reason might be missing release calls by the pascal code...))

The initial sizes are matching the measurements taken by process-explorer (w32)

41870000 open lazarus, with about a dozen units - close one unit, to trigger the output
198591312 open 450 univint - close one unit, to trigger the output 63014880 close all 450 univint => ok so some meory was kept

193982864 open 450 univint - close one unit, to trigger the output => hey we are even lower than last tim
62963664 close all 450 univint => no extra memory kept on 2nd run

194002192 open again
63734976 close again
193996224 open
63027440 close

So that means:
- there may be memory allocated by the widgetset; some of this may not be returned (and heaptrc would not report this as leak) IF some of this isn not returned, then this might be an issue in Lazarus/LCL, BUT it could also be an issue of the widgetset itself.

- A huge amount of memory is allocated by pascal code, via the memory manager.
- Most of it is returned correctly.
- A certain amount is allocated and kept => but it does not increase

Those tests were done using heaptrc

...

- A certain amount is allocated and kept => but it does not increase

One think is that the project session does remember every unit that was onced open (it will forget it only, if not accessed for a long time (many open/close of project ).

The project session remembers status info, such as scrolled-pos and caret-pos for each unit, that was once opened.
So if you re-open a unit, you had once open before, you will have the caret, at the same pos, it used to be when you closed the unit (unless you open it via code jump, which will override the caret pos)

As I said there is an algorithm that will lead to units eventually be removed from the session, but only if they do not get accessed over many open/close cycles of the project.

I do nor know how much of the 20 MB that makes. Would NOT expect it to be much. But it serves as example, that there exists information, that can with full intention be kept.

I noticed the TCodeCache created in DoOpenEditorFile with LoadCodeBuffer aren't destroyed when closing the editorfiles. In DoCloseEditorFile the IsDeleted is set which sets FSource to ''. So that should be fine. Under gtk2 however, one has to prudent here, I traced into DoCloseEditorFile and noticed that the refcount for FSource was 2 before setting it to '' on all but the first file. This means that somebody has made a copy of the source and is hanging on to it. As mentioned before, on gtk2, DoOpenEditorFile allocated memory that continued to grow.

On windows I also tracked a particular FSource address across multiple project loads/unloads and the original source was kept in memory until lazarus closed. The debugger in lazarus didn't allow me to inspect the refcount of the string though and crashed on me. So this is no proof. I'll have to redo this test.

Ludo

Ludo Brands wrote

The debugger in lazarus didn't allow me to inspect the refcount of the string though and crashed on me.

Afaik:

(^longint(str)-2)^

on 32 bit:
make sure you use dwarf, and gdb7.2

I tried to do something similar but that is where gdb crashed on me...

In the mean time I found where the refcount increases: DoOpenUnknownFile
->CodeToolBoss.GetSourceType
->FCurCodeTool.GetSourceName
->TPascalReaderTool.BuildTree <- this where the refcount increases to 2

BuildTree is a complex piece of code and I'll need a good night of rest before attacking this. BeginParsing(Range); increments the refcount but that is as far as I got. Perhaps the author of the code could shed some light?

Ludo

That be Mattias.

Given his current silence, and that it is summer. I would say he might be enjoying some holiday...

I think I managed to trap that one.

for the FIRST codebuffer opened, I got the following inc-ref. I have not checked dec ref.

check the attached file, unrar, and import in debug history.

unfortunately clicking stack frames will not jump. due to absolute path used. ( on my todo)

requires laz trunk)

the 2nd watch shows the ref count value. not sure if that is before or after change....

-------------- next part --------------
A non-text attachment was scrubbed...
Name: fsource_ref.rar
Type: application/octet-stream
Size: 4015 bytes
Desc: not available
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20110815/899494d6/attachment.obj>

Martin wrote

unfortunately clicking stack frames will not jump. due to absolute path used. ( on my todo)

you can open the xml, and substitute the path, then it should work

That'll be after the change. When I traced upto beginparsing, refcount is 1 (loaded the source form file).

This seems to match my findings. The 4th break where the refcount is dropping to 1 is unfortunately an incomplete stack trace. Are these all breakpoints you encountered for loading a single file? Ie. This is the end of DOOPENEDITORFILE.
Since you have more success with watchpoints than I have (I just checked: my gdb version is 7.1 on windows and linux), would you mind tracing into DoOpenUnknownFile. This is used when opening a unit for the first time. That is where I got the refcount=2. DoOpenEditorFile is used for files that where already opened before.

Thanks, Ludo

Martin hat am 15. August 2011 um 17:27 geschrieben:

Correct. One for the scanner that scans the compiler directives and extracts the "cleaned source" and one for the parser that reads the pascal.? Additionally the codetools caches the file sources (i.e. the files converted to UTF-8).?
The biggest part of the files are kept in cache and closing the project does not free this cache.?
For example once the codetools have loaded and parsed the classes.pp it is kept in memory.?
This was so from the beginning (at least since 9 years).? Of course the RTL and FCL sources grew over the years and so the codetools need more memory.
?
The other big caches of the codetools are the code nodes. The memory need varies greatly depending on the code, but they typically need somewhat less than the sources.
There is a lot of other stuff, but this makes only a few percent.? ?
In my experiments memory fragmentation can tripple the amount of memory needed by the codetools.?
?
AFAIK the several hundred MB of ram needed by the IDE was no problem yet - even for student machines.
?
?
Mattias
?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20110815/cada9379/attachment.html>

so there we go. looking at the numbers I got from the memory manager:

41870000 open lazarus, with about a dozen units - close one unit, to trigger the output 198591312 open 450 univint - close one unit, to trigger the output 63014880 close all 450 univint => ok so some meory was kept

The 450 files covered 16.7 MB text.

After they were closed, all but 21.1 MB were freed. Given that it will take a bit of overhead to store the data, given that session info is stored too..... I believe the numbers match pretty well. I would say all memory is accounted for.

As for why memory returned to memory manager, does not get back to the OS (apparently not even when using cmem...) no idea. But memory is definitely returned to the mem manager.

As for the issue Ludo reported with QT, and requiring even a reset of his system. *IF* any handles or other resources of the widgetset (not the lazarus part, but the actual QT or GTK lib or w32) are leaked, then it needs to be traced. (btw there were people with issues on win98, so it could be).
But such leaks, would be necessarily be likely to be noticeable as huge memory consumers.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20110815/2d8b114a/attachment.html>

Keep in mind though: on windows the memory reported by windows grows too => yet if I ask the fpc memory manager, then there is no report of additional usage.

You may have the same on linux. Check the figures reported by fpc memory manager.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20110815/11b73d7c/attachment.html>

so there we go. looking at the numbers I got from the memory manager:

41870000 open lazarus, with about a dozen units - close one unit, to trigger the output
198591312 open 450 univint - close one unit, to trigger the output 63014880 close all 450 univint => ok so some meory was kept

The 450 files covered 16.7 MB text.

After they were closed, all but 21.1 MB were freed. Given that it will take a bit of overhead to store the data, given that session info is stored too..... I believe the numbers match pretty well.
I would say all memory is accounted for.

As for why memory returned to memory manager, does not get back to the OS (apparently not even when using cmem...) no idea. But memory is definitely returned to the mem manager.

As for the issue Ludo reported with QT, and requiring even a reset of his system. *IF* any handles or other resources of the widgetset (not the lazarus part, but the actual QT or GTK lib or w32) are leaked, then it needs to be traced. (btw there were people with issues on win98, so it could be). But such leaks, would be necessarily be likely to be noticeable as huge memory consumers.

And GTK2 growing. PThread and TCodeBuffer. Also still some work there.

Ludo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20110815/7f7812b6/attachment.html>

Yes.
The IDE uses only one codebuffer per file.
There are some exceptions, but these buffers live only for very a short time.

With the GTK2 widgedset I see an increase in memory allocated by TCodeBuffer when opening/closing/reopening the same files.

How much increase?

Mattias

[...] It's a dream of mine to optimize that. eg. to have synedit allocate a bigger chunk, and have synedit use it knowlege about lifetime of data, to organize it better. But there is just to much other important work...

Indeed.
But small chunks have a high chance of reuse. That's why most mem managers have special optimizations for them and that's why many small chunks can actually be a good thing.
I did some experiments with codetools allocating bigger chunks. In artificial tests it helped but in normal context it decreased performance. The mem fragmentation was worse than the gain of cache locality.

Mattias

Al 13/08/2011 10:35, En/na Ludo Brands ha escrit:

So, 50+% is program, resources and libraries. Codetools, which has been pointed to frequently, is peanuts. 20M in PixmapCache is enormuous. 17M unknown is a mistery.

Unfortunately I know nothing about gtk-2 or qt internals (either alone or wrt the LCL), so I cannot really help debug the issue, but I know I have memory leaks when using them depending on font usage. It's probably a different issue (lazarus itself only uses a handful of fonts) but maybe it's related on how gtk resources are used in the lcl.

http://bugs.freepascal.org/view.php?id=18727

Bye

The test I did is the following:
1 Open lazarus with --skip-last-projet
2 File Open, select * files in fpc/packages/univint/.src (450 files, 18M on disk)
3 close all editor files
4 repeat 2,3
5 repeat 2,3
Looking at a valgrind massif trace I find
-after step 2:
53M allocated through DOOPENUNKNOWNFILE (first time load) -after step 4:
53M allocated through DOOPENUNKNOWNFILE
16M allocated through DOOPENEDITORFILE (main.pp:9367 reload file) -after step 5:
53M allocated through DOOPENUNKNOWNFILE
32M allocated through DOOPENEDITORFILE (main.pp:9367 reload file)

There are intermediate samples and matching the massif samples with the steps taken isn't straightforward. The last sample taken has 53M+36M. The default timing is in instrictions, not in seconds. I'll have to retake the trace and time exactly the steps taken. I did make quit large pauses between steps but still...

Ludo

Al 14/08/2011 16:04, En/na Martin ha escrit:

I tried the process about 20 times (on windows).

IME (with user program, but that should translate to lazarus itself), there's a big difference between windows and linux/gtk-2 (and also linux/qt, but I don't use that, only to do some tests). Windows uses much less memory and memory usage doesn't grow with time, while linux/gtk-2, doing the same things, uses much more initial memory and its usage constantly grows.
Given that my programs have to run unattended 24/7, that's a big issue for me (more than the memory usage by lazarus).
Now under linux I routinely add a test to see if there's a pause in the production line and use that to restart the program :-(

Bye

I think also that it's a better idea to leave pooling to the heap manager. Allocating bigger chunks and split them is only useful if this requires no additional book keeping about used/free memery.

Recompiled lazarus gtk2 with cmem and memory growth is almost gone. After step 5 I have 3.4M allocated by DOOPENEDITORFILE (main.pp:9367 reload file). Apparently the memory manager is holding on to large blocks.

Ludo

Ok, to make my point more clear, there are at least 2 scenarios that came to mind when I wrote the above:

The first is very concrete, but actually a bigger change than just looking after memory.

1)
the pascal highlighter keeps nodes in a tree. They represent the parsing state per line. In most files their is a limited numbers of style that repeat themself. (lines refer to the node representing them). Currently no ref count is kept. So the highlighter doe not know when a node becomes unused. Nodes are never freed.
I would like to see how it works if I refcount them. However if that is the case, a certain amount of unused nodes should be kept. Some of the nodes can even be requested and released, when an already known line is accessed. So there would be a permanent alloc/dealloc of the same data. So I would be interested, if that would benefit from keeping nodes.

Sofar all this, is actually not about allocating big blocks, and dividing them, but rather about keeping a pool of normally allocated small block.
And also all of this is very low prior. I have to much other stuff, that I wanna do first.

For any one interested: opening Lazarus with about 10 units from lazarus.lpi, results in 1150 nodes, opening the 450 univint on top adds another 80 nodes.

On top of this (assuming ref count, and freeing whatever isn't kept in the pool), it would be of interest if allocating memory for those nodes in blocks of 50 or 100. but that would more be to satisfy curiosity, I don't think there will be much of a benefit. It is even likely to make things worse.
It would reduce fragmentation just a little, but since blocks could only be returned as a whole, a single node still in use, can prevent a block from being returned..

2)
Which is not yet even confirmed that any issue like this exists.

Search for memory (like temporary buffers) that is frequently allocated and deallocted, and keep a permanent buffer instead. This is done already in paint routines, where individual words are re-assembled to the line (as long as they match in style and color). Instead of using a string that grows, and re-allocs with each grow, a permanent pchar buffer is used.

So the idea is to go looking for other code, that may work on text or other data, causing frequent small memory changes, that could be kept in an existing place.

During testing Lazarus I found a big improvement with memory increasing on linux when using cmem. There are probably disadvantages using cmem but when your main concern is the increasing memory it is worth giving a try.

Ludo

Al 16/08/2011 17:27, En/na Ludo Brands ha escrit:

Yes, I always try with cmem under linux. Memory usage still grows :-(

Bye

A gain of 7M for the 450 files (still using cmem) and, more important, the load time (first time load) dropped from 17.60 secs to 13.50 secs. This was tested twice to remove the effect of file caching. Not bad for a quick patch!

Ludo

As for why memory returned to memory manager, does not get back to the OS (apparently not even when using cmem...) no idea. But memory is definitely returned to the mem manager.

As for the issue Ludo reported with QT, and requiring even a reset of his system. *IF* any handles or other resources of the widgetset (not the lazarus part, but the actual QT or GTK lib or w32) are leaked, then it needs to be traced. (btw there were people with issues on win98, so it could be). But such leaks, would be necessarily be likely to be noticeable as huge memory consumers.

Got also further on Qt: Lazarus compiled with cmem running outside valgrind doesn't grow the memory used doing multiple load/unload of the 450 files. Memory doesn't drop at all when closing files (same on gtk2 using cmem), but there is no increase. The memory increase and the related pthread memory problem seems to be linked to running inside valgrind massif with the --pages-as-heap=yes option. One of the side-effects is that all the memory that top marks as VIRT is taken into account, including shared with other processes. The hint came from a trace on the GTK2 machine where a very big chunk of pthread memory was allocated by arena.c which is a memory pool library.
The problem with QFileSystemWatcher spawning threads is apparently a known problem: https://bugreports.qt.nokia.com//browse/QTBUG-20028

Ludo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20110816/fd1112c9/attachment.html>

Well I guess I will windows it saves 3 or 4 secs too

There is another potential TextDrawer. Each synedit has one, and it does calculations on the font (so it accesses the OS).
However, it is one of the few areas of synedit that has had very little work, and is really not nice code.
Not sure if I will have time to look at it...

Luca, out of curiosity, is it related to the same font issue or something else? In other words can you point to some gtk2 control that being inserted and removed on the form leads to memory increase?

Max
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20110817/fdb2baf9/attachment.html>

Al 16/08/11 22:29, En/na Max Vlasov ha escrit:

No, it's just the font issue.

Bye

Luca,

I had a look with valgrind massif at the problem in http://bugs.freepascal.org/view.php?id=18727. The problem is indeed fonts not being released. I have a patch attached to the issue. The problem with valgrind and glib is that it generates a lot of false positives. Setting the evironment variable G_SLICE=always-malloc helps a lot. It is the equivelent for glib to using cmem in fpc.

Ludo

Al 17/08/2011 14:20, En/na Ludo Brands ha escrit:

Thank you!. I hope next time I stumble into a similar problem[*] I'll remember to use the massif tool instead of memcheck.

[*]TPicture.LoadFromFile I think also leaks memory under linux, but I have to check.

Bye

Al 17/08/11 15:05, En/na Luca Olivetti ha escrit:

[*]TPicture.LoadFromFile I think also leaks memory under linux, but I have to check.

Nope, cannot reproduce it now.

Bye