Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-pascal / 2007-02

Question on how to avoid memory trouble

16 messages · 11 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Jason P Sage
Show quoted text 7 lines
The error is not from a lack of stack space, either. The error is EOutOfMemory that indicates a less easily solved problem. Can anyone recommend a method to search a whole drive, of arbitrary size, without running out of memory. Stu Cox

This is exactly the kind of thing my string class works great with.

It's a dynamic double linked list - and the way its set up - there are three versions - the simple tracks just pointers, the next has name, value and Description ansistrings and I make pretty huge lists with it, and it has Append, insert, findfirst, findnext, etc. I have an unofficial release on www.jegas.org and you need to use the compile switch that allows += construct.

Fpc -Sc yourprogram.pas

Worth a shot.

Reply by email

Cox, Stuart TRAN:EX
to Jason P Sage

Shucks everybody...

I think that its great that the hornet's nest that I stirred up might mean that a new unit of container classes gets written. I didn't think that my seemingly simple problem would create such a cascade of interest. Lists are catalysts for invention!

My trouble remains that I seem unable to devise or use canned routines to get the names of all the files on my drive. Just writing them out to con: in a console application would be acceptable to me to get me started. TurboPower's technique in their SysTools seemed simple enough but I seem to have created a monster.

Does somebody have something in source that they'd share with this poor guy, please?

Stu Cox

Earlier message 30 lines
-----Original Message----- From: On Behalf Of Jason P Sage Sent: Sat, February 3, 2007 5:13 PM To: Subject: [fpc-pascal] Question on how to avoid memory trouble >The error is not from a lack of stack space, either. The error is >EOutOfMemory that indicates a less easily solved problem. >Can anyone recommend a method to search a whole drive, of arbitrary >size, without running out of memory. >Stu Cox This is exactly the kind of thing my string class works great with. It's a dynamic double linked list - and the way its set up - there are three versions - the simple tracks just pointers, the next has name, value and Description ansistrings and I make pretty huge lists with it, and it has Append, insert, findfirst, findnext, etc. I have an unofficial release on www.jegas.org and you need to use the compile switch that allows += construct. Fpc -Sc yourprogram.pas Worth a shot.

Reply by email

Wayne Sherman
to Cox, Stuart TRAN:EX
I think that its great that the hornet's nest that I stirred up might mean that a new unit of container classes gets written... ...My trouble remains...

Getting a better containers unit is nice, but the root of the problem seems to be the memory manager. Delphi's old memory manager had fragmentation issues also. That is why NexusMM, BucketMM and FastMM we created. FastMM is now included with Delphi as the main memory manager. Not only does it solve memory fragmentation issues, it is much faster than the old memory manager (especially in multithreaded apps).

FastMM is open source, has anyone ported it to FPC?

I seem to recall that when it began to be used in Delphi, it uncovered lots of memory bugs in the IDE and runtime. When these were fixed, Delphi became a much more stable product.

Wayne

Reply by email

Micha Nelissen
to Wayne Sherman
Wayne Sherman wrote
Getting a better containers unit is nice, but the root of the problem seems to be the memory manager. Delphi's old memory manager had

No.

fragmentation issues also. That is why NexusMM, BucketMM and FastMM we

The memory manager cannot fix fragmentation in TStringList. Delphi's memory manager fragmented memory with (repeated) different memory size usages.

Micha

Reply by email

Peter Vreman
to Wayne Sherman: Getting a better containers unit is nice, but the root of the problem seems to be the…
you wrote 13 lines
>I think that its great that the hornet's nest that I stirred up might >mean that a new unit of container classes gets written... >...My trouble remains... Getting a better containers unit is nice, but the root of the problem seems to be the memory manager. Delphi's old memory manager had fragmentation issues also. That is why NexusMM, BucketMM and FastMM we created. FastMM is now included with Delphi as the main memory manager. Not only does it solve memory fragmentation issues, it is much faster than the old memory manager (especially in multithreaded apps). FastMM is open source, has anyone ported it to FPC?

There are users that have got FastMM working with FPC. But it is not needed. The standard FPC heap manager is for single-threaded applications as fast as FastMM.

Peter

Reply by email

Helmut Hartl
Show quoted text 5 lines
>FastMM is open source, has anyone ported it to FPC? There are users that have got FastMM working with FPC. But it is not needed. The standard FPC heap manager is for single-threaded applications as fast as FastMM.

A little bit shortsighted (IMHO),as "everyone seriously concerning performance"
is concentrating on multithreaded, multiprocessor applications, wich are lock and waitfree, right now.

Is there anyone around who is using FastMM in FPC? A guide what is necessary to get FastMM working with FPC is much appreciated.

Thanks, helmut

Reply by email

Florian Klaempfl
to Helmut Hartl
Show quoted text 9 lines
> >FastMM is open source, has anyone ported it to FPC? > > There are users that have got FastMM working with FPC. But > it is not needed. The standard FPC heap manager is for > single-threaded applications as fast as FastMM. A little bit shortsighted (IMHO),as "everyone seriously concerning performance"

Well, depending on FastMM is short sight too because FastMM is using heavily assembler, so it is i386 only.

I think a better approach is to extend the current fpc memory manager to perform better in multithreaded environments.

is concentrating on multithreaded, multiprocessor applications, wich are lock and waitfree, right now.

Well, the lockless multithreaded memory managers are a myth. Atomic exchange operations take a also a lot of time and are locked by the cpu as well.

Is there anyone around who is using FastMM in FPC? A guide what is necessary to get FastMM working with FPC is much appreciated.

Reply by email

George Lober
to Florian Klaempfl
Florian Klaempfl wrote 38 lines
Helmut Hartl schrieb: > > >FastMM is open source, has anyone ported it to FPC? > > > > There are users that have got FastMM working with FPC. But > > it is not needed. The standard FPC heap manager is for > > single-threaded applications as fast as FastMM. > > A little bit shortsighted (IMHO),as "everyone seriously concerning > performance" > Well, depending on FastMM is short sight too because FastMM is using heavily assembler, so it is i386 only. I think a better approach is to extend the current fpc memory manager to perform better in multithreaded environments. > is concentrating on multithreaded, multiprocessor applications, wich are > lock and waitfree, right now. > Well, the lockless multithreaded memory managers are a myth. Atomic exchange operations take a also a lot of time and are locked by the cpu as well. > Is there anyone around who is using FastMM in FPC? > A guide what is necessary to get FastMM working with FPC > is much appreciated. > _______________________________________________ fpc-pascal maillist - http://lists.freepascal.org/mailman/listinfo/fpc-pascal

One nice side benefit of FastMM is that it automatically lets you know of memory leaks with no fuss and bother on the programmers part.

George

Reply by email

Jonas Maebe
to George Lober
George Lober wrote
One nice side benefit of FastMM is that it automatically lets you know of memory leaks with no fuss and bother on the programmers part.

FPC has support for that built in. Just compile your application with -ghl (that also does a lot of other memory sanity checking)

Jonas

Reply by email

Helmut Hartl
Show quoted text 5 lines
Florian Klaempfl schrieb: Well, the lockless multithreaded memory managers are a myth. Atomic exchange operations take a also a lot of time and are locked by the cpu as well.

The XEN Developers at Cambridge have nice papers to read ...

http://www.cl.cam.ac.uk/~kaf24/

(Others are researching this topic too, General Infos at: ACM / IBM / Linux Kernel)

http://www.research.ibm.com/people/m/michael/pldi-2004.pdf

I agree that as always in life it depends on the situation on where to use lock free techniques, and where not.
Memory Management is a field which can take advantages of such techniques in MT Apps.

helmut

Reply by email

George Lober
to Jonas Maebe: FPC has support for that built in. Just compile your application with -ghl (that also…
Jonas Maebe wrote 9 lines
On 12 Feb 2007, at 20:27, George Lober wrote: > One nice side benefit of FastMM is that it automatically lets you > know of memory leaks with no fuss and bother on the programmers part. FPC has support for that built in. Just compile your application with -ghl (that also does a lot of other memory sanity checking)

Good to know, thanks for the tip

George

Reply by email

Florian Klaempfl
to Vincent Snijders: This one is not lock free, because it uses atomic instructions used by the cpu, which are…
Show quoted text 7 lines
Helmut Hartl schreef: > http://www.research.ibm.com/people/m/michael/pldi-2004.pdf > This one is not lock free, because it uses atomic instructions used by the cpu, which are essentially fine grained locks.

Exactly, and cmpxchg etc. are really expensive too. Since the original FPC heap uses already several pools, it should be easily possible to make it scale better in multi threaded applications.

Reply by email

Helmut Hartl
Show quoted text 9 lines
> > This one is not lock free, because it uses atomic instructions used by > the cpu, which are essentially fine grained locks. Exactly, and cmpxchg etc. are really expensive too. Since the original FPC heap uses already several pools, it should be easily possible to make it scale better in multi threaded applications.

Fine that sounds really nice - Do you already have a idea what is not optimal and can be enhanced ?
Maybe I will find the time to give it a try

helmut

Reply by email

Jonas Maebe
to Helmut Hartl
Helmut Hartl wrote 7 lines
> Exactly, and cmpxchg etc. are really expensive too. Since > the original FPC heap uses already several pools, it should > be easily possible to make it scale better in multi threaded > applications. Fine that sounds really nice - Do you already have a idea what is not optimal and can be enhanced ?

The default memory manager is always entirely locked, while differently sized blocks are allocated from different pools. So it should be possible to lock only one pool per allocation. Additionally, getting a block from a pool may even be possibly using a few atomic operations rather than by acquiring a lock.

See rtl/inc/heap.inc

Jonas

Reply by email