Hello.

In a audio library, it uses a buffer to record audio.

There is a main loop that store the audio-data given by the audio-input device into the buffer.
For this TFileStream.WriteBuffer() is used.

Ok, all works well.

But I noted that the ram used increase at each loop and when the max ram is reached, the program crash.

How must I deal with Data.WriteBuffer(), must I write it to a temp file ?

I was thinking that the memory manager will do the job when TFileStream.WriteBuffer() is called.

But it seems not.

All advice are welcome (with code even better ;-)).

Many thanks.
Fred

-----
Many thanks ;-)

WriteBuffer will write the data to file. No extra buffer is allocated. The memory manager does not come into play.

What happens with the buffer in which you had data, this we do not know.

Michael.

Hello.

Thanks Michael for answer.

What happens with the buffer in which you had data, this we do not know.

Here is the schema of recording:

1) At creation:
DataRec := TMemoryStream.Create;

2 ) In loop:
...
DataRec.WriteBuffer(InputBuffer[0], Length(InputBuffer)); // RAM used increase at each loop
...

3) At end of recording:

function WriteWave(FileName: ansistring; aDataRec: TMemoryStream): word;
var
...
  f: TFileStream;
 ...
begin
    f := TFileStream.Create(FileName, fmCreate);
    ...
    f.WriteBuffer(ID, 4);  // add some stuff for wav header.
    ...
    f.CopyFrom(aDataRec.Data, aDataRec.Data.Size);
   ...
   f.free;

________________________________________

... buffer in which you had data

The buffer (InputBuffer) used to get the samples from input device is re-filed at each loop.
The data of (InputBuffer) are added to the other buffer (DataRec) used for WriteWave() (procedure at end of recording) in each loop.

________________________________________

Huh, maybe I did not understand your question.

Thanks.

Fre;D

-----
Many thanks ;-)

The RAM increase is normal...

And where do you free DataRec ?

The increase in RAM is normal, since you store the input in memory before writing it to disc.

Make sure you free the datarec after writing to file.

Michael.

Hello Michael.

The RAM increase is normal...

OK, I was thinking that too but... is it normal that the app crash if there are no more ram available ?
If so, how to know how many ram is disponible (and then create a temporally file if too few).

One minute of recording stereo 16 bit is +- 10 megas.

And where do you free DataRec ?

When the recording stops (at end of WriteWave() procedure).

Thanks for helping.

Fre;D

-----
Many thanks ;-)

fredvs wrote

OK, I was thinking that too but... is it normal that the app crash if there are no more ram available ?

The problem is most likely that you are running out of contiguous virtual address space, rather than out of (virtual) memory altogether.

And yes, it is normal that your program terminates with a run time error (or exception) if it cannot allocate further memory. While there is an option to instead get nil back from getmem
(http://www.freepascal.org/docs-html/current/rtl/system/returnnilifgrowheapfails.html ), almost no code has been written with this possibility in mind and hence will probably crash if you set this variable to true.

If so, how to know how many ram is disponible (and then create a temporally file if too few).

Apart from the fact that this is probably not the problem that you are facing: such an approach simply cannot work on operating systems that run more than one application at a time. Right after you checked whether there is still memory available, another application could allocate all that memory.

The proper way to handle this is to make the buffer size configurable, and to choose a sensible default (a few hundreds of megabytes on a 32 bit system, possibly more on a 64 bit system). It is not just about your program crashing or not: if you use more memory than is physically available, the system will start swapping and that will kill all usability. Since you cannot predict what the user will do while running your program, you have to make this configurable.

Jonas

There is no reliable way, the OS these days allocates as needed and starts swapping.

OK.

Thanks for helping.

I would write as soon as some hardcoded size limit is reached.

As an alternative, you could try to work with threads: one thread reads the input and stores it in blocks of memory, and a second thread writes the blocks as soon as it detects a new block.

But that is a more difficult architecture...

Michael.

Hello,

You working schema for recording is memory related, you must transform it in a static approach based in expected requirements. In other words, you are about to write a WAVE, in a regular PC you can grab it directly to the TFileStream from InputBuffer, if you are in a very low power device you can use a doble buffer and if your requirements are a low power PC with very high bitrate spike you must swap to threading model (but it is overkill for almost anything nowadays).

It you don't want to directly write to TFileStream use the double buffer, let the audio grabber write in a block of memory while you dump the other one to TFileStream. It is quite difficult to show an example because I do not known the API you are using to grab the audio, most of them work with callbacks, some blocks execution, and others uses Windows events. If execution in API is blocking your thread I'm quite sure there is another one that do not block it.

fredvs wrote

One minute of recording stereo 16 bit is +- 10 megas.

at what sampling rate? that makes a big difference, too... if you record at 44hz and you drop to 22hz, the data size will be half...

@ Jonas and Michael:

OK, to resume: it is not so simple... ;-)
Before i find a good solution, I will block the recording to maximum one hour.
If the user want to record more, he should do other recording each hour.

@ Jose : uos uses PortAudio library to grap input from devices. I will deeply study your propositions.

@ Wkitty: Thanks for the tip.

@ All : Many thanks.

Fre;D

-----
Many thanks ;-)

fredvs wrote

@ Jose : uos uses PortAudio library to grap input from devices. I will deeply study your propositions.

Hello,

After a fast view of PortAudio API you can produce a code like:

procedure WriteStreamToFile;
var
   F: TFileStream;
   lAvailableFrames: integer;
   lAvailableBytes: integer;
   lLocalBuffer: array [0..16383] of Byte;
begin
   F:=TFileStream.Create('YourFile.Extension',fmCreate);
   F.write(SomeHeaderData,SizeOf(SomeHeaderData));
   while NotUserBreak_OrLimitReached do
   begin
     lAvailableFrames:=Pa_GetStreamReadAvailable(YourAudioStream);
     lAvailableBytes:=lAvailableFrames * (YourStream_BYTES_PER_FRAME);
     if lAvailableBytes<16384 then
     begin
       // Don't read, just wait a bit.
       Sleep(1);
     end
     else
     begin
       Pa_ReadStream(
         YourAudioStream,
         @lLocalBuffer[0],
         16384 div (YourStream_BYTES_PER_FRAME
       );
       F.WriteBuffer(lLocalBuffer[0],16384);
     end;
   end;
   F.WriteBuffer((SomeTailData,Sizeof(SomeTailData));
   F.Position:=XXXX;
   F.WriteBuffer(SomeDataWithStreamSize,Sizeof(SomeDataWithStreamSize));
   F.Free;
end;

This code should work because PortAudio (and almost 100% APIs of this kind) has an internal buffer which is large enough to hold new data while you are writing to disk and performing other tasks. Some even use a dynamic buffer which can hold audio for several seconds before you extract them using the API.

Note: Of course the maths are the expected ones, you must know the BYTES_PER_FRAME size and the format of that frames, maybe int16, maybe float, maybe ....

@ Jose : Wow many thanks.

I will test and commit your code (with your name as contributor, of course).

https://github.com/fredvs/uos

Fre;D.

-----
Many thanks ;-)

Hello José.

I have study your code.
It is nearly the one used by uos.

But I do not see how it can resolve the too less RAM available.

IMO, the problem does not come from PortAudio but from TFileStream.WriteBuffer that does not have enough RAM for writing.

But maybe I do not catch something :-(

Thanks

Fre;D

-----
Many thanks ;-)

fredvs wrote

IMO, the problem does not come from PortAudio but from TFileStream.WriteBuffer that does not have enough RAM for writing. But maybe I do not catch something :-(

Hello,

In your code you were writing from uos to TMemoryStream (RAM) and when finished copy everything to TFileStream (Disk), so instead the TMemoryStream use a TFileStream and skip the last copy. Of course you may need to rewrite file header when the process is finished.

Currently (pseudo code):

TMemoryStream.Create;
while Recording do begin
   TMemoryStream.Write(AudioBuffer);
end;
TFileStream.Create;
TFileStream.CopyFrom(TMemoryStream);
TFileStream.Free;
TMemoryStream.Free;

So change it by:

TFileStream.Create;
while Recording do begin
   TFileStream.Write(AudioBuffer);
end;
// Here you may need to rewrite file header (if any).
TFileStream.Free;

Hello Jose.

Many thanks for your help.

I will deeply study your code tonight.

Write you later.

Fre;D

-----
Many thanks ;-)

Hello Jose.

Huh, of course, your code is more logical.

OK, I understand what you mean.

I will commit your idea asap.

Many thanks.

Fre;D

-----
Many thanks ;-)

Hello.

Ok, your idea is committed.

https://github.com/fredvs/uos commit da316a2..5f11b62

You are added in the list of contributors.

Many thanks.

Fre;D

-----
Many thanks ;-)