Need to read the last 4 lines (txt) of a very large log file 100M bytes, 100k lines or more lots of times - of course I can read through the file in the normal way but this is rather slow. Is there a way to 'read lines from the end'? I note that unix tail or its port to windows does this very fast so guess there must be a fast way to do it in pascal too?

TIA for any ideas John

Hello FPC-Pascal,

Monday, April 11, 2011, 6:32:20 PM, you wrote:

JL> Need to read the last 4 lines (txt) of a very large log file 100M bytes, JL> 100k lines or more lots of times - of course I can read through the file in JL> the normal way but this is rather slow. Is there a way to 'read lines from JL> the end'? I note that unix tail or its port to windows does this very fast JL> so guess there must be a fast way to do it in pascal too?

The idea is easy, instead reading forward, you read backwards in blocks of, say, 64 Kb and count lines back. If not enougth lines you insert another 64 Kb before and so on. Average line is usually less than 100 bytes, so each 64 Kb are usually more than 600 lines. Read of 64 Kb usually takes the same time as read any lower amount when the block is 4096 bytes aligned.

You can not really read backwards. You can use
something like TFileStream.Seek(-4096,soEnd).

Mattias

Hello FPC-Pascal,

MG> You can not really read backwards. You can use MG> something like TFileStream.Seek(-4096,soEnd).

I think that the "read backwards in blocks" already stated that fact, but nice annotation.

So the idea is to open file eg as file of array [1:65536] of char or something similar, seek & read to end of file, then go through the last array converting to strings by finding crlfs correct? Is this the simplest way? If not any outline code?

John

José Mejuto wrote

Hello FPC-Pascal,

Monday, April 11, 2011, 7:55:01 PM, you wrote:

MG> You can not really read backwards. You can use MG> something like TFileStream.Seek(-4096,soEnd).

I think that the "read backwards in blocks" already stated that fact, but nice annotation.

Hmm, could first have the reader enumerate the number of lines in the file, then read through the files using the "total number of lines - 4" variable and then start displaying the lines after that variable has been reached. If necessary, I would be happy to see what I can come up with here and then post it to the mailing list.

Hello FPC-Pascal,

Tuesday, April 12, 2011, 1:09:34 AM, you wrote:

John Lee wrote

> Need to read the last 4 lines (txt) of a very large log file 100M > bytes, 100k lines or more lots of times - of course I can read

RW> Hmm, could first have the reader enumerate the number of lines in the RW> file, then read through the files using the "total number of lines - 4" RW> variable and then start displaying the lines after that variable has RW> been reached. If necessary, I would be happy to see what I can come up RW> with here and then post it to the mailing list.

Go get the amount of lines of a file you must read the whole file as each line does not have a fixed size. To put an non programming example you think in an elevator and I tell you that the elevator can lift up 600 kilograms. How many people are inside ? You can routhly calculate that 8 persons as in average one persons is 75 kilograms in weigth, but what happends if the elevator is filled with childrem ( sometimes called people too :-) ) between 5 and 6 years old ?

Hello FPC-Pascal,

Tuesday, April 12, 2011, 12:51:17 AM, you wrote:

jgc> So the idea is to open file eg as file of array [1:65536] of char or jgc> something similar, seek & read to end of file, then go through the last jgc> array converting to strings by finding crlfs correct? Is this the simplest jgc> way? If not any outline code?

There is the right way which is a bit complex as there are 3 different line ending sequences based in the platform. The code get simpler if you support only one end of line, but as in your case you only need 4 lines and this lines for sure are not too large there is a more simple way, not very elegant but simple:

function GetLast4Lines(const aFileName: ansistring): ansistring;
var
  Lines: TStringList;
  FileToRead: TFileStream;
  Buffer: ansistring;
  ToReadBytes: int64;
  i: integer;
begin
  //Open the file with fmShareDenyNone, so other process can write in
  //the file while I'm reading
  FileToRead := TFileStream.Create(aFileName,fmOpenRead or fmShareDenyNone);
  //Can I read at least 4096 bytes ?
  if FileToRead.Size > 4096 then ToReadBytes := 4096 else ToReadBytes := FileToRead.Size;
  //Locate read position at 4096 bytes from the end
  FileToRead.Position := FileToRead.Size - ToReadBytes;
  //Setup buffer to space needed
  SetLength (Buffer, ToReadBytes);
  //Read the last 4096 or less bytes in Buffer
  FileToRead.Read (Buffer[1], ToReadBytes);
  //Release the file, not needed anymore.
  FileToRead.Free;
  //Create a lines array
  Lines := TStringList.Create;
  //Assign the buffer to the lines array
  Lines.Text := Buffer;
  //Initialize output to nothing
  Result:='';
  //Read from lastline-4 to lastline
  for i := Lines.Count-1-4 to Lines.Count-1 do
  begin
    //Add each line to the output
    Result := Result + Lines[i] + LineEnding;
  end;
  //Release lines array
  Lines.Free;
  //Result now have the last 4 files.
end;

Hello John,

Thursday, April 14, 2011, 3:26:48 PM, you wrote:

JL> Hi there, thanks for this - looks a very complete function. I hadn't JL> realised that best way would be to use streaming! I'm trying to write a 3 JL> line test program that just gets last 4 lines of a file.

JL> I'm not very familiar with obj pascal or streaming. What vars & uses JL> clauses do I need to call your function?

JL> this is what I have - I still gets some error messages, missing tfilestream, JL> tfilelist.

Just only "Classes". You can do the same using regular pascal files but to me TStream is more clear.

Hi Jose, all.
I've now (at last!) started using your last4 lines program. Thanks a lot- it works really well and even on a 100M log file it is really really fast - faster than 1 msec, which I didn't believe until I timed it using gettickcount. Also it doesn't generate much disk i/o. Reading the whole log file would take longer and generate disk i/o - I'm guessing that it doesn't read the file ... so not sure how it works...?

I've had to make few changes eg increasing the buffer size, because the lines in my log file are of varying length ans some are rather long eg > 1000 bytes.

Two issues I've encountered, I'm getting an exception abort (efopenerror) if no file exists and also I get an exception abort if looking for the last 4 lines and it overflows the buffer. Can you pls suggest some code to stop these, and eg return a 0 length string if either happens?

Thanks again for this help.

John