I often need to use a function which reads a file into a string, as is so common in so many scripting languages. Can it be considered to add something like this to the RTL? Since we have a refcounted Ansistring type it's natural for this to be a one-liner. Not saying we should use TStringList but that's the closest thing I found in the RTL.

function ReadFile(path: Ansistring): Ansistring;
var
  list: TStringList;
begin
  list := TStringList.Create;
  list.LoadFromFile(ExpandFileName(path));
  result := list.Text;
  list.Free;
end;

Regards,
Ryan Joseph

I am inclined to agree, but there are some complications.

This will not work without some form of passing on the expected encoding. You will also need overloads for UnicodeString. Some people will want a PChar or a buffer or a TBytes. Then you must allow the filename to be specified as a single-byte string or 2-byte string.

So I think you're looking at 6 or even 8 versions of this hypothetical function...

Michael.

On Oct 5, 2020, at 3:23 PM, Michael Van Canneyt via fpc-pascal wrote: So I think you're looking at 6 or even 8 versions of this hypothetical function...

Ouch. :)

I'm sure this code already exists in the RTL though, right? I assume it's part of some classes like TStringList and we can just pull it up into a collection of functions which are easily searchable in the RTL docs.

If not what is lowest level cross-platform layer which already exists and can we build up from? Maybe the "TextFile" type helps us? (I know it's ancient and I've never used it before honestly).

Regards,
Ryan Joseph

Hi!

Great idea!

Longueur  should be Int64

Winni

_______________________________________________ fpc-pascal maillist - https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

This looks good to me what about the concerns raised by Michael? I don't know enough about text formats but using AnsiString always just seems to work for me (I assume the compiler did something magic behind the scenes).

Regards,
Ryan Joseph

Using TStringList would be a mistake, since it will parse your file while there is no need.

No, I think that if you want this, it must be built from the basic calls.

If not what is lowest level cross-platform layer which already exists and can we build up from? Maybe the "TextFile" type helps us? (I know it's ancient and I've never used it before honestly).

I don't think that would be a good strategy.

A simple filecreate, allocate buffer, fileread, fileclose will probably be easiest.

Michael.

El 6/10/20 a les 9:01, Michael Van Canneyt via fpc-pascal ha escrit:

A simple filecreate, allocate buffer, fileread, fileclose will probably be easiest.

Lazarus has a ReadFileToString in fileutil.

Bye

I use this function to read a file into a string:

function ReadFile(const FileName: TFileName): String;
var
  InputFile: THandle;
  FileSize, BytesRead: Integer;
  Buffer: String='';
begin
  try
    InputFile := FileOpen(FileName, fmOpenRead);
    if InputFile = -1 then
      begin
        WriteLn(Format('Error opening file "%s".', [FileName]));
        Halt; // Exit?
      end;
    FileSize := FileSeek(InputFile, 0, fsFromEnd);
    SetLength(Buffer, FileSize);
    FileSeek(InputFile, 0, fsFromBeginning);
    BytesRead := FileRead(InputFile, Buffer[1], FileSize);
    if BytesRead < FileSize then
      begin
        WriteLn(Format('Error reading file "%s".', [FileName]));
        Halt; // Exit?
      end;
    Result := Buffer;
  finally
    FileClose(InputFile);
  end;
end;

On 6 Oct 2020, at 09:25, Luca Olivetti via fpc-pascal wrote:

El 6/10/20 a les 9:01, Michael Van Canneyt via fpc-pascal ha escrit:

A simple filecreate, allocate buffer, fileread, fileclose will probably be easiest.

Lazarus has a ReadFileToString in fileutil.

Bye

Op 2020-10-05 om 20:45 schreef Ryan Joseph via fpc-pascal:

Stringlist static method loadstringfromfile ?

No, we don't deal in magic, only bits and bytes :-)

I added the following functions to the sysutils unit (rev 47056):

// Read raw content as bytes
Function GetFileContents(Const aFileName : RawByteString) : TBytes;
Function GetFileContents(Const aFileName : UnicodeString) : TBytes;
Function GetFileContents(Const aHandle : THandle) : TBytes;
// Read content as string
// Assume TEncoding.SystemEncoding
Function GetFileAsString(Const aFileName : RawByteString) : RawByteString;
// Specify encoding
Function GetFileAsString(Const aFileName : RawByteString; aEncoding : TEncoding) : RawByteString;
// Assume TEncoding.Unicode contents
Function GetFileAsString(Const aFileName : UnicodeString) : UnicodeString;
// Specify encoding, return Unicode string.
Function GetFileAsString(Const aFileName : UnicodeString; aEncoding : TEncoding) : UnicodeString;

These functions will raise an exception if the file cannot be opened or read. They cannot be used on handles that do not support FileSeek() (sockets, pipes, stdin/stdout etc.).

I did some tests on encoding conversion but not extensively. If you find any errors, please report them through the bugtracker.

Michael.

On Oct 6, 2020, at 2:12 AM, Michael Van Canneyt via fpc-pascal wrote: I added the following functions to the sysutils unit (rev 47056):

Great, thanks Michael.

I've always used AnsiString so why is UnicodeString preferable here?

So is the idea we need to specify an UTF-8 encoding for unicode otherwise it assumes ASCII format? I didn't specify the encoding with TStringList and I always seemed to get back what I wanted (maybe it sniffed the encoding from the file?).

Regards,
Ryan Joseph

Not 100% on this but I think the gist is that UnicodeString is compatible with AnsiString and a conversion is done on assignment.

Wouldn't it be better to implement it directly as *IOUtils.TFile* (like Deplhi)?

*class function ReadAllBytes(const Path: string): TBytes; static;class function ReadAllLines(const Path: string): TStringDynArray;overload; static;class function ReadAllLines(const Path: string; const Encoding: TEncoding): TStringDynArray; overload; static;class function ReadAllText(const Path: string): string; overload; inline; static;class function ReadAllText(const Path: string; const Encoding: TEncoding): string; overload; inline; static;*
Souce:
http://docwiki.embarcadero.com/Libraries/Sydney/en/System.IOUtils.TFile_Methods Then there would be only one way in the RTL for this task and not two once *IOUtils* (foundation webpage says it's planned anyway) is implemented.

Am Di., 6. Okt. 2020 um 10:12 Uhr schrieb Michael Van Canneyt via fpc-pascal :

Well, the answer is twofold:

1. IOUtils does not exist yet. Now the OP has a solution.

2. I don't use IOUtils and don't plan to.
Its .NET-like approach has no added value for me.
Just like I don't use rtii unit, but always use the more low-level typinfo unit.

But once TFile will be implemented, it can obviously reuse the functions from Sysutils. (like it presumably will for 99% of all other functions in it). If need be the API can be extended in SysUtils to match what IOUtils needs..

Michael.

Now I want to get the contents of a file, which contains either UTF8 or some single-byte encoding, and the filename is NOT in my locale (e.g. it may be chinese, which I cannot represent in RawByteString), but when I specify the filename as UnicodeString I am forced to receive the contents Unicode as well.

From my perspective that sucks.

If the original file contains UTF16, it will save you a conversion.

I usually do not use UTF16, you clearly also not, but other people do, and we try to cater for them as well :-)

So is the idea we need to specify an UTF-8 encoding for unicode otherwise it assumes ASCII format?

It will assume the system unit default encoding. What that is depends on your OS. Mac, Linux: UTF8, windows: ANSI, unless you use Lazarus in which case it will also be UTF8.

(in fact you can set it to any codepage you want, but the above are the defaults)

I didn't specify the encoding with TStringList and I always seemed to get back what I wanted (maybe it sniffed the encoding from the file?).

Depending on your environment TStringList will use ANSI or UTF8, just as above.

Michael.

Of course you can. That is why we use RawByteString for file opening functions. You specify it as UTF8 in that case, the RTL will do the rest.

The whole of lazarus works on this principle.

Michael.

Since we're on the topic how about another one-liner for reading all the files in directory into a dynamic array? This has the added benefit of getting enumeration for free. This is standard stuff for working with files in scripting languages so I think the FPC RTL should include something like this also.

================================

type
  TStringArray = array of string;
function FindAllFiles(path: string): TStringArray;
var
  info: TSearchRec;
begin
  if not DirectoryExists(path) then
    raise Exception.Create('Directory "'+path+'"" doesn''t exist');
  path := path+DirectorySeparator+'*';
  result := [];
  if FindFirst(path, faAnyFile, info) = 0 then
    begin
      repeat
        result += [info.name];
      until FindNext(info)  0;
      FindClose(info);
    end;
end;
begin
  // fastest way to read all text files in a directory.
  // only 2 lines doesn't leak memory
  for name in FindAllFiles('/usr/local/lib/fpc') do
    writeln(GetFileAsString('/usr/local/lib/fpc/'+name));
end.

Regards,
Ryan Joseph

On Oct 6, 2020, at 1:42 AM, Jer Haan via fpc-pascal wrote: I use this function to read a file into a string:

Not sure how Michael implemented it but we have this in the RTL now, so we can throw away our ReadFile functions. I have many of those littered around in various projects so I'm looking forward to killing it. ;)

Regards,
Ryan Joseph

I see Delphi has something similar, so I will add an implementation. But I need to study the options they provide so we can make a reasonably compatible version :)

Michael.

On Oct 7, 2020, at 2:19 AM, Michael Van Canneyt via fpc-pascal wrote: I see Delphi has something similar, so I will add an implementation. But I need to study the options they provide so we can make a reasonably compatible version :)

Excellent thanks. It looks like Lazarus has FindAllFiles but it doesn't return a dynamic array (the important part so it can be dropped into for..in loops). Many scripting languages have a similar function (I can think of C#, PHP, Python and Javascript off the top of my head).

Regards,
Ryan Joseph

Here's another interesting option I considered. You can call Scandir which returns a record with an enumerator. You can then use this to drop right into the for loop. It doesn't allocate memory and you can break the loop to stop the iteration. The benefit is you can avoid costly memory allocations for large directories but I don't think it's a replacement for a function that returns an array of TStringList.

type
	TDirectoryEnumerator = record
	  path: string;
	  count: integer;
		name: string;
		info: TSearchRec;
		function GetEnumerator: TDirectoryEnumerator;
		function MoveNext: boolean;
		property Current: string read name;
	end;
function TDirectoryEnumerator.GetEnumerator: TDirectoryEnumerator;
begin
	result := self;
end;
Function TDirectoryEnumerator.MoveNext: Boolean;
begin
  if count = 0 then
  	begin
  		if FindFirst(path, faAnyFile, info) = 0 then
  			begin
  				name := info.name;
  				inc(count);
  				result := true;
  			end
  		else
  			result := false;
  	end
  else if FindNext(info)  0 then
		begin
			FindClose(info);
			result := false;
		end
	else
		begin
			name := info.name;
			inc(count);
			result := true;
		end;
end;
function ScanDir(path: string): TDirectoryEnumerator;
begin
	if not DirectoryExists(path) then
		raise Exception.Create('Directory "'+path+'"" doesn''t exist');
	result.path := path+DirectorySeparator+'*';
	result.count := 0;
end;

Regards,
Ryan Joseph

Michael, Is it possible to extend this same set of functions to writing? Naturally I need to write back to the file now and the same problem presents itself. I have to search through other code bases to find a function or Google and find https://wiki.freepascal.org/File_Handling_In_Pascal which offers a whole host of options. SaveStringToFile from the wiki link wraps TFileStream but having a single low-level function would be best.

Regards,
Ryan Joseph

Should be easy enough.

Michael.

On Oct 8, 2020, at 3:01 PM, Michael Van Canneyt via fpc-pascal wrote: Should be easy enough.

Great, glad this is happening. Let me know if you want some help with these additions or if it's best for you to just blast through it on you own. Seems like you know what's best for the RTL though honestly.

Regards,
Ryan Joseph

Just nitpicking.
Shouldn't "try" be after the "reset" and after the "rewrite"? Why don't you allow to write an empty string?

They cannot be used on handles that do not support FileSeek() (sockets, pipes, stdin/stdout etc.).

Well, it would be better if it could

You can just incrementally resize the return array, when reading succeeds after seeking fails.

I have a string load function doing that:

https://github.com/benibela/bbutils/blob/master/bbutils.pas#L3652

Writing a file should write the data in a temporary file and then rename the temporary file to replace the target file. Otherwise it might destroy the target file, without writing the new content, when there is an error. Although not even renaming is always safe, some people say you need a filesystem-specific transaction log.

Benito

On Oct 9, 2020, at 7:02 AM, Benito van der Zander via fpc-pascal wrote: Writing a file should write the data in a temporary file and then rename the temporary file to replace the target file. Otherwise it might destroy the target file, without writing the new content, when there is an error. Although not even renaming is always safe, some people say you need a filesystem-specific transaction log.

I've seen in some API's they have an "atomic" flag for write file functions which do the swap you're describing. I agree it's a smart thing to include in the function but it's best to be an option since there is an additional operation being performed and affects performance.

Regards,
Ryan Joseph

Le 09/10/2020 à 10:15, Santiago A. via fpc-pascal a écrit :

Just nitpicking. Shouldn't "try" be after the "reset" and after the "rewrite"? Why don't you allow to write an empty string?

Yes, you're right, the Reset/Rewrite should be after the try. I admit this code is not general, just tailored to my own needs. In my use-cases, writing a file of size 0 as placeholder is not useful.