Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-pascal / 2017-02

FreePascal Windows - Force files to write to disk

42 messages · 16 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

James Richters

I have a freepascal Windows console application that I recently ported over from DOS Turbo Pascal. I am have a small settings file that I write to disk that keeps getting corrupted. It happens only occasionally, but it’s always this one file, no others. Do I need to do something specific to force windows to write the file to disk and not have it in some kind of cache or buffer? I am using the following code to write the file:

   Assign(BitFile,'BitSave.pax');
   ReWrite(BitFile);
   WriteLn(BitFile,XADJ:1:8);
   WriteLn(BitFile,YADJ:1:8);
   WriteLn(BitFile,ZADJ:1:8);
   WriteLn(BitFile,WADJ:1:8);
   WriteLn(BitFile,AADJ:1:8);
   WriteLn(BitFile,TADJ:1:8);
   WriteLn(BitFile,VADJ:1:8);
   WriteLn(BitFile,UADJ:1:8);
   WriteLn(BitFile,CurrentTool);
   WriteLn(Bitfile,P_Value[4]);
   Close(BitFile);

The above only happens when I make a change to one of the settings, and the system is never having any issue at that time, yet upon occasion I will have an error opening the file and when I inspect it, it’s just a long string of [nul]. I am not leaving the file open, I’m assigning it, writing to it, then closing it, so there should be no opportunity for data to be lost. I am having the same issue on about 8 different computers, most of which are running solid state hard drives. It’s always just this one file that is affected, nothing else. No other process ever uses this file, so it must be my freepascal program that is leaving the file in a vulnerable state.

My understanding was the Close(file); would save the file all the way to disk and flush any buffers that were holding it, but I’m wondering if that was only true for DOS and there is some other windows cache or buffer that also needs to be instructed to flush to truly save the file all the way to disk and close it.

Thanks for any advice on this

James

Tomas Hajny
to James Richters

On Tue, February 21, 2017 22:12, James Richters wrote:

Hello James,

Show quoted text 32 lines
I have a freepascal Windows console application that I recently ported over from DOS Turbo Pascal. I am have a small settings file that I write to disk that keeps getting corrupted. It happens only occasionally, but it’s always this one file, no others. Do I need to do something specific to force windows to write the file to disk and not have it in some kind of cache or buffer? I am using the following code to write the file: Assign(BitFile,'BitSave.pax'); ReWrite(BitFile); WriteLn(BitFile,XADJ:1:8); WriteLn(BitFile,YADJ:1:8); WriteLn(BitFile,ZADJ:1:8); WriteLn(BitFile,WADJ:1:8); WriteLn(BitFile,AADJ:1:8); WriteLn(BitFile,TADJ:1:8); WriteLn(BitFile,VADJ:1:8); WriteLn(BitFile,UADJ:1:8); WriteLn(BitFile,CurrentTool); WriteLn(Bitfile,P_Value[4]); Close(BitFile); The above only happens when I make a change to one of the settings, and the system is never having any issue at that time, yet upon occasion I will have an error opening the file and when I inspect it, it’s just a long string of [nul]. I am not leaving the file open, I’m assigning it, writing to it, then closing it, so there should be no opportunity for data to be lost. I am having the same issue on about 8 different computers, most of which are running solid state hard drives. It’s always just this one file that is affected, nothing else. No other process ever uses this file, so it must be my freepascal program that is leaving the file in a vulnerable state.

It's difficult to find out possible reasons without fully compilable source. If I should guess, I would check the types of the variables used for writing and also make sure that the content of those variables isn't overwritten in memory (i.e. even before they're written to the file). In particular, if your variables are strings and you compiled the source (originally written for Turbo Pascal, i.e. using shortstrings) e.g. with {$H+}, or in a compiler mode using ansistrings or
widestrings/unicodestrings by default _and_ the original source modified the strings in a non-standard way, you might overwrite the content of those variables.

BTW, what exactly is the error you get when opening the file?

Show quoted text 5 lines
My understanding was the Close(file); would save the file all the way to disk and flush any buffers that were holding it, but I’m wondering if that was only true for DOS and there is some other windows cache or buffer that also needs to be instructed to flush to truly save the file all the way to disk and close it.

Your understanding is correct, nothing else should be necessary.

Tomas

Jonas Maebe
James Richters wrote 7 lines
I have a freepascal Windows console application that I recently ported over from DOS Turbo Pascal. I am have a small settings file that I write to disk that keeps getting corrupted. It happens only occasionally, but it’s always this one file, no others. Do I need to do something specific to force windows to write the file to disk and not have it in some kind of cache or buffer? I am using the following code to write the file:

This reminds me of a bug report from a couple of years back, where someone else had a somewhat similar weird issue under Windows: http://bugs.freepascal.org/view.php?id=27670

Do you happen to use Flexraid as well?

Jonas

Santiago A.
to James Richters: I have a freepascal Windows console application that I recently ported over from DOS…
James Richters wrote 5 lines
Thanks for any advice on this

My first action would be to guarantee that the file is closed with a try...finally.

AssignFile(BitFile,'BitSave.pax');
ReWrite(BitFile);
try

....

finally
  CloseFile(BitFile);
end;

Second:

Do you have the file opened with an editor, or things like that, while running the program?.

Third:

Check that all variables have valid values and that values are converted to string properly to be printable.

Fourth:

Do you use somewhere {$I+} {$I-}? I also had some problems with that with old programs.

Fourth:

What's the type BitFile?
BitFile: TextFile;
BitFile: File;

Maybe using writeln with non-textfile files may cause problems.

Saludos Santiago A.
Jürgen Hestermann
to James Richters: I have a freepascal Windows console application that I recently ported over from DOS…
James Richters wrote 16 lines
I have a freepascal Windows console application that I recently ported over from DOS Turbo Pascal. I am have a small settings file that I write to disk that keeps getting corrupted. It happens only occasionally, but it’s always this one file, no others. Do I need to do something specific to force windows to write the file to disk and not have it in some kind of cache or buffer? I am using the following code to write the file: Assign(BitFile,'BitSave.pax'); ReWrite(BitFile); WriteLn(BitFile,XADJ:1:8); WriteLn(BitFile,YADJ:1:8); WriteLn(BitFile,ZADJ:1:8); WriteLn(BitFile,WADJ:1:8); WriteLn(BitFile,AADJ:1:8); WriteLn(BitFile,TADJ:1:8); WriteLn(BitFile,VADJ:1:8); WriteLn(BitFile,UADJ:1:8); WriteLn(BitFile,CurrentTool); WriteLn(Bitfile,P_Value[4]); Close(BitFile); The above only happens when I make a change to one of the settings, and the system is never having any issue at that time, yet upon occasion I will have an error opening the file and when I inspect it, it’s just a long string of [nul]. I am not leaving the file open, I’m assigning it, writing to it, then closing it, so there should be no opportunity for data to be lost. I am having the same issue on about 8 different computers, most of which are running solid state hard drives. It’s always just this one file that is affected, nothing else. No other process ever uses this file, so it must be my freepascal program that is leaving the file in a vulnerable state. My understanding was the Close(file); would save the file all the way to disk and flush any buffers that were holding it, but I’m wondering if that was only true for DOS and there is some other windows cache or buffer that also needs to be instructed to flush to truly save the file all the way to disk and close it.

How is "Bitfile" declared?
According to http://wiki.freepascal.org/Text
it should be either

var BitFile : TextFile;

or

var BitFile : Text;

Also, I am not sure whether "Close" is equivalent to "CloseFile" and "Assign" is equivalent to "AssignFile". I remenber that I had some issues using the old (TP7) notations but I am don't remember the details.

James Richters
to Santiago A.: My first action would be to guarantee that the file is closed with a try...finally.…

Thanks for the advice on things to look at.

Here is some more information. First of all, this happens extremely rarely, about once a month or two on a pc here and another there, but it’s odd that it’s always this one file, and it should not be ’m using turbo pascal compatibility mode, and it’s a console application, there is no multi threading going on, it’s straight inline code. No other programs on the system ever access this file and there is no reason to open it with a text editor or anything like that. The error I get is when trying to open the file to read it. It’s something like read past end of file, because the corrupted file is just one long line, once I read that, the second read is past the end of file. I can do checking to get the file length and avoid the error, but that doesn’t solve the real issue, which is that the data that is supposed to be in the file is just gone.

Bitfile is type TEXT like this:

Bitfile: Text;

and my variables are all doubles.

Normally what happens is, everything is fine for a month or two, but then one day someone will come in and notice the error on the screen, it was fine the night before, the program running overnight, in the morning the program is not running, and when the attempt is made to run it, The error is found and the cause of the error is BitSave.pax has been corrupted and is now a long string of [nul] the computers this runs on are all windows 10 with Samsung solid state hard drives. I suspect that the majority of the time, this has happened due to winders doing an automatic update and restarting without permission as windows 10 annoyingly likes to do, thus crashing my program in order to do this restart that is SO important. I have shut off windows update via group policy, but still had the problem since, however I cannot be certain there was not a power failure. The computers are set to re-power up after power loss in BIOS and the program is set to run on startup, in fact the computers have no other function than to run this one program, they are never used for anything else. The procedure that writes out this file is only ever used when one of the variables is changed, and they can only be changed by manual input, which is not happening anymore when the issue occurs. The thing is, even if windows forced a restart while my program was running this file should have been closed at the time, because if someone was standing there editing the variables, they would see the restart notification and close the program first, or be able to tell me they had a power failure.

As a temporary measure, I’m just writing out the file twice so at least I have a backup, but I don’t see why the backup would not be corrupted by this same issue, so that’s probably pretty useless.

Unfortunately this is difficult to troubleshoot because it happens so rarely. I have tried to force it to happen by terminating my program with task manager and cannot cause the file to be corrupted. I have not yet tried to duplicate the error by shutting down the system while the program is running. I could try that. If it does turn out to be an issue with my program running during shutdown, is there some way I can detect a pending shutdown and exit normally before the shutdown happens? It wouldn’t really solve the cause of the issue, but it would possibly help prevent some of the occurrences related to shutdowns, but would not help with power failures.

I am unable to use Try in turbo pascal compatible mode. If I use it, I get identifier not found ‘Try’ If I attempt to compile in some mode other than turbo pascal compatible, I end up with thousands of other errors, so I would pretty much have to do a complete re-write. I can’t see how the close could not be run, there is no way out of my procedure without running it since it’s a single thread console application.

James

Earlier message 58 lines
From: On Behalf Of Santiago A. Sent: Wednesday, February 22, 2017 3:48 AM To: FPC-Pascal users discussions Subject: Re: [fpc-pascal] FreePascal Windows - Force files to write to disk El 21/02/2017 a las 22:12, James Richters escribió: Thanks for any advice on this My first action would be to guarantee that the file is closed with a try...finally. AssignFile(BitFile,'BitSave.pax'); ReWrite(BitFile); try .... finally CloseFile(BitFile); end; Second: Do you have the file opened with an editor, or things like that, while running the program?. Third: Check that all variables have valid values and that values are converted to string properly to be printable. Fourth: Do you use somewhere {$I+} {$I-}? I also had some problems with that with old programs. Fourth: What's the type BitFile? BitFile: TextFile; BitFile: File; Maybe using writeln with non-textfile files may cause problems. -- Saludos Santiago A.
Sven Barth
to Jürgen Hestermann: How is "Bitfile" declared? According to http://wiki.freepascal.org/Text it should be…
"Jürgen Hestermann" wrote
Also, I am not sure whether "Close" is equivalent to "CloseFile" and

"Assign" is equivalent to "AssignFile".

I remenber that I had some issues using the old (TP7) notations but I am

don't remember the details.

They are the same. The ones without "File" are from TP times and were renamed in Delphi to avoid confusing them with the Assign and Close methods of the component hierarchy.

Regards,
Sven

Santiago A.
to James Richters: Thanks for the advice on things to look at. Here is some more information. First of all,…
James Richters wrote 7 lines
The error I get is when trying to open the file to read it. It’s something like read past end of file, because the corrupted file is just one long line, once I read that, the second read is past the end of file. I can do checking to get the file length and avoid the error, but that doesn’t solve the real issue, which is that the data that is supposed to be in the file is just gone.

Just two lines of research ;-)

A) While you are reading the file you call the procedure that writes it.

B) The is a bug, not in writing the part, but in the reading code: Somewhere when you read the file, you don't close it, so the file remains open. Usually it's not a big issue, you can open the file several times, and when you close the program everything is closed. But some times, when a parameter is changed you rewrite it while it is still open and everything gets messed. So this two events must happen, the program has executed the part of code that lets the file open and next a parameter is changed.

Show quoted text 14 lines
The thing is, even if windows forced a restart while my program was running this file should have been closed at the time, because if someone was standing there editing the variables, they would see the restart notification and close the program first, or be able to tell me they had a power failure. As a temporary measure, I’m just writing out the file twice so at least I have a backup, but I don’t see why the backup would not be corrupted by this same issue, so that’s probably pretty useless.

Yes, I think that's the way to go.

1) Before writing, read it and if it's ok make a backup of the file
2) Write the file.
3) Read again what you have written to check whether it's ok.
4) Make a second backup.

That way, when you load parameters you have two backups to recover. And maybe a clue of what's going wrong.
Beside, I would add a timestamp inside the file.

In addition, you could use flush and {$I-}

Saludos Santiago A.
James Richters
to Santiago A.

Here is the procedure that reads the file. It is only read once when the program is started, then written to when changes are made.

I don’t have a good reason for having a different file variable name in my read procedure than my write procedure, but those are local varaibles to the procedures anyway so it shouldn’t matter what I call them.

Var
   locfile : text;
If FileExists('Bitsave.pax') Then
Begin
   Writeln('BitSave.pax Found, Loading... ');
   Assign(locfile,'BitSave.pax');
   reset(locfile);
   ReadLn(locfile,XAdj);
   ReadLn(locfile,YAdj);
   ReadLn(locfile,ZAdj);
   ReadLn(locfile,WAdj);
   ReadLn(locfile,AAdj);
   ReadLn(locfile,TAdj);
   ReadLn(locfile,VAdj);
   ReadLn(locfile,UAdj);
   ReadLn(locfile,CurrentTool);
   ReadLn(locfile,P_Value[4]);
   Close(locfile);
End

I will do as you suggest and read what I just thought I wrote back to verify it. I may also incorporate some kind of checksum as well and it wouldn’t be a bad idea to check how many lines are in the file before I try to read it, just to make sure I don’t get read past end of file error. It seems like a lot just to save just 10 numbers, but it’s a pain if those numbers are gone! I will also try flush and {$i-} as well.

Thanks to everyone for your advice and help

Jim

Earlier message 49 lines
From: On Behalf Of Santiago A. Sent: Wednesday, February 22, 2017 11:07 AM To: FPC-Pascal users discussions Subject: Re: [fpc-pascal] FreePascal Windows - Force files to write to disk El 22/02/2017 a las 13:17, James Richters escribió: The error I get is when trying to open the file to read it. It’s something like read past end of file, because the corrupted file is just one long line, once I read that, the second read is past the end of file. I can do checking to get the file length and avoid the error, but that doesn’t solve the real issue, which is that the data that is supposed to be in the file is just gone. Just two lines of research ;-) A) While you are reading the file you call the procedure that writes it. B) The is a bug, not in writing the part, but in the reading code: Somewhere when you read the file, you don't close it, so the file remains open. Usually it's not a big issue, you can open the file several times, and when you close the program everything is closed. But some times, when a parameter is changed you rewrite it while it is still open and everything gets messed. So this two events must happen, the program has executed the part of code that lets the file open and next a parameter is changed. The thing is, even if windows forced a restart while my program was running this file should have been closed at the time, because if someone was standing there editing the variables, they would see the restart notification and close the program first, or be able to tell me they had a power failure. As a temporary measure, I’m just writing out the file twice so at least I have a backup, but I don’t see why the backup would not be corrupted by this same issue, so that’s probably pretty useless. Yes, I think that's the way to go. 1) Before writing, read it and if it's ok make a backup of the file 2) Write the file. 3) Read again what you have written to check whether it's ok. 4) Make a second backup. That way, when you load parameters you have two backups to recover. And maybe a clue of what's going wrong. Beside, I would add a timestamp inside the file. In addition, you could use flush and {$I-} -- Saludos Santiago A.
James Richters
to Jonas Maebe

I suppose a virus scanner could access it, I didn't think of that. I guess I should have said no other program intentionally access the file. It's just a way for me to store some variables and get them back, it serves no other purpose.

Earlier message 14 lines
-----Original Message----- From: On Behalf Of Jonas Maebe Sent: Wednesday, February 22, 2017 3:47 PM To: FPC-Pascal users discussions Subject: Re: [fpc-pascal] FreePascal Windows - Force files to write to disk On 22/02/17 13:17, James Richters wrote: > No other programs on the system ever access this file Not even virus scanners? Jonas
Mark Morgan Lloyd
to James Richters
James Richters wrote
I suppose a virus scanner could access it, I didn't think of that. I guess I should have said no other program intentionally access the file. It's just a way for me to store some variables and get them back, it serves no other purpose.

A virus scanner might disagree. .pax is a known type which might contain an archive hence a whole lot of nasty stuff, and it probably isn't wise to use it casually... call it .txt or use the standard .ini format.

Mark Morgan Lloyd markMLl .AT. telemetry.co .DOT. uk [Opinions above are the author's, not those of his employers or colleagues]
Klaus Hartnegg
to James Richters: I have a freepascal Windows console application that I recently ported over from DOS…
James Richters wrote
Assign(BitFile,'BitSave.pax');
    {$I-}
ReWrite(BitFile);
    if ioresult  0 then
       writeln ('error opening pax file');
Show quoted text 10 lines
WriteLn(BitFile,XADJ:1:8); WriteLn(BitFile,YADJ:1:8); WriteLn(BitFile,ZADJ:1:8); WriteLn(BitFile,WADJ:1:8); WriteLn(BitFile,AADJ:1:8); WriteLn(BitFile,TADJ:1:8); WriteLn(BitFile,VADJ:1:8); WriteLn(BitFile,UADJ:1:8); WriteLn(BitFile,CurrentTool); WriteLn(Bitfile,P_Value[4]);
    writeln (bitfile, 'EOF');
    if ioresult  0 then
       writeln ('error writing pax file');
Close(BitFile);
    if ioresult  0 then
       writeln ('error closing pax file');
    {$I+}

This writes something at the end of the file that does not come from a variable. Next time the error happens, you can check if that is present. Then your variables were overwritten in memory. Also it checks for errors. And I would disable antivirus or at least tell it to not scan this file.

James Richters
to Klaus Hartnegg: {$I-} if ioresult 0 then writeln ('error opening pax file'); writeln (bitfile, 'EOF');…

Thanks for the advice everyone. It is very much appreciated. I have been working on a combination of ideas here. Currently I am doing an MD5sum of a string created from all my variables concatenated together and writing that as the second to the last line of the file, I also am writing some fixed text at the beginning and end of the file as suggested so I can check for variables being overwritten in memory, great idea! I am then reading the file back to verify it has written properly, making a backup, checking that, and also using ioresults as below. I will probably rename the file as well, and exclude it from virus checkers. When I read the file I can verify the MD5sum and if that test fails, I'll just automatically try the backup, If that also fails the MD5sum I'll notify the user.

Thanks again for all the help

James

Earlier message 35 lines
-----Original Message----- From: On Behalf Of Klaus Hartnegg Sent: Thursday, February 23, 2017 6:33 AM To: Subject: Re: [fpc-pascal] FreePascal Windows - Force files to write to disk Am 21.02.2017 um 22:12 schrieb James Richters: > Assign(BitFile,'BitSave.pax'); {$I-} > ReWrite(BitFile); if ioresult 0 then writeln ('error opening pax file'); > WriteLn(BitFile,XADJ:1:8); > WriteLn(BitFile,YADJ:1:8); > WriteLn(BitFile,ZADJ:1:8); > WriteLn(BitFile,WADJ:1:8); > WriteLn(BitFile,AADJ:1:8); > WriteLn(BitFile,TADJ:1:8); > WriteLn(BitFile,VADJ:1:8); > WriteLn(BitFile,UADJ:1:8); > WriteLn(BitFile,CurrentTool); > WriteLn(Bitfile,P_Value[4]); writeln (bitfile, 'EOF'); if ioresult 0 then writeln ('error writing pax file'); > Close(BitFile); if ioresult 0 then writeln ('error closing pax file'); {$I+} This writes something at the end of the file that does not come from a variable. Next time the error happens, you can check if that is present. Then your variables were overwritten in memory. Also it checks for errors. And I would disable antivirus or at least tell it to not scan this file.
James Richters
to James Richters

I am still having this issue. I've managed to narrow down what is happening some. The problem is my data is not actually being written all the way to disk. I now have a repeatable proven method to reproduce the issue. Here is the sequence.

In my freepascal console application, I create the file Assign file
Rewrite file
Writeln file
Writeln file
Writeln file ...
Flush file
Close file

I then immediately read back the file, and all is good. I even look at the file with notepad++ to see that the data is correct.... and it is fine.

If I then power off the test computer (not doing a correct windows shutdown, this simulates an unattended power failure) when it boots back up, my file has the correct date and time and the correct length, hovever the contents of the file is a string of $00 with no other characters, no carriage returns and no linefeeds, however the number of $00s is EXACTLY how many bytes the file should have been. It's like the file was allocated but data not actually written to disk yet... but I thought Close (file) was supposed to do that. I think I am dealing with some kind of disk caching going on here, but I'm not sure how to force my files to commit all the way to disk. I should note that these systems are all using Samsung SSDs in them, perhaps there is some SSD weirdness going on? I also notice it's not just this one file that is affected, but EVERY SINGLE FILE I create with freepascal, this is the only one I notice because the other ones are all work files that end up being re-created every time I run my program, but I checked them all and they are all exactly the same.. correct lengths but all data is a string of 00s

I'm thinking of just disabling all write behind caching in windows and on the SSD if I can figure it out, it's not needed anyway anymore because SSDs are way more than fast enough, and I would rather wait for my data to be written NOW instead of it NOT getting written later!! The problem with solving this by disabling write caching is that I can never be sure that this was done. So I would like to find a real solution that would force my files all the way to disk on my own without depending on the system being configured a certain way. I really need these tiny files to survive power failures.

Thanks for any advice on how to do this

James

Earlier message 57 lines
-----Original Message----- From: On Behalf Of James Richters Sent: Friday, February 24, 2017 1:53 PM To: 'FPC-Pascal users discussions' Subject: Re: [fpc-pascal] FreePascal Windows - Force files to write to disk Thanks for the advice everyone. It is very much appreciated. I have been working on a combination of ideas here. Currently I am doing an MD5sum of a string created from all my variables concatenated together and writing that as the second to the last line of the file, I also am writing some fixed text at the beginning and end of the file as suggested so I can check for variables being overwritten in memory, great idea! I am then reading the file back to verify it has written properly, making a backup, checking that, and also using ioresults as below. I will probably rename the file as well, and exclude it from virus checkers. When I read the file I can verify the MD5sum and if that test fails, I'll just automatically try the backup, If that also fails the MD5sum I'll notify the user. Thanks again for all the help James -----Original Message----- From: On Behalf Of Klaus Hartnegg Sent: Thursday, February 23, 2017 6:33 AM To: Subject: Re: [fpc-pascal] FreePascal Windows - Force files to write to disk Am 21.02.2017 um 22:12 schrieb James Richters: > Assign(BitFile,'BitSave.pax'); {$I-} > ReWrite(BitFile); if ioresult 0 then writeln ('error opening pax file'); > WriteLn(BitFile,XADJ:1:8); > WriteLn(BitFile,YADJ:1:8); > WriteLn(BitFile,ZADJ:1:8); > WriteLn(BitFile,WADJ:1:8); > WriteLn(BitFile,AADJ:1:8); > WriteLn(BitFile,TADJ:1:8); > WriteLn(BitFile,VADJ:1:8); > WriteLn(BitFile,UADJ:1:8); > WriteLn(BitFile,CurrentTool); > WriteLn(Bitfile,P_Value[4]); writeln (bitfile, 'EOF'); if ioresult 0 then writeln ('error writing pax file'); > Close(BitFile); if ioresult 0 then writeln ('error closing pax file'); {$I+} This writes something at the end of the file that does not come from a variable. Next time the error happens, you can check if that is present. Then your variables were overwritten in memory. Also it checks for errors. And I would disable antivirus or at least tell it to not scan this file.
Henry Vermaak
to James Richters
James Richters wrote 12 lines
I am still having this issue. I've managed to narrow down what is happening some. The problem is my data is not actually being written all the way to disk. I now have a repeatable proven method to reproduce the issue. Here is the sequence. In my freepascal console application, I create the file Assign file Rewrite file Writeln file Writeln file Writeln file ... Flush file

Have you tried using the FlushFileBuffers() Windows API? Something like this:

FlushFileBuffers(TextRec(AFile).Handle);

Add "windows" to the uses clause, obviously.

Henry

Jonas Maebe
to James Richters: I am still having this issue. I've managed to narrow down what is happening some. The…
James Richters wrote 8 lines
If I then power off the test computer (not doing a correct windows shutdown, this simulates an unattended power failure) when it boots back up, my file has the correct date and time and the correct length, hovever the contents of the file is a string of $00 with no other characters, no carriage returns and no linefeeds, however the number of $00s is EXACTLY how many bytes the file should have been. It's like the file was allocated but data not actually written to disk yet... but I thought Close (file) was supposed to do that.

No, it does not do that. It flushes everything from the internal program buffers to the operating system, but the operating system will perform write-back caching. This is not specific to FPC programs, it does this almost all writes to a regular hard drive or SSD (Windows does not do it for USB sticks, afaik). Not doing so would slow down you system significantly.

Jonas

James Richters
to Henry Vermaak: Have you tried using the FlushFileBuffers() Windows API? Something like this:…

I have not tried FlushFileBuffers() yet, so I just tried it now, I am getting the same results. I have also tried disabling write caching in disk management and STILL have the same results. I am now thinking this is some caching being done on the SSD that is ignoring FlushFileBuffers() Probably a 'feature' designed to help extend the life of the SSD, but making it useless to store data that needs to survive a power failure.

I have tried running my freepascal program on a normal mechanical hard disk and flushfilebuffers() seems to be working as expected, Data is getting actually written to the disk and my 'power failure' is not affecting the data at all, but on the SSD, it's still just a correct length file of 00s

Does anyone know of a way to force critical data to be written to a SSD so it's not lost during a power failure?

James

Earlier message 24 lines
-----Original Message----- From: fpc-pascal On Behalf Of Henry Vermaak Sent: Tuesday, March 21, 2017 11:08 AM To: FPC-Pascal users discussions Subject: Re: [fpc-pascal] FreePascal Windows - Force files to write to disk On Tue, Mar 21, 2017 at 10:39:23AM -0400, James Richters wrote: > I am still having this issue. I've managed to narrow down what is happening > some. The problem is my data is not actually being written all the way to > disk. I now have a repeatable proven method to reproduce the issue. Here > is the sequence. > > In my freepascal console application, I create the file Assign file > Rewrite file Writeln file Writeln file Writeln file ... > Flush file Have you tried using the FlushFileBuffers() Windows API? Something like this: FlushFileBuffers(TextRec(AFile).Handle); Add "windows" to the uses clause, obviously. Henry
Marco van de Voort
to Henry Vermaak: Have you tried using the FlushFileBuffers() Windows API? Something like this:…

In our previous episode, Henry Vermaak said:

Show quoted text 10 lines
> Writeln file > Writeln file ... > Flush file Have you tried using the FlushFileBuffers() Windows API? Something like this: FlushFileBuffers(TextRec(AFile).Handle); Add "windows" to the uses clause, obviously.

He closes the file. That should sync. Sounds to me he is using some dodgy FS (network?) driver

Jonas Maebe
to Marco van de Voort
Marco van de Voort wrote
He closes the file. That should sync.

Not necessarily. On Unix there is also a difference between closing a file and calling fsync() on it. And even then there are sometimes additional things you may have to do to survive sudden power loss, such as first using fcntl(fd,F_FULLFSYNC) on OS X in case you want to OS to also instruct the disk controller to flush its internal caches (if supported by the disk).

Jonas

Henry Vermaak

On 21 Mar 2017 18:59, "James Richters"
wrote:

I have not tried FlushFileBuffers() yet, so I just tried it now, I am getting the same results. I have also tried disabling write caching in disk management and STILL have the same results. I am now thinking this is some caching being done on the SSD that is ignoring FlushFileBuffers() Probably a 'feature' designed to help extend the life of the SSD, but making it useless to store data that needs to survive a power failure.

Can you turn off the drive write cache in device manager? You'll have to double click on the ssd and look around in there.

Henry

Ralf Quint
to James Richters: I have not tried FlushFileBuffers() yet, so I just tried it now, I am getting the same…
James Richters wrote
I have not tried FlushFileBuffers() yet, so I just tried it now, I am getting the same results.

Well, that is expected, as the issue has nothing to do with your program, it's the caching of the operating system that is responsible for the actual physical write to the disk.
I see something similar with some text files that I use while doing some cross-platform work that I am doing recently.

The actual test file is on a Windows 8.1 machine, while most of the access to that file is from either a Windows 10 laptop or a macOS Sierra MacBook Pro. Occasionally, more pronounced when one of these two machines is connected via WiFi instead of wired connection (on the same logical subnet though!), when reloading the file from disk, it shows up corrupted on the other machine, while it shows perfectly fine on the one that did the last write. Force a new write and in 99.9% of all cases, it will show up just fine on another reload...

Any chance that the file(s) you are working with are on a shared drive?

Ralf

---
This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus

Santiago A.
to James Richters: I am still having this issue. I've managed to narrow down what is happening some. The…
James Richters wrote
I should note that these systems are all using Samsung SSDs in them, perhaps there is some SSD weirdness going on?

Yes, I think so. SSD is special.

The SSD technology like any flash memories have a limited number of rewrites, so there are strategies to rewrite as less as possible. In the case of SSD they are even more aggressive, because they are expensive .Some of these strategies are implemented in the OS in driver level, but some in firmware of the device. So, it is possible that OS can't completely force the real write.

I also notice it's not just this one file that is affected, but EVERY SINGLE FILE I create with freepascal,

Only with freepascal?

Try it for a file generated from command line

echo "test" > filetest.txt

And do the same checks... edit notepad++... turnoff power.... etc

configured a certain way. I really need these tiny files to survive power failures.

Power failures are always a problem for persistent storage. Caching is a trade-off between speed and security. In the case of SSD there is a third point: Minimize the real writes on disk. So, if you want security against power failures, SSD is not the best idea.

By the way. What about a SAI? ;-)

Saludos Santiago A.
Henry Vermaak
to Ralf Quint: Well, that is expected, as the issue has nothing to do with your program, it's the…
Ralf Quint wrote 5 lines
On 3/21/2017 11:58 AM, James Richters wrote: >I have not tried FlushFileBuffers() yet, so I just tried it now, I am getting the same results. Well, that is expected, as the issue has nothing to do with your program, it's the caching of the operating system that is responsible for the actual physical write to the disk.

No, FlushFileBuffers() flushes the OS cache - see the MSDN page. The problem here seems to be that the SSD has its own cache (in the FTL).

Henry

James Richters
to Santiago A.: Yes, I think so. SSD is special. The SSD technology like any flash memories have a…

No, it is not only freepascal, but not every program either.

I have just done some quick tests check this. I have not done any real testing to prove whether these are repeatable all the time or not yet. A file saved with notepad ++ is lost
Partial email drafts in outlook are saved
A file created with Echo "test" > test.txt is saved Copying a file from the command line seems to end up saving both the source and destination. I tried saving a file with notepad++ then copying the file, and after the power failure, both the source and destination were complete. But saving with notepad ++ will always be a string of 00s.

I wonder if this is a 32/64 bit issue. Notepad ++ and my free pascal programs are both 32bit.

Perhaps until I can figure out what the copy command is doing to save the file to disk, I can just shell execute a copy command on my files and that will end up saving them. I guess if I did that I could run my entire program from a ram drive and save some SSD activity and only copy the important files back to the SSD and only when they actually change.

James

Earlier message 36 lines
-----Original Message----- From: fpc-pascal On Behalf Of Santiago A. Sent: Wednesday, March 22, 2017 4:40 AM To: FPC-Pascal users discussions Subject: Re: [fpc-pascal] FreePascal Windows - Force files to write to disk El 21/03/2017 a las 15:39, James Richters escribió: > I should note that these systems are all using Samsung SSDs in them, > perhaps there is some SSD weirdness going on? Yes, I think so. SSD is special. The SSD technology like any flash memories have a limited number of rewrites, so there are strategies to rewrite as less as possible. In the case of SSD they are even more aggressive, because they are expensive .Some of these strategies are implemented in the OS in driver level, but some in firmware of the device. So, it is possible that OS can't completely force the real write. > I also notice it's not just this > one file that is affected, but EVERY SINGLE FILE I create with > freepascal, Only with freepascal? Try it for a file generated from command line echo "test" > filetest.txt And do the same checks... edit notepad++... turnoff power.... etc > configured a certain way. I really need these tiny files to survive power > failures. Power failures are always a problem for persistent storage. Caching is a trade-off between speed and security. In the case of SSD there is a third point: Minimize the real writes on disk. So, if you want security against power failures, SSD is not the best idea. By the way. What about a SAI? ;-) -- Saludos Santiago A.
Karoly Balogh (Charlie/SGR
to James Richters

Hi,

James Richters wrote
No, it is not only freepascal, but not every program either.

I'm no Windows expert, but Windows API seems to have a way do to this directly. CreateFile() API seems to have a flag, which has a value FILE_FLAG_WRITE_THROUGH, which will cause the file not go through the write cache, but gets written directly to the underlying disk, without delays.

However, this obviously has performance implications on write (will be a lot slower), and as the documentation notes, it's not supported on every hardware or file system. Also, I'm not sure if you can do this somehow together with Free Pascal's file handling API, or you have to restrict yourself to using direct Windows API calls entirely for writing this file.

Relevant MSDN page:
https://msdn.microsoft.com/en-us/library/windows/desktop/aa363858(v=vs.85).aspx#caching_behavior

However, please note that transactional file handling on power loss is a quite delicate scenario, so you might not be able to solve all the related problems with this - what happens exactly when you get a power loss during a write, might be matter of pure luck.

The usual way to work this problem around on Linux systems at least, is to write a new file, then do an overwriting rename to the old file name. There rename is an "atomic" operation, which will be either committed to the disk or not, but if it was unsuccessful, you won't end up with half-written files. But IIRC Windows didn't support atomic renames. Maybe someone with more Windows knowledge will fix me. You definitely don't want to implement a copy though, and that's anything but atomic.

Charlie

Giuliano Colla
to James Richters: No, it is not only freepascal, but not every program either. I have just done some quick…

Il 22/03/2017 13:20, James Richters ha scritto:

No, it is not only freepascal, but not every program either.

A wild guess. The vilain could be the journal logic.

The journal tells that the write operation has not been completed at power off, and restores the last valid version of the file, which was empty, because of the rewrite you performed after assigning.

You might try overwriting the old file instead of clearing it. Something like:

Assign.
Seek to the beginning of the file. (instead of rewrite) Write whatever you need.
Truncate.
Close.

This will give the journal a different story to cope with. Maybe it will help.

Giuliano

Karoly Balogh (Charlie/SGR
to Karoly Balogh (Charlie/SGR: I'm no Windows expert, but Windows API seems to have a way do to this directly.…

Hi,

Karoly Balogh wrote 12 lines
However, please note that transactional file handling on power loss is a quite delicate scenario, so you might not be able to solve all the related problems with this - what happens exactly when you get a power loss during a write, might be matter of pure luck. The usual way to work this problem around on Linux systems at least, is to write a new file, then do an overwriting rename to the old file name. There rename is an "atomic" operation, which will be either committed to the disk or not, but if it was unsuccessful, you won't end up with half-written files. But IIRC Windows didn't support atomic renames. Maybe someone with more Windows knowledge will fix me. You definitely don't want to implement a copy though, and that's anything but atomic.

Actually, this blogpost which I've just found also details the algorithm how you should do it, from a Windows perspective:

https://blogs.msdn.microsoft.com/adioltean/2005/12/28/how-to-do-atomic-writes-in-a-file/

Charlie

Karoly Balogh (Charlie/SGR
to Giuliano Colla: Il 22/03/2017 13:20, James Richters ha scritto: A wild guess. The vilain could be the…

Hi,

Giuliano Colla wrote
Il 22/03/2017 13:20, James Richters ha scritto: > No, it is not only freepascal, but not every program either. A wild guess. The vilain could be the journal logic.

Wild indeed. :)

Show quoted text 11 lines
You might try overwriting the old file instead of clearing it. Something like: Assign. Seek to the beginning of the file. (instead of rewrite) Write whatever you need. Truncate. Close. This will give the journal a different story to cope with. Maybe it will help.

Even if this would work (no idea), it builds completely on assumptions and the behavior might be different depending on the underlying storage system. It's not atomic as well, because if you get a power loss during the above process, you could still end up with a broken file with random contents. So it is dangerous, and could be very hard to reproduce if it causes problems.

See my other mails about the documented way, and what Microsoft seem to suggest.

Charlie

Henry Vermaak
to Karoly Balogh (Charlie/SGR: I'm no Windows expert, but Windows API seems to have a way do to this directly.…
Karoly Balogh wrote 7 lines
The usual way to work this problem around on Linux systems at least, is to write a new file, then do an overwriting rename to the old file name. There rename is an "atomic" operation, which will be either committed to the disk or not, but if it was unsuccessful, you won't end up with half-written files. But IIRC Windows didn't support atomic renames. Maybe someone with more Windows knowledge will fix me. You definitely don't want to implement a copy though, and that's anything but atomic.

I remember discussions about atomic rename for go and python (you can still find the bug reports online). The consensus seems to be that MoveFileEx with MOVEFILE_REPLACE_EXISTING and MOVEFILE_WRITE_THROUGH flags will be atomic if the file is on the same volume, and if that volume supports changing the metadata in one transaction (which most do, afaik).

MSDN recommends ReplaceFile() instead of TxF:

https://msdn.microsoft.com/en-us/library/windows/desktop/hh802690(v=vs.85).aspx

But I _think_ this is just a convenience function that uses MoveFileEx().

So you can basically follow the same strategy as on POSIX, but with MoveFileEx().

Henry

Giuliano Colla
to Karoly Balogh (Charlie/SGR: Wild indeed. :) Even if this would work (no idea), it builds completely on assumptions and

Il 22/03/2017 14:21, Karoly Balogh (Charlie/SGR) ha scritto:

Show quoted text 29 lines
Hi, On Wed, 22 Mar 2017, Giuliano Colla wrote: > Il 22/03/2017 13:20, James Richters ha scritto: >> No, it is not only freepascal, but not every program either. > A wild guess. The vilain could be the journal logic. Wild indeed. :) > You might try overwriting the old file instead of clearing it. Something > like: > > Assign. > Seek to the beginning of the file. (instead of rewrite) > Write whatever you need. > Truncate. > Close. > > This will give the journal a different story to cope with. Maybe it will > help. Even if this would work (no idea), it builds completely on assumptions and the behavior might be different depending on the underlying storage system. It's not atomic as well, because if you get a power loss during the above process, you could still end up with a broken file with random contents. So it is dangerous, and could be very hard to reproduce if it causes problems. See my other mails about the documented way, and what Microsoft seem to suggest.

There's no doubt that the best solution would be writing a new file and then renaming it, second only to using an UPS to protect against accidental power loss, which costs just a fraction of the time value spent in this discussion.

My suggestion was just a test to get an idea on how the MS Journaling works, and which pitfalls it may hide. If they store/execute the pending operations not in the order they where given, but giving somehow precedence to the operations faster to execute, and dropping the actions which are no more executable, you might end up with a renamed file which is still empty, because its creation and renaming are by far much faster than writing it!

As far as Microsoft's suggestion are related, a friend of mine with some decades of career as a MCSE once told me is that, according his experience, the best course to take is always *not* to follow Microsofts suggestions, but to do exactly the opposite....:-)

Giuliano

James Richters
to Karoly Balogh (Charlie/SGR: Actually, this blogpost which I've just found also details the algorithm how you should…

Thanks for the link and information on atomic writes. It seems like a much better method. My current scheme is to copy everything to a ramdrive upon boot. Here I can do what I want with my files and if the machine crashes, the files I started with on the hard drive have not been affected. Now for the tricky part, If I change a file, and want to save the change back to the hard drive in a way that is either 100% successful or 0% successful. My thinking is that if I create the new file on my ram drive, and my program is still running, I can then perform the following sequence:

Copy the file to the hard drive with a temporary name Flush all writes to above file
Delete previous backup file
Rename the original file to backup file name
Rename the new recently copied file to the original name

At this point I should have the current data with the original name and a backup of the previous data in case something goes wrong before the sequence completes.

I would like to do this without shell execute, and it seems my best bet is to use windows api calls instead of pascal functions. I have FlushFileBuffers() working. For the copy should I use MoveFileEx() Should I use DeleteFile() to delete the previous backup and Rename() to rename the files or is there a better way? Will I need to do something for commit the rename to disk?

Thanks for the help with this

James

Earlier message 28 lines
-----Original Message----- From: fpc-pascal On Behalf Of Karoly Balogh (Charlie/SGR) Sent: Wednesday, March 22, 2017 9:13 AM To: FPC-Pascal users discussions Subject: Re: [fpc-pascal] FreePascal Windows - Force files to write to disk Hi, On Wed, 22 Mar 2017, Karoly Balogh (Charlie/SGR) wrote: > However, please note that transactional file handling on power loss is > a quite delicate scenario, so you might not be able to solve all the > related problems with this - what happens exactly when you get a power > loss during a write, might be matter of pure luck. > > The usual way to work this problem around on Linux systems at least, > is to write a new file, then do an overwriting rename to the old file name. > There rename is an "atomic" operation, which will be either committed > to the disk or not, but if it was unsuccessful, you won't end up with > half-written files. But IIRC Windows didn't support atomic renames. > Maybe someone with more Windows knowledge will fix me. You definitely > don't want to implement a copy though, and that's anything but atomic. Actually, this blogpost which I've just found also details the algorithm how you should do it, from a Windows perspective: https://blogs.msdn.microsoft.com/adioltean/2005/12/28/how-to-do-atomic-writes-in-a-file/ Charlie
José Mejuto
to James Richters
James Richters wrote 5 lines
Copy the file to the hard drive with a temporary name Flush all writes to above file Delete previous backup file Rename the original file to backup file name Rename the new recently copied file to the original name

Hello,

Your problem seems to be that you are using an SSD without a big capacitor (or partially damaged one) to preserve writes on power loss. This fact combined with how TRIM works could create that effect.

You can try this steps:

1) Copy file content to .bak
2) Create new file with data (.datnew)
3) Close file (This forces SSD to write to an spare block)
4) Delete old file (This forces a TRIM sent to the SSD)
5) Rename file to original name.

In step 4, the delete should send a TRIM which could force a SSD-RAM to NAND-SSD write to handle the TRIM. You must not delete 0 bytes (no TRIM) nor files with a size less than 1024 (no TRIM in NTFS as most of them are stored as resident data in MFT).

James Richters
to Klaus Hartnegg

I'm trying to use copyfile and cannot get it to work, anyone have any ideas?

I'm using the procedure below. My writeln showing the contents of the variables seems to indicate they are correct. The souce file does exist and is correct, the destination file does not exitst. I get an error 50 which is

ERROR_NOT_SUPPORTED 50 (0x32)
The request is not supported.

I don't see how copying a file I just created is not supported. Does anyone have any ideas how to get this to work or what I may be doing wrong, or if there is a better way I should be copying my file? I've tried using single quotes and double quotes around the file names.

Thanks

James

Uses
   Serial,Windows,jwanative,sysutils;
Procedure Copy_Backup_Rename(CBR_Filename:String);
var
  fileSource, fileDest: string;
  resultsofit : Boolean;
Begin
   fileSource := chr(39)+Prog_Drive+Prog_Path+'\'+CBR_Filename+'.tmp'+chr(39);
   fileDest := chr(39)+'C:'+Prog_Path+'\'+CBR_Filename+'.temp'+chr(39);
   writeln (filesource+'    '+filedest);
   resultsofit := CopyFile(PChar(fileSource), PChar(fileDest), False);
   writeln (resultsofit);
   writeln(getlasterror);
End;

Console output:
'C:\ProMill\BitTool.tmp' 'C:\ProMill\BitTool.temp' FALSE
50

Sandro Cumerlato
to James Richters

Try " char instead of ' (char 39). I never use ' under Windows.

Best regards
Sandro Cumerlato

"James Richters" wrote 47 lines
I'm trying to use copyfile and cannot get it to work, anyone have any ideas? I'm using the procedure below. My writeln showing the contents of the variables seems to indicate they are correct. The souce file does exist and is correct, the destination file does not exitst. I get an error 50 which is ERROR_NOT_SUPPORTED 50 (0x32) The request is not supported. I don't see how copying a file I just created is not supported. Does anyone have any ideas how to get this to work or what I may be doing wrong, or if there is a better way I should be copying my file? I've tried using single quotes and double quotes around the file names. Thanks James Uses Serial,Windows,jwanative,sysutils; Procedure Copy_Backup_Rename(CBR_Filename:String); var fileSource, fileDest: string; resultsofit : Boolean; Begin fileSource := chr(39)+Prog_Drive+Prog_Path+' \'+CBR_Filename+'.tmp'+chr(39); fileDest := chr(39)+'C:'+Prog_Path+'\'+CBR_Filename+'.temp'+chr(39); writeln (filesource+' '+filedest); resultsofit := CopyFile(PChar(fileSource), PChar(fileDest), False); writeln (resultsofit); writeln(getlasterror); End; Console output: 'C:\ProMill\BitTool.tmp' 'C:\ProMill\BitTool.temp' FALSE 50 _______________________________________________ fpc-pascal maillist - http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
James Richters
to Sandro Cumerlato

Thanks for the response, it got me on the right track and I got it working.

I originally did try “ instead of ‘ and have the same results. I was using ‘ because of the example here: http://wiki.freepascal.org/CopyFile

I was able to get it to work without any ‘ or “ I even did a test with a space in the file name and it worked fine without “

It appears The function must put “ around it for you.

James

Earlier message 56 lines
From: fpc-pascal On Behalf Of Sandro Cumerlato Sent: Thursday, March 23, 2017 1:30 AM To: FPC-Pascal users discussions Subject: Re: [fpc-pascal] FreePascal Windows - Force files to write to disk Try " char instead of ' (char 39). I never use ' under Windows. Best regards Sandro Cumerlato On 22 Mar 2017 23:47, "James Richters" < > wrote: I'm trying to use copyfile and cannot get it to work, anyone have any ideas? I'm using the procedure below. My writeln showing the contents of the variables seems to indicate they are correct. The souce file does exist and is correct, the destination file does not exitst. I get an error 50 which is ERROR_NOT_SUPPORTED 50 (0x32) The request is not supported. I don't see how copying a file I just created is not supported. Does anyone have any ideas how to get this to work or what I may be doing wrong, or if there is a better way I should be copying my file? I've tried using single quotes and double quotes around the file names. Thanks James Uses Serial,Windows,jwanative,sysutils; Procedure Copy_Backup_Rename(CBR_Filename:String); var fileSource, fileDest: string; resultsofit : Boolean; Begin fileSource := chr(39)+Prog_Drive+Prog_Path+'\'+CBR_Filename+'.tmp'+chr(39); fileDest := chr(39)+'C:'+Prog_Path+'\'+CBR_Filename+'.temp'+chr(39); writeln (filesource+' '+filedest); resultsofit := CopyFile(PChar(fileSource), PChar(fileDest), False); writeln (resultsofit); writeln(getlasterror); End; Console output: 'C:\ProMill\BitTool.tmp' 'C:\ProMill\BitTool.temp' FALSE 50
Sandro Cumerlato
to James Richters

On 23 Mar 2017 11:34, "James Richters"
wrote:

Thanks for the response, it got me on the right track and I got it working.

Happy to help you!

I originally did try “ instead of ‘ and have the same results. I was using ‘ because of the example here: http://wiki.freepascal.org/CopyFile

CopyFile('c:\autoexec.bat','c:\windows\temp\autoexec.bat.backup');

' delimiters are mandatory within this example because we are writing paths as string parameters directly into the source code.

I believe that I should use both ' and " in case of spaces within paths:

CopyFile('"c:\my silly path\autoexec.bat"','"c:\windows\temp\my silly
path\autoexec.bat.backup"');

or is the CopyFile function smart enough to include delimiters automatically?

Best regards,
Sandro Cumerlato

Marco van de Voort
to Sandro Cumerlato

In our previous episode, Sandro Cumerlato said:

Show quoted text 7 lines
' delimiters are mandatory within this example because we are writing paths as string parameters directly into the source code. I believe that I should use both ' and " in case of spaces within paths: CopyFile('"c:\my silly path\autoexec.bat"','"c:\windows\temp\my silly path\autoexec.bat.backup"');

No. The "" is a shell/cmdline convention, and API calls are generally not shell.