I know that it seems absurd, and it probably is.

Short version:

FindInvalidUTF8Character could access the string beyond its length since it uses p[1], p[2] and p[3] without properly checking that those aren't beyond the end of the string.
In the worst case this should cause nothing more than a sigsev, however here, under circumstances that I cannot determine exactly (i.e. random), it seems to hang linux hard.

Long version:

I have a (relatively simple) program that has been running flawlessly 24x7 the last 4 years.
All it does is detect the presence of a piece in a production line (using the control lines of a serial port as an input), read some data via another serial port and display the data.
The main loop (triggered by a timer on the main form) checks the input and signals the reading thread. Everything is using the correct (I hope) synchronization primitives.
The class implementing the reading protocol and the related logging is a class that I'm using in a lot of programs, and when I update it, all programs compiled after the update obviously inherit the update. Last summer I had to do a trivial modification to this program and I switched it to gtk2 from the original gtk1, since it's now mature enough. After a while they told me that the program is hanging every two to three weeks, so I prepared a test rig (same hardware, same version of the os, same everything) to simulate the operation and see where the program was hanging.
After a week and around a million cycles, it hanged, but it wasn't the application, the whole os was completely frozen.
This is scary, since if it's a problem with the application I can hunt it down, but the os hanging is beyond my control, besides, an app should never freeze the os, unless there's a bug in the kernel or in the X server. Since the biggest change was the switch to gtk2, I thought to switch back the program to gtk1.
Well, with gtk1 it reliably hanged after less than some hundreds cycles. I tried everything (using a different X driver, vesa instead of the hardware specific one, using a newer version of linux, testing on a different machine, etc.) and it still hanged, so I checked my mercurial repository to see what I changed.
I excluded lazarus and the widgetset (without using the serial port, i.e. simulating the data reading it from files, it never hanged, so the display part of the program was fine), and the only "big" difference is that now, in the logging part of the serial port routine, I call FindInvalidUTF8Character to check that there are no printable characters. I removed the call (actually I removed the whole logging routine, that, as I said earlier, has been working fine for the last 4 years and in a number of programs) and now, crossing fingers, it's done 12000 cycles without hanging.
The most intriguing part is that, under the test conditions, FindInvalidUTF8Character is only given pure ASCII strings, so it should never enter the branches where it checks p[1], p[2] or p[3], so I'm really puzzled.

Bye

Me too.
Nevertheless, I fixed FindInvalidUTF8Character.

Mattias

En/na Mattias Gaertner ha escrit:

Thanks.
Meanwhile the program has been running all the week-end without hanging the system.

Bye

En/na Luca Olivetti ha escrit:

Well, it hanged again today, with no call to FindInvalidUTF8Character in sight. I hoped I nailed it but it was a red herring :-( Note that the version that worked through the week-end was compiled with gtk1, the one that hanged today with gtk2 (but I think that's another red herring).
I'm re-resting now the gtk1 version to see what happens.

Bye

En/na Luca Olivetti ha escrit:

Well, I think (though I cannot be sure until I had the system running more time) that I found a solution: I reverted the development environment to the same situation as 2 years ago (when I made the last successful modification to this program), i.e:

fpc 2.0.4 (with a couple of patches so that the ide works with gtk2 even if the final program is for gtk1)

lazarus svn 11880 (I think I used an earlier version at the time, but I couldn't determine the exact version, this one is one of the latest compiling with fpc 2.0.4 and working with the ide compiled for gtk2)

If this works (crossing fingers), I'm *not* happy with the solution, since I'd really like to know what happened to make linux hang hard using a newer fpc and a newer lazarus (I cannot say which one is the culprit).
This is running on an epia board, though it's a nehemiah and it shouldn't be affected by the cmov issue (if that could be the cause of the problem, which I doubt).

Bye

with "linux hang hard" you mean the machine is completely frozen? Does not react on ping and so on? Could it be a hardware problem? Maybe the BIOS settings are to "aggressive". Maybe there is a hardware memory problem, which is only popping up when you use the modern (perhaps more memory consuming) fpc/lazarus/gtk environment.

Regards, Bernd.

En/na Bernd Mueller ha escrit:

with "linux hang hard" you mean the machine is completely frozen?

Yes. I even left it on a text console to see if there was an Oops or something, but it just froze.

Does not react on ping and so on?

Nothing, not even to the magic sysreq key.

Could it be a hardware problem?

I doubt it: it happens on four machines (five if counting my test rig). They're all the same and the problem manifests in all of them since I installed the new version of the program.
Oh, and I tested on a different machine too (though it's even more underpowered than the epia, but I have that class of machine working reliably with a similar, if not quite the same, kind of program, and with gtk2 and lazarus from around 1 year ago).

Maybe the BIOS settings are to "aggressive".

I tried many different bios settings. Knowing that there's a problem with dma on this board, I even updated the bios. Alas, it made no difference (not that I was expecting one: this system doesn't use neither the disk nor the network).

Maybe there is a hardware memory problem, which is only popping up when you use the modern (perhaps more memory consuming) fpc/lazarus/gtk environment.

Memory and cpu consumption is just about the same. I used memtest to check for memory errors.
What's more important, is that the memory usage (VSZ/RSS as reported by ps) of the program is constant once it has reached a plateau value, so apparently it shouldn't be allocating more memory. Note that the freeze happens at a random time: it could be 10 minutes, it could be a week (hence, I'm not sure that reverting the development environment actually cured the issue, I need to leave it running for at least a couple of week).
It always seems to happen doing the same thing though: (i.e., I display various fields on the screen, when reading new data I display dashes in the fields and "Reading" at the bottom of the page, and the crash always happens at that time).

Unfortunately I think I covered all my bases :-(

Bye

Do you have a machine, where it does not hang?

Mattias

I am no Linux expert, but I think it should be impossible to freeze the kernel with a wild running "userland" application. I still think it is somehow hardware related.

You could run your application in several virtual machines parallel. May be you could use a different Linux distribution/kernel version. The host pc (for the hypervisor) should work reliable of course ;-)

Regards, Bernd.

En/na Mattias Gaertner ha escrit:

No, I mean, I just tested on the real machine (with 3 different kernels) and another one. I could try on another machine, yes.

Bye

En/na Bernd Mueller ha escrit:

My thoughts exactly

I still think it is somehow hardware related.

It could be, but since it happens also on a completely different machine, with different processor, memory, video and an onboard serial port instead of the usb adapter, maybe it's an obscure kernel bug that's triggered only under some special conditions :-(

You could run your application in several virtual machines parallel. May be you could use a different Linux distribution/kernel version. The host pc (for the hypervisor) should work reliable of course ;-)

Maybe that could be the only way to see where it hangs (if it happens to hang on the virtual machine).
I'll see what debugging options VirtualBox has.

Bye

[...] I am no Linux expert, but I think it should be impossible to freeze the kernel with a wild running "userland" application. I still think it is somehow hardware related.

It is ridiculously easy to "freeze" linux.
Just start hundreds of threads or use a lot of memory.

Use the ulimit command to set some limits for your program.

Mattias

yes, you are right, I would not have expected that:

procedure TForm1.Button1Click(Sender: TObject);
var
    p: Pointer;
    i: Integer;
begin
    i:= 0;
    repeat
       p:= GetMem(1024);
       inc(i);
       Caption:= IntToStr(i) + ' KB';
       Application.ProcessMessages;
    until FALSE;
end;

froze the desktop/console (Ubuntu 8.04) after a couple of minutes. The same with producing a lot of threads (377 in my case). This took only seconds.

But I was still able to ping the machine in both cases.

Regards, Bernd.

2009/11/19 Mattias Gaertner :

Hmm, I thought the OOM killer will kick in at some stage and terminate the process. Maybe it's not configured into the kernel?

Henry

Probably the system is just struggling to fill a ridiculously large swap partition ;-)

-Fl?vio

En/na Mattias Gaertner ha escrit:

Not my case: just 3 threads (created at the beginning) and memory doesn't grow.

Bye

En/na Henry Vermaak ha escrit:

> Just start hundreds of threads or use a lot of memory. Hmm, I thought the OOM killer will kick in at some stage and terminate the process. Maybe it's not configured into the kernel?

For a while I had the program to simulate the device I communicate with running on the same machine (now it's on a different one, to have a more accurate shapshot of where it hangs), and the first version had a memory leak (a memo that was continuously added to). The OOM killed it, so, yes, it is configured and it is worked.
The program "crashing" the machine doesn't use more memory as time goes by.

Bye

2009/11/20 Luca Olivetti :

Ugh, this sounds like a nightmare to find the problem.

Henry

En/na Luca Olivetti ha escrit:

An update: the program has been running for almost two weeks now (actually 13 days) and I deployed it at the customer's site (crossing fingers).
This threw me into paranoia mode (I made a small modification to another project and I used the original fpc version to avoid this kind of problems), so I'd appreciate if anyone could give me some pointers on how to debug this on a virtual machine (what I found with google involves recompiling the kernel with debug support and using a special pci card to generate a NMI when the kernel gets stuck).

Bye

2009/11/30 Luca Olivetti :

The best way to do this is probably with netconsole. This gets your kernel to spit out the printk messages over the network so you can use netcat on another machine to see the latest messages before the kernel hangs. I don't know if there's a problem with doing this on a virtual machine. Google around or read the kernel docs to see how this works.

Henry

En/na Henry Vermaak ha escrit:

Just speculating at this point, but if it simply hangs (e.g. stuck in a tight loop with interrupts disabled) it will be of no use. With my previous tests on real hardware I could see no messages neither on the console nor on a remote ssh session.
I though that, maybe, using a virtual machine, I could examine the status of the guest, but I could not find anything in virtualbox documentation.

Bye

2009/11/30 Luca Olivetti :

Just had an idea. You can use the magic sysrq key to output a backtrace after it's crashed (you probably have to use this in conjunction with netconsole to be able to save/scroll the output). See http://en.wikipedia.org/wiki/Magic_SysRq_key for more info. Otherwise you're stuck with logging as much as you can.

Henry

En/na Henry Vermaak ha escrit:

Just had an idea. You can use the magic sysrq key to output a

The magis sysrq didn't work (and, yes, it is configured in the kernel and it works before it hangs).

Bye

2009/11/30 Luca Olivetti :

Ow, not good. Well, I guess the only thing you can do is to compile your kernel with debug on and look at the netconsole output.

Henry