I have a pretty sizable console app written with Delphi 15 years ago but ported to Linux using FreePascal (3.2.0) with Lazarus (2.0.12) as IDE. It runs as a systemd service on a Raspberry Pi3.

Basically it is a scheduler, which checks every minute if there is a task to run, otherwise it waits for the next minute to pass.

Meanwhile in another thread there is a TCP/IP socket server active for communicating with the app over the network. So it is listening for incoming connections.

This is working seemingly OK, but today when I checked the RPi I found using top that it was running 11% CPU, which is strange because it has nothing to do at the moment.

I have tried to be as conservative as possible regarding wait loops etc so in such loops I always have a sleep() call, which in my Windows experience used to stop excessive CPU usage.

So I was surprised to find the high CPU usage and now I am at a loss on how to find *where* this is happening...

Any ideas on how to proceed?
Is there some Lazarus way to find this?
(But I cannot really run the application in service mode from within Lazarus...)

This is what top displays:

top - 15:34:13 up 10 days, 12:46, 2 users, load average: 0.10, 0.04, 0.14 Tasks: 142 total, 1 running, 141 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.6 us, 0.6 sy, 0.0 ni, 98.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st MiB Mem : 924.2 total, 279.1 free, 523.7 used, 121.4 buff/cache MiB Swap: 2048.0 total, 1495.2 free, 552.8 used. 340.2 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 15283 pi 20 0 51764 7948 6260 S 6.3 0.8 3:00.16 SSRemoteServerL
26357 pi 20 0 10436 2968 2516 R 1.0 0.3 0:01.66 top

PID 15283 is my service application.
It now runs at about 6% all the time, but when I discovered the problem it was % CPU.
After I restarted the service it dropped to 6% or thereabouts.

I'm not positive, but I'm pretty sure the sleep command in linux does not behave the same way it does in windows.

As you know, in windows, a sleep command (even if delivered with a parameter of 0) gives up time slices to other programs on the system.  This does not appear to be the case on linux.

On linux, the sleep command simply suspends the process for the specified amount of time, but so far as I can tell, does nothing for unused cpu cycles.

I've done a little digging, but I can't find any way on linux to give away unused cpu cycles.

Perhaps the linux task switcher doesn't allow for this capability?

if it's waiting on keyboard input you might be better off using select() for that instead of looping and checking for keyboard input each go around.

sleep() might already do something similar with a zero timeout but even that would probably be insufficient

As I tried to explain at the start of this thread my application runs as a *systemd service* on Linux, so there is absolutely no keyboard input.

The main program looks like this after initializations etc:

  try
    try
      bSTerm := False;
      bSInt := False;
      bsHup := False;
      {$IFDEF UNIX}
      fpSignal(SigTerm, SignalHandler(@handleSignal));
      fpSignal(SigInt, SignalHandler(@handleSignal));
      fpSignal(SigHup, SignalHandler(@handleSignal));
      {$ENDIF}
      Debug_Writeln('Enter eternal loop');
      FLogServ.StdLog('Enter eternal loop');
      While not (bSTerm or bSInt or bsHup) do
      begin
        //Here is where the server runs as defined elsewhere
        //Eternal loop to wait for system messages
        Sleep(1); //To not hog the CPU
        CheckSynchronize; //To get thread comm working
      end;
      Debug_Writeln('Exit eternal loop');
      FLogServ.StdLog('Exit eternal loop');
    except
      on E: Exception do
      begin
        FLogServ.ExceptionLog('Unhandled exception: ' + E.Message);
      end;
    end;
  finally
    ShutdownServer;
  end;

As you can see it sits in the eternal loop until the system sends a "signal" to it to stop. So systemd does that if I issue a sudo systemctl stop myservice and this works fine.
Could this tiny loop consume 6-7% CPU cycles?

Apparently, you can release cpu cycles, but it's with the sched_yield (section 2 in the man pages), not the sleep command on linux.

So, apparently, the sleep is still using full system recourses, even though it's just waiting.  You apparently have to release cpu time with the sched_yield command to make it actually give up time slices., and even then, it's not clear it will actually do so, the description claims it simply moves your application to the end of the cue, which doesn't really give away it's cpu cycles, just changes it's location in the execution order.

Perhaps it will help with the cpu usage though, I've not done any testing to see how (or not) it affects cpu.

https://www.freepascal.org/docs-html/rtl/baseunix/fpsigtimedwait.html

there we go :) i think this should do the trick. might need to rearrange some things though..

sleep 1 is not that much. How does it change if you increase it to sleep(50) / just for testing?

Also CheckSyncronize afaik takes a timeout.
So if you do not need to check the variables every millisecond, then do

       While not (bSTerm or bSInt or bsHup) do
       begin
         //Here is where the server runs as defined elsewhere
         //Eternal loop to wait for system messages
         CheckSynchronize(50); //To get thread comm working
       end;

--------
For all, else you can try
valgrind --tool callgrind
and kcachegrind to view the result.

but not sure if that works on a service, maybe you can run your code in a foreground process under valgrind?

Bo Berglund wrote

Could this tiny loop consume 6-7% CPU cycles?

Thanks for all of the responses, I will test tomorrow.

I have noted that there are no other service processes that are waiting for some action but show up with CPU cycles above 0 in the top display... So there is obviously something that can be done, possibly not by free pascal though...

Most other services are written in other languages, I assume.

Example for the two openvpn service processes I have on another RPi:

top - 01:57:13 up 91 days, 11:04, 2 users, load average: 0.04, 0.04, 0.00 Tasks: 2 total, 0 running, 2 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.1 us, 0.0 sy, 0.0 ni, 99.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 948304 total, 45656 free, 506952 used, 395696 buff/cache KiB Swap: 102396 total, 86780 free, 15616 used. 357904 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 458 openvpn 20 0 8588 5180 4508 S 0.0 0.5 5:47.97 openvpn 438 openvpn 20 0 8588 5328 4568 S 0.0 0.6 5:45.86 openvpn

Op 2021-05-18 om 23:29 schreef Travis Siegel via fpc-pascal:

This is what "suspend" means, that process doesn't compete for cycles, so they go to the other processes. Since the sleep is a kernel call, I can't really imagine why this would not be the case.

So my guess something else is spinning (e.g. in a thread).

Apparently, you can release cpu cycles, but it's with the sched_yield (section 2 in the man pages), not the sleep command on linux.

What sleep command are you referring to?
What do you mean with cpu cycles?

Sleep works pretty well under Linux:

uses sysutils;
var i: integer;
begin
  for i:=1 to 10000 do sleep(1);
end.

time ./test1

real 0m10,791s
user 0m0,021s
sys 0m0,018s

Mattias

So I tested by changing the loop to:

  While not (bSTerm or bSInt or bsHup) do
  begin
    //Here is where the server runs as defined elsewhere
    //Eternal loop to wait for system msg bSTerm, bSInt or bsHup
    CheckSynchronize(5); //To get thread comm working
  end;

It still worked as before so I can get rid of the sleep() but it still runs at CPU = 6% while idling.

I guess this "proves" that the main loop is not what is causing this CPU usage.

Next I will search all of the code for instances of sleep() and see if there are other CPU "thieves" around.

No doubt your sleep code works just fine.

I'm not saying the sleep command doesn't work.

I'm saying the sleep command doesn't release unused cpu cycles for other threads/programs to use.

Apparently, if you want that behavior, you need to yield the cpu time your process would otherwise take, that's done with a different kernel function than sleep.

Alexander hit the nail on the head though with his solution, so I'm satisfied that the original poster got what he needed, and I learned something new about linux processes as well, which makes for a good all around solution.

No, fpc uses nanosleep() inside sysutils.sleep() which is documented to suspend execution (i.e. no busy waiting) so the kernel will switch to another thread/process.

Henry

Marco van de Voort wrote

So my guess something else is spinning (e.g. in a thread).

Seems likely based on my test with giving ChecSynchronize an argument 5 and not noticing any difference...

So the original conundrum still applies:

** How to find where my app consumes CPU? **

The "other" items might be hidden inside other used classes such as the Indy10 components I use to implement the TCP/IP communications. But when searching for sleep through the complete project sources I came up empty-handed.
Something else must be going on.

I have a thread to handle the measuring sequence server and this uses timers (TFpTimer) in order to check if it is time to run a task, but when idling no task is running so no task execution thread spins off either...

That's useful information.

I did run across nanosleep in my digging, but I wasn't aware the sleep command called it.

I'm still a (little) puzzled though, why there are so many different suspend/sleep modes in linux kernel implementations. They appear to do similar things, so perhaps it's just incremental differences, depending on what's trying to be accomplished, or maybe, under the hood, they all call the same routines, I can't really say, but it sure has been educational digging around to find answers.

So now I am down to the timers...
I am using TFPTimer timers in the scheduler to handle various things, some of them are just one-shots to delay an action for some predetarmined time. These are only executing as one-shots.
But the schedule timer restarts itself with an interval of 60 s. Another timer is used to handle a message queue between parts of the system. It runs at a shorter time, like a second or so. Restarts itself too. But it runs only if scheduling is active.

I have no idea how TFpTimers work internally, maybe these cause extra CPU cycles while waiting for the time they should fire?
I could switch off the scheduling timer, but it made no difference. The other timers running for other purposes I canot disable. An these fire more often.

Right now the server is running at 10% when doing nothing, not even checking schedules...
After I did a service restart it drops to about 5%

Even it’s a service, running it with gprof enabled should be possible, no?

Bo Berglund wrote

I have a pretty sizable console app written with Delphi 15 years ago but ported to Linux using FreePascal (3.2.0) with Lazarus (2.0.12) as IDE.

I have the same problem with high CPU usage for a 20-year-old Delphi App compiled in Lazarus(2.0)

When run in debug mode within the Lazarus IDE it runs fine.

When run stand-alone as a Windows exe the CPU usage shoots up to 30% and stays there.

I found that the problem is in the Lazarus Application.OnIdle event, which fails to honour the Done parameter and runs continuously:

http://docwiki.embarcadero.com/Libraries/Sydney/en/Vcl.Forms.TApplication.OnIdle

Martin.

Florian Klämpfl wrote

Even it’s a service, running it with gprof enabled should be possible, no?

I do not know what you mean, but the program is installed as a service but it is essentially a normal console application with no GUI parts. The reason it is run as a service is that it must run 24/7 on the target machine so this is how I am testing it too. But initially I always tested it by starting it in a terminal (actually a PuTTY SSH session from my Windows PC).

Googling gave me this:
https://www.freepascal.org/docs-html/user/userse56.html and this: https://wiki.freepascal.org/Profiling

I have checked that gprof is found on path on the development machine, but I have never ever heard of it let alone use it.
But if it is able to find out where the cycles are spent, then very good!

Seems like there is a bit of read-up to do...

Thanks for the suggestion, btw!

The link you posted shows a page where Forms are used. My app does not use Forms.

Is your application a GUI type app?

Mine is specifically ported to be a non-GUI console app so there are no GUI-realted items in there like Form dependencies. For example Application.Processmessages is totally non-existent.

I had a lot of trouble removing all of the GUI-related stuff from the Delphi code when porting. Had to rewrite parts of the underlying system to get it done.

[...] I found that the problem is in the Lazarus Application.OnIdle event, which fails to honour the Done parameter and runs continuously:

How do you know, it is a bug in OnIdle, and not something continuously emitting events?
Have you tested in a small test application?

Mattias

So now I am down to the timers... I am using TFPTimer timers in the scheduler to handle various things, some of them are just one-shots to delay an action for some predetarmined time.

I don't know how TFpTimers work either, but when I want a timer in one of my console apps, I just do it by comparing a variable to "Now" like this:

Var
MyTimerVariable, MyTimerDelay : TDateTime;
//Start Timer
MyTimerVariable := Now + MyTimerDelay;
Then in my main loop where I also have a Sleep (), I just do:
If Now >= MyTimerVariable Then
   Do_Whatever;

I can't imagine why you would need anything more complicated than that for a general timer. As I recall "Now" has a quite small resolution, much less than milliseconds. This is pretty much guaranteed to not use any system resources other than the compare at whatever frequency the main loop runs at.. in fact I regulate the frequency of the main loop in a similar way.. At the end of the main loop I have a tight loop with Sleep and the comparison to Now and when I'm past the minimum time I want for the main loop, I stop doing the loop:

Repeat
   Sleep(1);
Until Now >= MyTimerVariable;

James