Hello,

I am using Lazarus 1.2.0. On Windows 2000 I am able to debug an application via the remote debugger (gdbserver) on an ARM-LINUX target. Using the same setup does not work reliable on Windows 7.

If I step through the application, at some point, the grey bar, which marks the current source line in the IDE disappears and no further stepping is possible. I can see, that the target has executed the last line correctly and seems to wait for the next command from IDE/debugger.

Looking at the last lines from the debug output, it seems, as if the IDE misses the answer from the debugger, when the target has stopped:

<-exec-next>
^running
*running,thread-id="all"
(gdb)

The message *stopped,reason=... is missing.

Remote debugging the application without Lazarus is working without problems.

Any help would be greatly appreciated.

Regards, Bernd.

Please run with a logfile:
http://wiki.lazarus.freepascal.org/GDB_Debugger_Tips#Log_info_for_debug_session

And then try, what happens if you hit the pause button in the IDE?

Martin Frb wrote

Please run with a logfile: http://wiki.lazarus.freepascal.org/GDB_Debugger_Tips#Log_info_for_debug_session

I have the log file attached.

And then try, what happens if you hit the pause button in the IDE?

I stepped through the program until the problem occurred and hit then pause. As far as I could see, nothing happened. Then I hit stop. This was recognized by the target with the message, that the remote side had terminated the connection.

Regards, Bernd.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: log_file.zip
Type: application/x-zip-compressed
Size: 7046 bytes
Desc: not available
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20140409/18b6914c/attachment.bin>

What code is on this line?

Stepping will only work, if that code actually reaches the next line. E.g.:

    while (always_true) do inc(a);

Same
   Call_Foo_that_will_go_into_an_endless_loop();

would never reach the next line. Stepping over it, is like Run. It will keep running.

Normally you can hit pause. But In remote-debugging pause only works if gdb is in "async" mode.

Your gdb did accept async for run(f9). Note the "&"

> TCmdLineDebugger.SendCmdLn "-exec-continue &"

<< TCmdLineDebugger.ReadLn "^running"
<< TCmdLineDebugger.ReadLn "*running,thread-id="1"" << TCmdLineDebugger.ReadLn "(gdb) "

But it did not for stepping

> TCmdLineDebugger.SendCmdLn "-exec-next &"

<< TCmdLineDebugger.ReadLn "^error,msg="A syntax error in expression, near `&&'.""

Somehow strange, but that is what it did.

So the IDE used none async to do the step. But that means it can not interrupt (pause/stop) the remote, until the remote stops on its own (breakpoint/end of step)

The error might be bound to the "mi" made of gdb. That is "n &" might work. But it will take some work to make the IDE deal with none "mi" stepping

mi is a interface designed to control gdb from another app / gdb uses structured answers, instead of human readable. Human readable answers can differ in different gdb, and a harder to deal with by the IDE.

--------------------------
You could try, if you are on that line, instead of stepping (F8), set a breakpoint the next line (lines, if there is a conditional), and use F9.

Then, with F9 you should be able to use the pause button

So the remote printed connection lost, when you pressed stop in the IDE. Then it was still connected?

Martin Frb wrote

> > I have the log file attached. What code is on this line?

This is not always the same line when the problem appears.

    repeat
       Writeln('Test');
       Sleep(500);
    until FALSE;

It can happen, that I press F8, when the grey bar is on the line with the Writeln statement and the debugger does not come back to the next line with the Sleep statement.

I made another simple test case, which fails on Windows 7 (but works on Windows 2000):

program project2;
begin
    Writeln('Test');
    Writeln('Test');
    Writeln('Test');
    Writeln('Test');
    Writeln('Test');
    Writeln('Test');
    Writeln('Test');
    Writeln('Test');
    Writeln('Test');
    Writeln('Test');
    Writeln('Test');
    Writeln('Test');
end.

On Windows 7, I am not able to step until the end of the program. At some time, when I step over with F8, the grey selection bar does not come back to the next Writeln statement. I can see, that the target has stopped, because there is no further output in the console.

[valuable information snipped]

yes, pause seems to work.

So the remote printed connection lost, when you pressed stop in the IDE. Then it was still connected?

The exact message from the target is:

"Remote side has terminated connection. GDBServer will reopen connection. Listening on port 3333"

Regards, Bernd.

Well from the log, there is nothing wrong in the IDE <-> gdb communication. So it is some issue in GDB itself.

As for trying to find a workaround:

If gdb server can detect the connection loss (at "stop"), then it was still connected. So gdb has failed on a different level. I assume that the gdbserver on the arm target is the same, when you try from win7, as it is when you try from win2000?

Maybe something with the AV or firewall on win7?
Maybe trying a different version of gdb on win7?

gdbserver seems to have some options to track problems, maybe try them?

Options: --debug Enable general debugging output. --remote-debug Enable remote protocol debugging output.

You can also try the following.

1.3: components\lazdebuggergdbmi\gdbmidebugger.pp (line 877) 1.2: debugger\gdbmidebugger.pp (line ??)

defines a "const"
GDBMIExecCommandMap: array [TGDBMIExecCommandType] of string =

containing
'-exec-next', // ectStepOver,

replace it by
'n', // ectStepOver,

Rebuild the IDE. Then in the log, watch out for

> TCmdLineDebugger.SendCmdLn "n &"

If the next line is an error (like)
<< TCmdLineDebugger.ReadLn "^error,msg="A syntax error in expression, near `&&'.""

Then it does NOT help.
If it is followed by
<< TCmdLineDebugger.ReadLn "^running"

then wait till the error happens, and see what happens if you press pause.

----------
Though, yes, it will not stop the error from happening. But it may allow pause to work, if the error happens.

Fixing the error, can probably only be done in gdb

Bernd Mueller wrote

I stepped through the program until the problem occurred and hit then pause. As far as I could see, nothing happened. Then I hit stop.

I once was told that with using gdbserver, "Pause" (while not hitting a breakpoint) is not possible.

That is why I found that using ssh to remote-control "normal" gdb on the target (instead of locally controlling gdb and having it connect to gdbserver on the target) would be more appropriate.

Moreover I found that, with an embedded target, finding or creating a normal gdb might be a lot easier that finding or creating the "cross gdb" (running on PC but understanding the Target's files) plus the gdbserver for the target.

I found that - at that time - the code for ssh'ing gdb was present in the Lazarus source but was commented out (supposedly because it's not yet ready for use).

-Michael

That changed. But only where gdb supports async mode. Back then the IDE did not support using gdb in async mode. Now it does.

But, as the logfile showed, the gdb/gdbserver used, supported async for run(f9), but not step(F8).

The latter, may be due to the use of the mi syntax. Which is why I wrote changing it to "n" instead of -exec-next

That is why I found that using ssh to remote-control "normal" gdb on the target (instead of locally controlling gdb and having it connect to gdbserver on the target) would be more appropriate.

That may work outside the IDE, if you can either send a "break" (ctrl-c) to gdb, or have a 2nd ssh connection, and can do "kill SIGINT pid_of_yourapp".

The IDE ssh debugger, can not pause the app either.

See above. It does not pause the remote.

To pause an app in the debugger, the IDE sends a SigInt. But it can only do that for local targets.

The gdbserver debugger in the IDE, can tell gdb to interupt the target, (again) if gdb is in async mode.
In async mode, gdb will accept commads while the target is running. So you can do

r & // run async (background)
interrupt // pause

Martin Frb wrote

If gdb server can detect the connection loss (at "stop"), then it was still connected. So gdb has failed on a different level. I assume that the gdbserver on the arm target is the same, when you try from win7, as it is when you try from win2000?

yes, I have only one target. So I am working always with the same gdbserver.

Maybe something with the AV or firewall on win7?

Not impossible. But I doubt it.

Maybe trying a different version of gdb on win7?

I have done this already.

gdbserver seems to have some options to track problems, maybe try them? > Options: > --debug Enable general debugging output.

This is very strange. If I activate --debug, then everything seems to work. As if slowing down the target (by printing all the debug messages in its console) is helping somehow with the communication.

You can also try the following. Rebuild the IDE. Then in the log, watch out for >> TCmdLineDebugger.SendCmdLn "n &"

This is the log:

> TCmdLineDebugger.SendCmdLn "n &"

<< TCmdLineDebugger.ReadLn "&"n &\n""
<< TCmdLineDebugger.ReadLn "^running"

then wait till the error happens, and see what happens if you press pause.

Then the assembler window pops up. But I have the impression, that IDE/Debugger and target are not in sync any more. But maybe, I am doing something wrong. I have to check this in detail.

Regards, Bernd.

Odd, that makes it even more look like a gdb bug.

Though, I am not sure it is communication, it might be timing on the local interaction between gdb and the debuggee.

It is well possible that the connection with the win2000 also affects speed, and that gdbserver may hit a race condition. (gdb has one on win32 where it fails to step, because it interrupts the target, before the target fully resumed / well something roughly like that). It is possible that the arm gdbselver has is own race condition. And that the connection speed with win7 increases the likelyhood.

But of course that is just a guess.

Yes, that means it runs async (so the gdb mi implementation is simple incomplete. Something that affects the local gdb, but not gdb server.)

Check the thread window, and stack for each thread.

But as I said above it is possible that gdbserver screwed up, and that your target ended up in an unusable state.

I do no think (but again its all guessing), tha the communication is broken.

Martin Frb wrote

The IDE ssh debugger, can not pause the app either.

I don't understand why this should be the case.

Manually using gdb via ssh (to bash) can do it, So why should the IDE not be able to ? (Or does - when in non-remote mode - the IDE really send a SigInt *directly* to the application and not via the debugger )

-Michael

Well how do you do it, if you do it manually? Ctrl-C ?

I have not spent to much time on it, but if you can make the IDE send this to the remote then cool.

Btw, IIRC I did try, and sending #$03 does not do it. It is not the ctrl-c character that needs to be sent, but the break that a terminal emulation generate upon receiving the key combo.

Or do you have another way?

-----
Oh btw: IIRC gdb over ssh can do it in the IDE, if and only if gdb supports async (need to double check). gdbserver supports that for almost all targets. gdb only for a selected few.

Hum.
Re-Thinking:

When manually using gdb (directly on the target) can't you Pause the application under debug ? If yes, the IDE should be able to do this via SSH, as well.

-Michael

Any comment on this ?

IMHO the by for most needed application for remote debugging is PC (Linux or Windows) ---> ARM (Linux).

Will there once be some "out of thge box" support for this (providing gdbm gdbserver and documentation) ?

(In fact - as you might remember - a pal from Austria and myself did not get this running about y year ago.)

Thanks,
-Michael

Yes the IDE, does send a SigInh directly to the target. Not via the debugger.

The debugger does (except for async mode) not accept any commands.

Using gdb on a terminal, ctrl-c generates the Signal (and afaik it goes through to the debuggee). But in the IDE, the gdb is run without terminal. and that does not work. If you find a way, well patches welcome.

Martin Frb wrote

Or do you have another way?

A wild Idea might be to open two SSH sessions: one for gdb (maybe in sync mode of this makes sense), and another that just accesses a little SIGINTServer application that is used to send the signal when appropriate..

-Michael

I will work on improving the IDE gdb/gdbserver support, where I can (and dependent on feedback).

I do not plan, to try and build, or searh for gdb/gdbservers. Nor do I have (or plan on having) any arm (real or simulated) environment.

So I will not add those gdb/gdbserver myself.
I do not know, if anyone else wants to work on that.

Also which installer should they be added to? The only installer which does arm, is the wince. (and that may also need some maintenance, not sure, if it comes with gdb out of the box)

That would work. (and I would accept a patch (adding an option / not making it mandatory)

But at this stage, I prefer the async solution.
Did you test, if the gdb (without gdbserver) you intend to use supports async?

If it does, then that should do the trick (or can be fixed to do it). It may need the "n" instead of -exec-next, change. I may commit that, when I have the time, to make a clean implementation.

Martin Frb wrote

That would work. (and I would accept a patch (adding an option / not making it mandatory)

Sounds nice to me, especially regarding that gdbserver is not that "common" and hence not easily to come by in some cases.

But at this stage, I prefer the async solution.

I see.

Did you test, if the gdb (without gdbserver) you intend to use supports async?

In fact we have not been able to find/create a remote-gdb gdbserver pair with an effort low enough to recommend to anybody.

-Michael

If I use the standalone gdb on the Win7 Host instead of Lazarus, debugging on the remote target works correct.

I had a look into the network traffic between target and host while debugging with Lazarus. If the stepping fails, one can see that the target is sending its breakpoint replay packet "+$Txxxx" which seems to be ignored completely by the host.

If the stepping does not fail, the host acknowledges this packet with "+" and starts to read memory from the target via the "$mxxxx" Packet.

I am trying to go more into detail...

Regards, Bernd.

async may in some cases work with just gdb, no gdbserver needed. But I do not know, if it will on the arm target you look at.

I just checked. I added IDE support. So using gdb over SSH with Laz 1.2 it will try async.

Look at the log, if you see the -exec-... & and the work then all is ok.

There are several possibilities, that I can think off:

1) some setting the IDE uses.
- Gather all the "set foo=" commands that the IDE sends, and see if applying them to your manual driven gdb session makes a different.
- Run your manual gdb with "gdb.exe -i mi " to enable mi mode (I do not think that is it, but ....)

2) Timing (again, and not unlikely)
If you open gdb in a terminal, it prints to that terminal. In the IDE it prints to a pipe. That may take different amount off time.

Also, does your sniffer give the time, between the "step over" sent to the target, and the packet that comes pack, and is ignored? compared to when it works?

Martin Frb wrote

Look at the log, if you see the -exec-... & and the work then all is ok.

I have the ARM machine in question (a "topless" QNAP NAS) at home and will try to make sense of this during the weekend.

BTW.:
In fact the helpful person from Austria did start to write a documentation on how to install and use cross compiling and remote debugging towards an embedded ARM target. I promised to translate this to English. when one and verified.

But he was only able to create the "cross" gdb / gdbserver pair after installing the complete multi Gigabyte partly propriety "CodeSourcery" cross environment (including Eclipse) and so we decided that documenting _this_ does not make sense.

-Michael

Martin Frb wrote

So I will not add those gdb/gdbserver myself. I do not know, if anyone else wants to work on that.

I did try (see the other mail).

Also which installer should they be added to?

I suppose you mean installer for Lazarus ?

The only installer which does arm, is the wince. (and that may also need some maintenance, not sure, if it comes with gdb out of the box)

Why that ?

ARM Linux is a very viable target today. Very famous hardware like Raspberry Pi and (IMHO more appropriate) BeagleBone (black) can run Lazarus native, as they do have rather decent video hardware. I suppose the same installer would be able to support both. (I don't know anything about how installers support different Linux distributions.) A friend of mine successfully runs Lazarus on Debian on BeagleBone.

For headless devices it would be possible to do Lazarus native with a remote GUI tool (VNC / NoMachine / X via TCP/IP). I don't know if this is really appropriate.

A "cross" distribution supposedly would need a dedicated installer (like that for WinCE). I could imagine that a Windows--->ARM-Linux and a PC-Linux--->ARM-Linux both could be viable. Same would need contain the cross fpc, the appropriate "cross" gdb and gdbserver pair (or just the standard gdb for ARM when using the "dual SSH" way).

-Michael

Actually I do not use the linux installers. So I dont know anything about them. Just form the look at Sourceforge I concluded they did not seem to have an arm target.

thank you, for pointing this out.

Also, does your sniffer give the time, between the "step over" sent to the target, and the packet that comes pack, and is ignored? compared to when it works?

In both cases, it shows nearly identically times. It takes about 1,9 ms until the breakpoint reply packet has reached the host. Nearly the same on Win2000.

If I activate gdbservers debug log, then it takes about 5,2 ms.

Regards, Bernd.

Strange.

But this goes beyond my gdb knowledge. It may be something to ask on the gdb list.

I have made changes (trunk revision 44763), so that both remote debuggers(gdbserver/ssh) will fallback to use none mi command, if mi commands fail with "&"
The target still needs to support the none mi "&".

There is a new option as well: UseNoneMiRunCommands It can be
- Always: the debugger will use none MI always (even if not in async): "next"
- Never: As it used to be. Only use MI commands: "-exec-next"
- FallBack (DEFAULT): In none Async mode will always use MI. In async mode the debugger will try MI, but then fallback to none MI

remote debugger always try to run in async (depends on the result of "set target async")
none remote debugger has an option (and by default does not try)