Hi!
I'm working on a project with an external library (libtcl, it is written in C) which creates some problems with back traces for exceptions.
If a custom TCL command raises an exception, I catch it and use the function DumpExceptionBackTrace to print the back trace. This function uses RaiseList which returns the ExceptObjectStack pointer. The stack frame information is filled by fpc_PushExceptObj in except.inc using the functions get_frame, get_caller_addr and get_caller_frame.
These three functions are also used by dump_stack which can be used at any point in the code (outside of exceptions).
These three functions assume, that the stack frame is stored in the RBP register (x86_64 architecture) and that every function call starts with these two assembler instructions
push %rbp
mov %rsp,%rbp
which leads to the stack layout of the return address (pushed by the "call" instruction) followed by the RBP value.
As this is true for FreePascal code (I guess), this doesn't necessarily hold for external libraries as the TCL library mentioned above.
In my particular case, GDB shows the following back trace after hitting a breakpoint in my custom TCL function implemented in INSERTTRFSMWRAPPER:
#1 0x000000000040eb3c in INSERTTRFSMWRAPPER (this=0x7ffff7fe8340, OBJC=1, OBJV=0x79d080) at trfsmgen.lpr:2552 #2 0x000000000048f86f in CMDCALLER (CLIENTDATA=0x7ffff7fe8cc0, INTERP=0x78f700, OBJC=1, OBJV=0x79d080) at ../pas-tcl/src/tcloop.pas:317 #3 0x00007ffff7aea8f8 in TclEvalObjvInternal (, , , command=0x7ffff7fe9890 'insert_trfsm_wrapper', length=20, ) at /tmp/buildd/tcl8.5-8.5.15/unix/../generic/tclBasic.c:3708 #4 0x00007ffff7aeb85b in TclEvalEx (interp=0x78f700, script=0x7ffff7fe9890 'insert_trfsm_wrapper', numBytes=<optimized out>, flags=<optimized out>, , , outerScript=0x7ffff7fe9890 'insert_trfsm_wrapper') at /tmp/buildd/tcl8.5-8.5.15/unix/../generic/tclBasic.c:4407 #5 0x00007ffff7aeb096 in Tcl_EvalEx (interp=<optimized out>, script=<optimized out>, numBytes=<optimized out>, flags=<optimized out>) at /tmp/buildd/tcl8.5-8.5.15/unix/../generic/tclBasic.c:4064 #6 0x000000000048fe65 in EVAL (this=0x7ffff7fe8380, SCRIPT=0x7ffff7fe9890 'insert_trfsm_wrapper') at ../pas-tcl/src/tcloop.pas:385 #7 0x00000000004db8a2 in EXECUTE (this=0x7ffff7fde1c0, COMMAND=0x7ffff7fe9890 'insert_trfsm_wrapper') at ../pas-tcl/src/tclcmdline.pas:230 #8 0x00000000004da790 in EXECUTE (this=0x7ffff7fde1c0, COMMAND=0x7ffff7fe9890 'insert_trfsm_wrapper') at ../pas-tcl/src/tclcmdlinepredef.pas:184 #9 0x00000000004dbe49 in COMMANDLOOP (this=0x7ffff7fde1c0) at ../pas-readline/src/cmdline.pas:81 #10 0x00000000004daf89 in COMMANDLOOP (this=0x7ffff7fde1c0) at ../pas-tcl/src/tclcmdline.pas:76 #11 0x00000000004923ad in RUN (this=0x7ffff7fe8340) at ../pas-tcl/src/tclapp.pas:63 #12 0x00000000004133ed in main () at trfsmgen.lpr:3319
This list is very reasonable.
TclEvalObjvInternal() is disassembled as follows:
Dump of assembler code for function TclEvalObjvInternal: 0x00007ffff7aea6c0 <+0>: push %r15
0x00007ffff7aea6c2 <+2>: mov %rcx,%r15
0x00007ffff7aea6c5 <+5>: push %r14
0x00007ffff7aea6c7 <+7>: push %r13
0x00007ffff7aea6c9 <+9>: push %r12
0x00007ffff7aea6cb <+11>: mov %esi,%r12d
0x00007ffff7aea6ce <+14>: push %rbp
0x00007ffff7aea6cf <+15>: mov %rdi,%rbp
0x00007ffff7aea6d2 <+18>: push %rbx
0x00007ffff7aea6d3 <+19>: mov %r9d,%ebx
...
RBP is the 5th register pushed to the stack instead of the first one. The function TclEvalEx() looks similar but pushes different registers before RBP.
Now there are two problems:
1) DumpExceptionBackTrace and dump_stack don't show the full stack trace.
2) Both functions dereference values which were taken from the stack and go through a linked list (which is not guaranteed to be working), therefore dereferencing values which might not even be pointers.
Problem 1) is inconvenient but just a cosmetic thing. I had a look at the GDB sources (backtrace_command() in gdb/stack.c and get_current_frame() and get_prev_frame_1() in gdb/frame.c), but especially the latter function is extremely complicated and trying multiple approaches to unwind the stack. I gave up on this. :-(
The more important thing is problem 2) since it potentially creates a segfault. Two counter-measures are already met in dump_stack: the previous frame must be at a lower memory address as the current frame (because "push" decreases RSP), and zero-pointers are used as stop conditions.
I suggest that additionally the RBP values are checked that they are within the program stack address range and divisible by SizeOf(PtrUInt). Further, the caller address (=return address) should be checked that it is within the program code address range. This might get a bit tricky for shared libraries.
For this debugging stuff I've worked on a small routine which parses (and caches) /proc/self/maps to find the above mentioned memory regions. If this function is of interest here, I'll happily provide it (in any license you wish).
Additionally I found UnixGetModuleByAddr in rtl/unix/dl.pp which is used via UnixGetModuleByAddrHook by GetModuleByAddr in rtl/inc/exeinfo.pp. For Windows there are similar functions.
To summarize: Stack dumps currently are a bit dangerous if using different stack frame conventions as assumed.
Bye
Hansi