Unable to debug core file on RHEL7

[rjpontefract]

Using SE 27.0.0.1 64-bit Qt5 on Redhat Enterprise Linux 7.9.

I have the two standard debug configurations, Standard GDB and SlickEdit GDB.  Standard GDB is the default.  I am able to attach to processes and debug successfully and I am able to debug executables OK.  I cannot debug a core file using the Debug Executable>Analyse Core File menu option, SE reports that GDB ended prematurely.  I enabled debug logging in SE and that just showed that GDB terminates.

The debug core file dialog gives the option to choose the debugger configuration and Standard GDB is selected (seems to be the only option) which is odd.

I took a look at the GDB supplied with SE and it cannot find the following shared libraries, which explains why it terminated:

Code: [Select]

/opt/slickedit-beta/bin/gdb: /usr/lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /opt/slickedit-beta/bin/gdb.se)
/opt/slickedit-beta/bin/gdb: /usr/lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /opt/slickedit-beta/bin/gdb.se)
/opt/slickedit-beta/bin/gdb: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /opt/slickedit-beta/bin/gdb.se)
/opt/slickedit-beta/bin/gdb: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.22' not found (required by /opt/slickedit-beta/bin/gdb.se)

But why is SE using that version anyway as I selected standard GDB?  The dialog for attaching to a process doesn't have an option to choose a debugger and that uses the correct system supplied GDB in /usr/bin/gdb.

I worked around this by symlinking /usr/bin/gdb to /opt/slickedit-beta/bin/gdb which seems to work OK, but it'd be good to know what's going on with SE.

[Dennis]

The GDB that is shipped with SlickEdit also requires /opt/slickedit-beta/bin to be in the LD_LIBRARY_PATH.  When SlickEdit launches GDB, it sets that up.  There is also a "gdb.sh" script shipped in the bin directory which sets up LD_LIBRARY_PATH.  I suspect there is another issue causing GDB not to launch.

[Dennis]

I was able to reproduce the issue with it starting the wrong GDB configuration.  This will be fixed in Beta2.

I tested the GDB which we ship with SlickEdit on CentOS 7.7 (the closest thing I had to your platform), and it ran correctly.  But I may have some packages installed that you do not have.  Let me know what you get when you run bin/gdb.sh.  I also make a tweak to gdb.sh to make the Python setup a bit more bulletproof, but that should not have prevented GDB from starting.

[rjpontefract]

Hi Dennis
I ran gdb.sh and the SE version of gdb started correctly, apologies for that misunderstanding.
However, SE still fails to debug a core file.  The debug.log file contains the following

Code: [Select]

debug[2022-08-23T10:22:21Z]GDB starting gdb: /opt/slickedit-beta/bin/gdb
debug[2022-08-23T10:22:21Z]GDB startup arg[0]=/opt/slickedit-beta/bin/gdb
debug[2022-08-23T10:22:21Z]GDB startup arg[1]=-nx
debug[2022-08-23T10:22:21Z]GDB startup arg[2]=-interpreter=mi
debug[2022-08-23T10:22:21Z]GDB startup arg[3]=-quiet
debug[2022-08-23T10:22:21Z]GDB startup arg[4]=-data-directory=/opt/slickedit-beta/toolconfig/vsdebug/
debug[2022-08-23T10:22:21Z]GDB connect, pid=5717
debug[2022-08-23T10:22:21Z]GDB connect, alive=1
debug[2022-08-23T10:22:21Z]GDB setting PYTHONPATH: /opt/slickedit-beta/toolconfig/python/python3.10
debug[2022-08-23T10:22:21Z]======================================
debug[2022-08-23T10:22:21Z]  GDB command_reply: command=-list-features
debug[2022-08-23T10:22:21Z]  GDB command_reply: timeout=0
debug[2022-08-23T10:22:21Z]      GDB event: error reading next event -5702
debug[2022-08-23T10:22:21Z]GDB command_reply: error reading events, status=-5702
debug[2022-08-23T10:22:21Z]GDB event parser: Reply did not begin with ^ :
debug[2022-08-23T10:22:21Z]======================================
debug[2022-08-23T10:22:21Z]  GDB command_reply: command=-gdb-set auto-load off
debug[2022-08-23T10:22:21Z]  GDB command_reply: timeout=0
debug[2022-08-23T10:22:21Z]      GDB event: error reading next event -5702
debug[2022-08-23T10:22:21Z]GDB command_reply: error reading events, status=-5702
debug[2022-08-23T10:22:21Z]======================================
debug[2022-08-23T10:22:21Z]  GDB command_reply: command=-environment-path -r ../../output/Linux64/exe/ /usr/lib64/qt-3.3/bin /usr/local/bin /usr/local/sbin /usr/bin /usr/sbin /bin /sbin
debug[2022-08-23T10:22:21Z]  GDB command_reply: timeout=0
debug[2022-08-23T10:22:21Z]      GDB event: error reading next event -5702
debug[2022-08-23T10:22:21Z]GDB command_reply: error reading events, status=-5702
debug[2022-08-23T10:22:21Z]======================================
debug[2022-08-23T10:22:21Z]  GDB command_reply: command=-file-exec-and-symbols ../../output/Linux64/exe/testexe
debug[2022-08-23T10:22:21Z]  GDB command_reply: timeout=0
debug[2022-08-23T10:22:21Z]      GDB event: error reading next event -5702
debug[2022-08-23T10:22:21Z]GDB command_reply: error reading events, status=-5702
Good to know that there'll be a fix for using the wrong configuration.

[Dennis]

Did you put back the original SlickEdit GDB before running gdb.sh ?

[rjpontefract]

Yes, I did put the SE version back.  I've tried it with the system gdb and it doesn't start with gdb.sh due to the following error:

Code: [Select]

GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-120.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
(gdb) exit
rjp:centos7$ ./gdb.sh
  File "/opt/slickedit-beta/bin/../toolconfig/python/python3.10/site.py", line 99
    print(message, file=sys.stderr)
                       ^
SyntaxError: invalid syntax


[Dennis]

Looking at the debugger log, -5702 is the error code when GDB terminates for an unknown reason.  Most likely, in the environment GDB was started in, it had a problem with some shared library.  Hard to tell, I don't know where those error messages went.

It's good that you have a workaround (using the system GDB).

Did you have the same issues on this machine with 26.0.3 ?

[rjpontefract]

Hi Dennis

I do see the same behaviour when using 26.0.3.

I did some investigation and renamed the gdb executable in slickedit-beta/bin to gdb.se.  I then created a shell script called gdb that echoes PATH and LD_LIBRARY_PATH before executing gdb.  The result was that both paths were unaltered from my usual environment, which explains why gdb terminates.

So I renamed the SE provided gdb.sh to gdb and changed the last line to execute gdb.se rather than gdb and that makes it work as intended.

It seems that SE is running gdb directly not gdb.sh and hence not getting the desired values in LD_LIBRARY_PATH.  Of course, I have no idea how SE launches gdb, so I could be wrong about that.  I'd be interested to hear what you think.

[Dennis]

When SlickEdit launches GDB, it adds LD_LIBRARY_PATH to the environment before forking.

[rjpontefract]

Hi Dennis

It doesn't seem to do that on my system.  I replaced the gdb executable with a shell script that echoes the environment and then sleeps for a while.  There's no mention in the output of the SE library paths, nor does they appear in /proc/<pid>/environment.

I ran strace with -f -v on the vs executable and found the called to clone() followed by execve().  The call to execve() where it executes gdb shows the existing LD_LIBRARY_PATH from the parent process with no signs of any additions or alterations.

[Dennis]

I downloaded a copy of RHEL 7.9 and have reproduced the issue.  I'll see if I can find a fix for beta2.

[Dennis]

Found the problem.  Our build process for GDB tries to set -rpath to \$ORIGIN, but because GDB devs changed things in the newer releases (I think it uses cmake now), those flags do not get through.  Repairing the GDB executable rpath fixed things (mostly).  I'm going to add that step to the script to build GDB.

Code: [Select]

   patchelf --set-rpath "\$ORIGIN" gdb


[rjpontefract]

Hi Dennis
I'm happy you found the problem, I was beginning to think it was something specific to my environment.