Debugging on Mercury

The GNU gdb debugger is available and may be used with serial programs or core files from serial or parallel programs. For information on  using gdb see the online man page and the  gdb user manual. 

The ddd graphical interface for gdb is available on the login nodes. ddd can be used with c, c++, fortran, and perl source code. See the ddd user guide for more information and examples.

The Intel debugger idb is installed along with the Intel compilers.For more information on the Intel debugger, see the idb man page. idb is similar to gdb in operation and it will recognize most gdb commands if started with the -gdb flag. idb works well with C, C++, and Fortran codes.

idb provides a graphical interface when started with the -gui flag [example idb -gui screenshot].

The Totalview debugger [with graphical user interface] works with the supported MPI environment. It is our recommended debugger for MPI code.

Information on starting Totalview on Mercury (and other NCSA platforms) can be found here.

Totalview documentation can be found on the Etnus website here.

The Electric Fence malloc debugger is installed on the login hosts.  It can debug malloc() and pointer related bugs in c/c++ code. Electric Fence does not work simply with the default MPI environment, but it can be used with any of the mpich-tcp MPI environments available via softenv.  Electric fence slows execution and should only be used when debugging.

C/C++ programs using malloc(), calloc(), or realloc() can set the  MALLOC_CHEC_ env. variable.  From "man malloc": If MALLOC_CHECK_ is set to 0, any detected heap corruption is silently ignored; if set to 1, a diagnostic is printed on stderr; if set to 2, abort() is called immediately.

These libraries can be linked with your program to provide runtime checks for common MPI programming problems and MPI deadlock detection.  This debugging aid can scale with your MPI application to the maximum number of processes you can employ.  It's a good option when bugs appear when running at scalle

The MPICH2 MPI environment is installed and supports a text mode gdb interface. This debugging setup has been tested at large scale [> 200 processes] and is the only interactive debugging option available for large scale runs.

The techniques for trapping floating point exceptions [which are not always bugs] vary by compiler and operating system.  Since each case is a little different, see the link at left for the examples that match your situation.  Fortran compilers tend to have a more straightforward approach to floating point exceptions than c compilers.

Splint [for c code] and ftnchek [fortran] are available.  While not true debuggers, source code analysis tools can be very helpful when trying to track down a program bug.  They can also help you write clean maintainable code by providing useful feedback about coding style, unused variables, non-portable practices, ...   Don't be alarmed by the number of warnings generated by these tools, they're designed to detect a great variety of potential problems.

Strace produces a system call trace for any program you can run.

Each line in the trace contains the system call name as used by your program, followed by  its arguments in parentheses and its return value.

If you know what sort of system call may be failing, strace can be quite powerful.

For more information, see:

man strace

ltrace produces a library call trace for any program you can run.

Each line in the trace contains the library call name as used by your program, followed by  its arguments in parentheses and its return value.

If you know what sort of library call may be failing, ltrace can be quite powerful.

For more information, see-

man ltrace

ssh_pbs.pl and ~consult/debug/
gdbwhere.pl can be combined to get a gdb backtrace from processes in a running job.

See also:
ssh_pbs.pl -help ; cat ~consult/debug/
gdbwhere.pl