This article aims to help debugging software by providing traces and debug information. This information can then be used for the bug report to the (upstream) software developers or package maintainers.

Usually, executable files are stripped of human readable context to make them smaller. Obtaining traces without debugging information available drastically reduces their usefulness. For example, a trace from agdb session where debugging information is not available may look as follows:

[...]Backtrace was generated from '/usr/bin/epiphany'(no debugging symbols found)Using host libthread_db library "/lib/libthread_db.so.1".(no debugging symbols found)[Thread debugging using libthread_db enabled][New Thread -1241265952 (LWP 12630)](no debugging symbols found)0xb7f25410 in __kernel_vsyscall ()#0  0xb7f25410 in __kernel_vsyscall ()#1  0xb741b45b in?? () from /lib/libpthread.so.0[...]

?? shows where debugging info is missing, as well as the name of library or executable which called the function. Similarly, when(no debugging symbols found) appears, you should look for the stated file names.

To obtain a proper trace that is useful to the program developers, follow the next sections. Separate debug files are available for most official Arch packages and can be downloaded withDebuginfod (see#Getting the trace). When enhanced debugging information was not added to the executable in the first place, one has torebuild the package with debugging symbols enabled.

Use the complete stack trace to inform developers of a bug you have discovered before. This will be highly appreciated by them and will help to improve your favorite program.

The actual backtrace (or stack trace) can be obtained viagdb, the GNU Debugger. It can be used in several ways, depending on whether it should start a new instance of a program, attach to an existing process, or examine a previous crash.

Startgdb with an executable program that can be found in$PATH (or a full path):

$ gdbapplication

gdb automatically tries to download debug information and symbols for packages in theofficial repositories. Whengdb asks whetherDebuginfod should be enabled in the debugging session, answery:

This GDB supports auto-downloading debuginfo from the following URLs:  <https://debuginfod.archlinux.org>Enable debuginfod for this session? (y or [n]) yDebuginfod has been enabled.To make this setting permanent, add 'set debuginfod enabled on' to .gdbinit.Downloading separate debug info for /usr/bin/applicationReading symbols from /home/user/.cache/debuginfod_client/fbaee841e2ed2c11ecbbda26f39eeec1da23d6c3/debuginfo...

Then, withingdb, typerun followed by any arguments you wish the program to start with:

(gdb) runarguments

Now do whatever is necessary to evoke the bug.gdb will automatically halt the application when it crashes and prompt for commands. In case of freezes or similar issues, pressCtrl+c and you will be returned to the command prompt, too.

Then enable logging to a file:

(gdb) set logging enabled on

And finally have the backtrace written to the specified file in the current working directory:

(gdb) thread apply all backtrace full

If the program you want to debug is already running, you need to first find its process ID. Tools such aspidof(1) orpgrep(1) are available. For example:

$ pidof python3

907171 491909

When the output does not give a unique ID, you can try more filtering or look at the output ofps aux orpstree --show-pids.

Attaching as regular user does not work by default due to restrictedptrace scope. The restriction can be lowered temporarily withecho 0 > /proc/sys/kernel/yama/ptrace_scope or you can rungdb as a privileged user, e.g. usingsudo.

Startgdb attaching it to the process:

$ gdb --pid=PID

gdb will ask ifDebuginfod should be enabled in this debugging session, to which you should answery.

Note that attaching to a process has stopped it and it needs to be explicitly continued. This replaces therun command from the workflow in the#Starting a new instance of a program section:

(gdb) continue

Now do whatever is necessary to evoke the bug in the attached process. Then proceed with enabling logging and obtaining the trace same as in#Starting a new instance of a program.

To debug an application that has already crashed, you will want to invokegdb on itscore dump. SeeCore dump#Analyzing a core dump for details.

If debugging information for the crashed program is not available and a proper backtrace was not obtained, you may need torebuild the package and reproduce the crash again.

The first thing to do is to obtain the names of the packages whichrequire rebuilding or theinstall of a debug package.

[...]Backtrace was generated from '/usr/bin/epiphany'(no debugging symbols found)Using host libthread_db library "/lib/libthread_db.so.1".(no debugging symbols found)[...]

For example for the above extract from a trace, the package name for the associated package can be obtained withpacman:

$ pacman -Qo /lib/libthread_db.so.1

/lib/libthread_db.so.1 is owned byglibc 2.5-8

The package is calledglibc in version 2.5-8. Repeat this step for every package that needs debugging information.

A few mirrors currently distribute debug packages in accessible repositories. These are sponsored mirrors controlled by Arch Linux and are given access to the debug repositories.

• https://geo.mirror.pkgbuild.com (GeoDNS mirror)

To install a package you can install it directly from the repository. For example:

# pacman -U https://geo.mirror.pkgbuild.com/core-debug/os/x86_64/zstd-debug-1.5.2-2-x86_64.pkg.tar.zst

Another option is to add the repositories to your pacman configuration.

/etc/pacman.conf

# Testing Repositories[core-testing-debug]Include = /etc/pacman.d/mirrorlist[extra-testing-debug]Include = /etc/pacman.d/mirrorlist[multilib-testing-debug]Include = /etc/pacman.d/mirrorlist# Stable repositories[core-debug]Include = /etc/pacman.d/mirrorlist[extra-debug]Include = /etc/pacman.d/mirrorlist[multilib-debug]Include = /etc/pacman.d/mirrorlist

Place a mirror with debug packages as the first one in the mirrorlist file:

/etc/pacman.d/mirrorlist

Server = https://geo.mirror.pkgbuild.com/$repo/os/$arch...

If debug information is not exposed throughdebuginfod (for example, when the package originates from theAUR), then it can be rebuilt from source. SeeABS for packages in theofficial repositories, orAUR#Acquire build files for packages in theAUR.

To set the required#Compilation options, you can modify themakepkg configuration if you will only usemakepkg for debug purposes. In other cases, you should modify package'sPKGBUILD file only for each package you would like to rebuild.

As of pacman 4.1,makepkg.conf(5) has debug compilation flags inDEBUG_CFLAGS andDEBUG_CXXFLAGS. To use them, enable thedebugmakepkg option, and disablestrip.

These settings will force compilation with debug symbols and will disable their stripping from executables.

/etc/makepkg.conf

OPTIONS+=(debug !strip)

To apply this setting to a single package, modify thePKGBUILD:

PKGBUILD

options=(debug !strip)

Alternatively you can put the debug information in a separate package by enabling bothdebug andstrip, debug symbols will then be stripped from the main package and placed, together with source files to aid in stepping through the debugger, in a separatepkgbase-debug package. This is advantageous if the package contains very large binaries (e.g. over a GB with debug symbols included) as it might cause freezing and other strange, unwanted behavior occurring.

Certain packages such asglibc are stripped regardless. Check thePKGBUILD for sections such as:

strip $STRIP_BINARIES usr/bin/{gencat,getconf,getent,iconv,iconvconfig} \                      usr/bin/{ldconfig,locale,localedef,makedb} \                      usr/bin/{pcprofiledump,pldd,rpcgen,sln,sprof} \                      usr/lib/getconf/*strip $STRIP_STATIC usr/lib/*.astrip $STRIP_SHARED usr/lib/{libanl,libBrokenLocale,libcidn,libcrypt}-*.so \                    usr/lib/libnss_{compat,db,dns,files,hesiod,nis,nisplus}-*.so \                    usr/lib/{libdl,libm,libnsl,libresolv,librt,libutil}-*.so \                    usr/lib/{libmemusage,libpcprofile,libSegFault}.so \                    usr/lib/{audit,gconv}/*.so

And remove them where appropriate.

Packages usingClang as the compiler will not build with thedebug option due to the debug flag-fvar-tracking-assignments' not being handled (e.g. the previousjs78 PKGBUILD).

Add the following at the top of thebuild() function to only remove the flag for the affected package:

build() {  CFLAGS=${CFLAGS/-fvar-tracking-assignments}  CXXFLAGS=${CXXFLAGS/-fvar-tracking-assignments}[...]

UsingLink-time optimization (LTO) will, both during compiling and in a debugger, use more memory[1][2]. Depending on the application, especially if it is a large one like Firefox or Qt, it might exceed the available memory. Build the application without LTO if this happens.

All packages in the official repositories are generally built with LTO.

Build the package from source usingmakepkg while in thePKGBUILD's directory. This could take some time:

$ makepkg

Then install the built package:

# pacman -U glibc-2.26-1-x86_64.pkg.tar.gz

• Debian:HowToGetABacktrace
• Gentoo:Project:Quality Assurance/Backtraces
• Fedora:StackTraces
• GNOME - Getting Stack Traces
• gdb mini intro

• Package development
• Development

• Pages or sections flagged with Template:Out of date
• Pages or sections flagged with Template:Expansion
• Pages or sections flagged with Template:Merge