Requirements¶

• gdb 7.2+ (recommended: 7.4+) with python support enabled (typically truefor distributions)

Setup¶

• Create a virtual Linux machine for QEMU/KVM (see www.linux-kvm.org andwww.qemu.org for more details). For cross-development,https://landley.net/aboriginal/bin keeps a pool of machine images andtoolchains that can be helpful to start from.

• Build the kernel with CONFIG_GDB_SCRIPTS enabled, but leaveCONFIG_DEBUG_INFO_REDUCED off. If your architecture supportsCONFIG_FRAME_POINTER, keep it enabled.

• Install that kernel on the guest, turn off KASLR if necessary by adding“nokaslr” to the kernel command line.Alternatively, QEMU allows to boot the kernel directly using -kernel,-append, -initrd command line switches. This is generally only useful ifyou do not depend on modules. See QEMU documentation for more details onthis mode. In this case, you should build the kernel withCONFIG_RANDOMIZE_BASE disabled if the architecture supports KASLR.

• Enable the gdb stub of QEMU/KVM, either

  • at VM startup time by appending “-s” to the QEMU command line

  or

  • during runtime by issuing “gdbserver” from the QEMU monitorconsole

• cd /path/to/linux-build

• Start gdb: gdb vmlinux

  Note: Some distros may restrict auto-loading of gdb scripts to known safedirectories. In case gdb reports to refuse loading vmlinux-gdb.py, add:

  add-auto-load-safe-path /path/to/linux-build

  to ~/.gdbinit. See gdb help for more details.

• Attach to the booted guest:

  (gdb) target remote :1234

Examples of using the Linux-provided gdb helpers¶

• Load module (and main kernel) symbols:

  (gdb) lx-symbolsloading vmlinuxscanning for modules in /home/user/linux/buildloading @0xffffffffa0020000: /home/user/linux/build/net/netfilter/xt_tcpudp.koloading @0xffffffffa0016000: /home/user/linux/build/net/netfilter/xt_pkttype.koloading @0xffffffffa0002000: /home/user/linux/build/net/netfilter/xt_limit.koloading @0xffffffffa00ca000: /home/user/linux/build/net/packet/af_packet.koloading @0xffffffffa003c000: /home/user/linux/build/fs/fuse/fuse.ko...loading @0xffffffffa0000000: /home/user/linux/build/drivers/ata/ata_generic.ko

• Set a breakpoint on some not yet loaded module function, e.g.:

  (gdb) b btrfs_init_sysfsFunction "btrfs_init_sysfs" not defined.Make breakpoint pending on future shared library load? (y or [n]) yBreakpoint 1 (btrfs_init_sysfs) pending.

• Continue the target:

  (gdb) c

• Load the module on the target and watch the symbols being loaded as well asthe breakpoint hit:

  loading @0xffffffffa0034000: /home/user/linux/build/lib/libcrc32c.koloading @0xffffffffa0050000: /home/user/linux/build/lib/lzo/lzo_compress.koloading @0xffffffffa006e000: /home/user/linux/build/lib/zlib_deflate/zlib_deflate.koloading @0xffffffffa01b1000: /home/user/linux/build/fs/btrfs/btrfs.koBreakpoint 1, btrfs_init_sysfs () at /home/user/linux/fs/btrfs/sysfs.c:3636              btrfs_kset = kset_create_and_add("btrfs", NULL, fs_kobj);

• Dump the log buffer of the target kernel:

  (gdb) lx-dmesg[     0.000000] Initializing cgroup subsys cpuset[     0.000000] Initializing cgroup subsys cpu[     0.000000] Linux version 3.8.0-rc4-dbg+ (...[     0.000000] Command line: root=/dev/sda2 resume=/dev/sda1 vga=0x314[     0.000000] e820: BIOS-provided physical RAM map:[     0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable[     0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved....

• Examine fields of the current task struct:

  (gdb) p $lx_current().pid$1 = 4998(gdb) p $lx_current().comm$2 = "modprobe\000\000\000\000\000\000\000"

• Make use of the per-cpu function for the current or a specified CPU:

  (gdb) p $lx_per_cpu("runqueues").nr_running$3 = 1(gdb) p $lx_per_cpu("runqueues", 2).nr_running$4 = 0

• Dig into hrtimers using the container_of helper:

  (gdb) set $next = $lx_per_cpu("hrtimer_bases").clock_base[0].active.next(gdb) p *$container_of($next, "struct hrtimer", "node")$5 = {  node = {    node = {      __rb_parent_color = 18446612133355256072,      rb_right = 0x0 <irq_stack_union>,      rb_left = 0x0 <irq_stack_union>    },    expires = {      tv64 = 1835268000000    }  },  _softexpires = {    tv64 = 1835268000000  },  function = 0xffffffff81078232 <tick_sched_timer>,  base = 0xffff88003fd0d6f0,  state = 1,  start_pid = 0,  start_site = 0xffffffff81055c1f <hrtimer_start_range_ns+20>,  start_comm = "swapper/2\000\000\000\000\000\000"}

List of commands and functions¶

The number of commands and convenience functions may evolve over the time,this is just a snapshot of the initial version:

(gdb) apropos lxfunction lx_current -- Return current taskfunction lx_module -- Find module by name and return the module variablefunction lx_per_cpu -- Return per-cpu variablefunction lx_task_by_pid -- Find Linux task by PID and return the task_struct variablefunction lx_thread_info -- Calculate Linux thread_info from task variablelx-dmesg -- Print Linux kernel log bufferlx-lsmod -- List currently loaded moduleslx-symbols -- (Re-)load symbols of Linux kernel and currently loaded modules

Detailed help can be obtained via “help <command-name>” for commands and “helpfunction <function-name>” for convenience functions.