Debugging & Troubleshooting

This guide provides a collection of tips and tricks for debugging and troubleshooting the Sionna Research Kit.

Attaching a debugger (gdb andVS code)

To run agdbserver inside thegNB container to which we can attach debuggers to, we override thedocker compose command line as follows.You can find thedocker-compose.override.yaml file in theconfig/common/docker-compose.override.yaml directory of the Sionna Research Kit.

Listing 1docker-compose.override.yaml
services:oai-gnb:command:["gdbserver",":7777","/opt/oai-gnb/bin/nr-softmodem","-O","/opt/oai-gnb/etc/gnb.conf"]

Note that we simply prependedgdbserver :7777 to the pre-existing command line of theoai-gnb container, which can be obtained as follows:

$dockerinspect--format='{{json .Config.Cmd}}'oai-gnb["/opt/oai-gnb/bin/nr-softmodem","-O","/opt/oai-gnb/etc/gnb.conf"]

We can then attach usinggdb as follows:

$gdb>targetremote<CONTAINER_IP>:7777

To get the local IP of a running container, run:

$dockerinspect-f'{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}'<CONTAINER_NAME,e.g.oai-gnb>

To attach using theVS code debugger, the following examplelaunch.json configuration can be used:

{"version":"0.2.0","configurations":[{"name":"C++ Remote Debug","type":"cppdbg","request":"launch","program":"/opt/oai-gnb/bin/nr-softmodem","customLaunchSetupCommands":[{"text":"target remote <CONTAINER_IP>:7777","description":"attach to target","ignoreFailures":false}],"launchCompleteCommand":"None","stopAtEntry":false,"cwd":"/","environment":[],"externalConsole":false,"pipeTransport":{// needed if you are working on a different machine than the container host"pipeCwd":"${workspaceRoot}","pipeProgram":"ssh","pipeArgs":["user@<remote ip ?.?.?.?>"],"debuggerPath":"/usr/bin/gdb"},"sourceFileMap":{"/oai-ran":"${workspaceRoot}"},"targetArchitecture":"arm","linux":{"MIMode":"gdb","miDebuggerServerAddress":"<CONTAINER_IP>:7777","setupCommands":[{"description":"Enable pretty-printing for gdb","text":"-enable-pretty-printing","ignoreFailures":true}]}}}

Inspecting and debugging inside a container interactively

First, find out the commands that are usually run inside the container of interest, e.g., for theoai-gnb container

$dockerinspect--format='{{json .Config.Entrypoint}} {{json .Config.Cmd}}'oai-gnb["/tini","-v","--","/opt/oai-gnb/bin/entrypoint.sh"]["/opt/oai-gnb/bin/nr-softmodem","-O","/opt/oai-gnb/etc/gnb.conf"]

We can then override the entrypoint to run an interactive session instead of the default launch procedure. This can be done by adding the following to thedocker-compose.override.yaml file

Listing 2docker-compose.override.yaml
services:oai-gnb:stdin_open:true# docker run -itty:true# docker run -tentrypoint:/bin/bash

To attach to a running session after running./start_system.sh ordocker compose up -d oai-gnb, and for example start a debug session, run

$dockercontainerattachoai-gnb$gdb--args/tini-v--/opt/oai-gnb/bin/entrypoint.sh./bin/nr-softmodem-O/opt/oai-gnb/etc/gnb.conf

Running memcheck within an interactive Docker compose session

We can use thecompute-sanitizer tool from the NVIDIA Cuda Toolkit to run a GPU memcheck inside an interactive container session launched as above with the following commands:

$dockercontainerattachoai-gnb$/tini-v--/opt/oai-gnb/bin/entrypoint.shcompute-sanitizer--require-cuda-init=no--toolmemcheck./bin/nr-softmodem-O/opt/oai-gnb/etc/gnb.conf

Profiling with NVIDIA Nsight Systems

To enable interactive profiling inside a container, we can mount theNsight Systems CLI tools installed on the host. This can be done by adding the following to thedocker-compose.override.yaml file

Listing 3docker-compose.override.yaml
services:oai-gnb:stdin_open:true# docker run -itty:true# docker run -tentrypoint:/bin/bashcap_add:-SYS_ADMINvolumes:-/opt/nvidia/nsight-systems/:/opt/nvidia/nsight-systems

To collect system usage statistics using NVIDIA Nsight Systems,nsys can be run inside the gNB container as follows:

$dockercontainerattachoai-gnb$/tini-v--/opt/oai-gnb/bin/entrypoint.sh/opt/nvidia/nsight-systems/2024.2.2/bin/nsysprofile-tcuda,nvtx,osrt,cudnn,tegra-accelerators-o./sysprofile./bin/nr-softmodem-O/opt/oai-gnb/etc/gnb.conf

You can afterwards download the profile data from the Jetson and inspect it locally on your host machine.

Fixing missing linker error messages indocker build ran-build

In./cmake_targets/tools/build_helper, change the reprinted error context in thecompilations() function from

egrep-A3"warning:|error:"$dlog/$logfile||true

to

egrep-C6"warning:|error:"$dlog/$logfile||true

This modification shows additional context before and after lines containing ‘error’, which is particularly helpful for linker (ld) errors where the actual error message often appears before the ‘error’ line in the output.