Difference between revisions of "Gdb"

Latest revision as of 21:05, 25 October 2023

gdb GNU Debugger

This local wiki page a collection of external references and notes regarding `gdb` or GNU Debugger. In addition to the debugger itself there is also a gdb server piece, and another piece of software named `openocd`. The manual page for openocd describes this utility as a "free and open on-chip debugging, in-system programming and boundary-scan testing tool for ARM and MIPS systems", so perhaps this is a tool like gdb but not one normally used in conjunction with gdb itself.

Here in this first list of references are documentation pages for gdb. Not sure whether Sourceware is the development group for gdb:

• https://sourceware.org/gdb/onlinedocs/gdb/Server.html#Monitor-Commands-for-gdbserver
• https://sourceware.org/gdb/onlinedocs/gdb/Tracepoints.html

^ Basic Commands

Some basic `gdb` commands include:

• b breakpoint
• j jump (j --start)
• c continue
• n next instruction
• s step into

Printing range of internal flash memory, an example:
x/64xb 0x10000000

• info threads
• info breakpoints
• interrupt
• quit

^ gdb options

• https://stackoverflow.com/questions/10874298/gdb-disassemble-one-line gdb interactive command `set disassemble-next-line on`

^ Start gdb Server and Client

Note: the example gdb invocations here are narrowed by their use of Segger JLink debugger, associated Segger shared objects for Linux and debugging context of Zephyr RTOS.

First a link to a Nordic Semi Developers' Zone post, which gives an example invocation by one engineer Hakon on how to start gdb server and gdb client, and connect then to a program running on a remote target processor:

• https://devzone.nordicsemi.com/f/nordic-q-a/86265/gdb-commands-to-list-zephyr-app-threads

The manual set up and session start of gdb server involves typically two terminal windows. In the first terminal an example command

 JLinkGDBServer -if swd -device nrf9160_xxaa -rtos /opt/SEGGER/JLink/GDBServer/RTOSPlugin_Zephyr.so

When we have out of need a first instance of gdb server running and using default network ports, it is necessary to start further gdb server instances with alternate port selections. It may also be needful to inform gdb server not to attempt to halt given core, in cases where JLinkGDBServer is not able to signal the target core on target processor to halt:

 JLinkGDBServer -port 2334 -swoport 2335 -telnetport 2336 -if swd -device LPC55S69_M33_1 -rtos /opt/SEGGER/JLink/GDBServer/RTOSPlugin_Zephyr.so -nohalt
 

In the second terminal, attach manually to the target calling gdb as a client:

 arm-none-eabi-gdb path/to/build/zephyr/zephyr.elf

Once at gdb prompt in second terminal, connect to the server in a manner like:

 target remote localhost:2334

See also references:

• https://wiki.segger.com/J-Link_GDB_Server#Command_line_options . . . JLinkGDBServer command line options
• https://www.segger.com/supported-devices/ . . . Segger JLink supported devices
• https://www.segger.com/supported-devices/nxp/lpc55s6x . . . Segger Link support for NXP LPC55S6x family
• https://wiki.segger.com/NXP_LPC55xx . . . Segger JLink support for LPC55S69 cavaets, Segger device support wiki

^ Attaching gdb To Programs

Some example invocations to Zephyr RTOS `west` utility, to attach gdb to a running program . . ,

 $ west attach -d ./build/cpu0

 $ west attach -d ./build/cpu0 --reset-after-load

 $ west attach -r jlink -d ./build/cpu1 --reset-after-load

Note that for debugging threads in Zephyr apps a developer may likely want to enable Kconfig symbol `DEBUG_THREAD_INFO`, e.g. in prj.conf or similar project file add stanza `CONFIG_DEBUG_THREAD_INFO=y`. Further reading at Zephyr RTOS v3.4.0 documentation pages:

• https://docs.zephyrproject.org/3.4.0/kconfig.html#CONFIG_DEBUG_THREAD_INFO

Zephyr threads may be given names. This Zephyr RTOS feature requires a developer to enabled symbol `THREAD_NAME` in the project's Kconfig settings, CONFIG_THREAD_NAME=y. Threads otherwise are identified by 32-bit integers. It may be possible to figure out which thread backtrace or other information belongs to which thread, when issuing a gdb command which iterates through all threads to report some information. As an example gdb reports a backtrace of all threads when given the following command:

 (gdb) thread apply all bt

With thread awareness for gdb enabled by the application this can be a very powerful debugging facility.

^ Bugs in gdb

• https://github.com/riscvarchive/riscv-binutils-gdb/issues/230

^ Gdb and Python

It is possible to automate certain debugging steps in `gdb` using Python, and a gdb commands Python library:

• https://undo.io/resources/gdb-watchpoint/how-write-user-defined-gdb-commands-python

----- ----- ----- ----- ----- ----- ----- ----- -----

^ To Use openocd With Zephyr Projects

0325

• https://developer.nordicsemi.com/nRF_Connect_SDK/doc/1.4.2/zephyr/guides/debugging/host-tools.html

0323

Following link opens a page with four specific Zephyr RTOS topics related to debugging and thread study . . .
• https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/zephyr/guides/debug_tools/index.html
This link at Memfault forum or site mentioned and links to a related topic "Reproducible Firmware Builds", important for configuring relative paths in firmware projects so that gdb can find source files during debugging . . .
• https://interrupt.memfault.com/blog/advanced-gdb author is engineer who worked for Pebble as part of past work history, recommends to read whole of gdb manual . . .
• https://github.com/zephyrproject-rtos/zephyr/issues/23520
• https://gist.github.com/vaussard/581d62032670f118947edb2b503a18c4

2022-03-21 Monday -

The following video hosted by two engineers, contributors to Zephyr project and standing committee, may not mention `openocd` but gives some good introductory and overview material on set ups for debugging Zephyr based applications:

• https://www.youtube.com/watch?v=und6sMsjSHc "Best Practices for Debugging Connected Applications running Zephyr"

Say one or both of above engineers work for Memfault, and here is an article at Memfault which came up in a search for "gdb mon long ARM M33" explanations:

• https://interrupt.memfault.com/blog/cortex-m-fault-debug

2021-09-17 Debugging Zephyr applications:

• https://docs.zephyrproject.org/latest/guides/west/build-flash-debug.html#west-build-flash-debug

Openocd reference:

• https://openocd.org/doc/html/GDB-and-OpenOCD.html
• https://openocd.org/doc/pdf/openocd.pdf 2020-09-13 dated version of `openocd` manual
• https://openocd.org/doc/html/GDB-and-OpenOCD.html URL repeated here, see line:
  

However for some, eg. FreeRTOS, uC/OS-III and Zephyr, extra steps must be taken. Zephyr must be compiled with the DEBUG_THREAD_INFO option. This will generate some symbols with information needed in order to build the list of threads.

Zephyr project and DEBUG_THREAD_INFO symbol use, location in Kconfig files:

• https://docs.zephyrproject.org/latest/reference/kconfig/CONFIG_DEBUG_THREAD_INFO.html

</ul>

^ board dot cmake files

In [ Nordic's Debug Host Tools 1.4.2] document there is mention of a Zephyr board directory, which exists for those boards supported by Zephyr RTOS. In this directory there is a file named `board.cmake` with these contents:

  1 # Copyright (c) 2019 Nordic Semiconductor ASA.
  2 # SPDX-License-Identifier: LicenseRef-Nordic-5-Clause
  3 
  4 board_runner_args(nrfjprog "--nrf-family=NRF91" "--softreset")
  5 board_runner_args(jlink "--device=cortex-m33" "--speed=4000")
  6 include(${ZEPHYR_BASE}/boards/common/nrfjprog.board.cmake)
  7 include(${ZEPHYR_BASE}/boards/common/jlink.board.cmake)

Following the included file which looks to contain further `jlink` options, we find `ncs/zephyr/boards/common/jlink.board.cmake` with these contents:

  1 # SPDX-License-Identifier: Apache-2.0
  2 
  3 board_set_flasher_ifnset(jlink)
  4 board_set_debugger_ifnset(jlink)
  5 board_finalize_runner_args(jlink "--dt-flash=y")

Note 2022-03-25: an amendment to file jlink.board.cmake may solve the warning we have seen past three weeks about west flashing our board without verifying - TMH

^ First Page Entry or Section

How to have gdb stop on access to a memory location within a range of memory addresses, popular solution mentions use of valgrind:

• https://stackoverflow.com/questions/11004374/watch-a-memory-range-in-gdb

An excerpt from above StackOverflow page to highlight the issue:

11

I am debugging a program in gdb and I want the program to stop when the memory region 0x08049000 to 0x0804a000 is accessed. When I try to set memory breakpoints manually, gdb does not seem to support more than two locations at a time.

(gdb) awatch *0x08049000
Hardware access (read/write) watchpoint 1: *0x08049000
(gdb) awatch *0x08049001
Hardware access (read/write) watchpoint 2: *0x08049001
(gdb) awatch *0x08049002
Hardware access (read/write) watchpoint 3: *0x08049002
(gdb) run
Starting program: /home/iblue/git/some-code/some-executable
Warning:
Could not insert hardware watchpoint 3.
Could not insert hardware breakpoints:
You may have requested too many hardware breakpoints/watchpoints.

There is already a question where this has been asked and the answer was, that it may be possible to do this with valgrind. Unfortunately the answer does not contain any examples or reference to the valgrind manual, so it was not very enlightning: How can gdb be used to watch for any changes in an entire region of memory?

So: How can I watch the whole memory region?
c
linux
debugging
gdb
Share
Improve this question
Follow
edited May 23 '17 at 12:32
Community♦
111 silver badge
asked Jun 12 '12 at 20:30
iblue
26.9k1818 gold badges8282 silver badges125125 bronze badges

    Interesting fact: PowerPC has ranged breakpoints (but not watchpoints ?): stackoverflow.com/questions/13410941/… – Ciro Santilli 新疆再教育营六四事件法轮功郝海东 Jul 27 '15 at 16:09
    x86 supports small watch ranges up to 8 bytes: en.wikipedia.org/wiki/X86_debug_register – Ciro Santilli 新疆再教育营六四事件法轮功郝海东 Aug 12 '15 at 12:09

Add a comment
2 Answers
29

If you use GDB 7.4 together with Valgrind 3.7.0, then you have unlimited "emulated" hardware watchpoints.

Start your program under Valgrind, giving the arguments --vgdb=full --vgdb-error=0 then use GDB to connect to it (target remote | vgdb). Then you can e.g. watch or awatch or rwatch a memory range by doing rwatch (char[100]) *0x5180040

See the Valgrind user manual on gdb integration for more details
Share
Improve this answer
Follow
edited Jun 17 '12 at 17:57
iblue
26.9k1818 gold badges8282 silver badges125125 bronze badges
answered Jun 13 '12 at 20:00
phd
52133 silver badges33 bronze badges

    4
    After I spent the better part of the day fiddling with mprotect and abusing SIGSEV handlers to break on memory access, I tried this. It works perfectly. You saved my day. Thank you! – iblue Jun 13 '12 at 23:00
    Yes, +1 also. I've been hunting for a feature like this for months. – Crashworks Jun 14 '12 at 0:10
    So, how does one determine the heap address for the process started by valgrind? I usually do this via /proc/[pid]/maps but when I start python via this valgrind command, the maps file doesn't have an entry identified by [heap] as I'm used to finding. – Andrew Falanga Feb 11 '16 at 19:59 

I realise that this answer was written a long time ago, but I am confused by this answer, because gdb can do that by itself, without valgrind's help. At least it can now. It will resort to single-step and check repeatedly. That's obviously really expensive (like valgrind is), so you would narrow it down to when the corruption is about to happen soon, then create the watchpoint, so you don't have it running slow the whole debug run. – doug65536 Dec 10 '20 at 3:40

^ openocd Adapter Definitions Files

Mentioned in [ openocd documentation entry point] this debugging utility typically requires two key configuration files, one config file to describe the often physical, hardware based programming and debugging adapter, and a config file to describe the targeted board or SoC where firmware that is running will be debugged.

Temporary learning note here, local instances of openocd's "ships with" adapter files:

/home/ted/projects/openocd-code/tcl/interface/ftdi/
/home/ted/zephyr-sdk-0.12.4/sysroots/x86_64-pokysdk-linux/usr/share/openocd/scripts/interface/ftdi/
/opt/zephyr-sdk-0.12.4/sysroots/x86_64-pokysdk-linux/usr/share/openocd/scripts/interface/ftdi/
/opt/zephyr-sdk-0.13.0/sysroots/x86_64-pokysdk-linux/usr/share/openocd/scripts/interface/ftdi/
/usr/local/share/openocd/scripts/interface/ftdi/

Ugh, which one of these is referenced when calling `west debug` on a given local Zephyr project? In spite of this, a Nordic DevZone post points to possibility our Segger JLink probe compels `west` to look for a file named jlink.cfg. This appears in the five locations above:

ted@localhost:~$ locate jlink.cfg
/home/ted/projects/openocd-code/tcl/interface/jlink.cfg
/home/ted/zephyr-sdk-0.12.4/sysroots/x86_64-pokysdk-linux/usr/share/openocd/scripts/interface/jlink.cfg
/opt/zephyr-sdk-0.12.4/sysroots/x86_64-pokysdk-linux/usr/share/openocd/scripts/interface/jlink.cfg
/opt/zephyr-sdk-0.13.0/sysroots/x86_64-pokysdk-linux/usr/share/openocd/scripts/interface/jlink.cfg
/usr/local/share/openocd/scripts/interface/jlink.cfg

Invoking `west -v debug` gives:

ted@localhost:~/projects/zephyr-based/z9/aws-iot-stand-alone$ west -v debug
ZEPHYR_BASE=/home/ted/projects/zephyr-based/z9/zephyr (origin: configfile)
-- west debug: rebuilding
cmake version 3.21.1 is OK; minimum version is 3.13.1
Running CMake: /usr/bin/cmake --build /home/ted/projects/zephyr-based/z9/aws-iot-stand-alone/build
[0/17] Performing build step for 'spm_subimage'
ninja: no work to do.
[1/5] Performing build step for 'mcuboot_subimage'
ninja: no work to do.
-- west debug: using runner jlink
runners.jlink: JLINKARM_GetDLLVersion()=75001
-- runners.jlink: JLink version: 7.50a
-- runners.jlink: J-Link GDB server running on port 2331; thread info enabled
runners.jlink: JLinkGDBServer -select usb -port 2331 -if swd -speed 4000 -device cortex-m33 -silent -singlerun -nogui -rtos /opt/SEGGER/JLink_V750a/GDBServer/RTOSPlugin_Zephyr
runners.jlink: /opt/zephyr-sdk-0.12.4/arm-zephyr-eabi/bin/arm-zephyr-eabi-gdb /home/ted/projects/zephyr-based/z9/aws-iot-stand-alone/build/zephyr/zephyr.elf -ex 'target remote :2331' -ex 'monitor halt' -ex 'monitor reset' -ex load
SEGGER J-Link GDB Server V7.50a Command Line Version

JLinkARM.dll V7.50a (DLL compiled Jul  8 2021 18:20:53)

-----GDB Server start settings-----
GDBInit file:                  none
GDB Server Listening port:     2331
SWO raw output listening port: 2332
Terminal I/O port:             2333
Accept remote connection:      yes
Generate logfile:              off
Verify download:               off
Init regs on start:            off
Silent mode:                   on
Single run mode:               on
Target connection timeout:     0 ms
------J-Link related settings------
J-Link Host interface:         USB
J-Link script:                 none
J-Link settings file:          none
------Target related settings------
Target device:                 cortex-m33
Target interface:              SWD
Target interface speed:        4000kHz
Target endian:                 little

GNU gdb (crosstool-NG 1.24.0.212_d7da3a9) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "--host=x86_64-build_pc-linux-gnu --target=arm-zephyr-eabi".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/ted/projects/zephyr-based/z9/aws-iot-stand-alone/build/zephyr/zephyr.elf...
Remote debugging using :2331
arch_cpu_idle () at /home/ted/projects/zephyr-based/z9/zephyr/arch/arm/core/aarch32/cpu_idle.S:107
107		cpsie	i
Resetting target
Loading section rom_start, size 0x23c lma 0x20200
Loading section text, size 0x1cd4a lma 0x20440
Loading section .ARM.exidx, size 0x8 lma 0x3d18c
Loading section initlevel, size 0x80 lma 0x3d194
Loading section sw_isr_table, size 0x208 lma 0x3d214
Loading section net_socket_register_area, size 0xc lma 0x3d41c
Loading section log_const_sections, size 0xf0 lma 0x3d428
Loading section zephyr_dbg_info, size 0x3c lma 0x3d518
Loading section device_handles, size 0x38 lma 0x3d554
Loading section rodata, size 0x554c lma 0x3d590
Loading section datas, size 0x3e0 lma 0x42af4
Loading section devices, size 0xa8 lma 0x42ed4
Loading section _static_thread_data_area, size 0x60 lma 0x42f7c
Loading section k_heap_area, size 0x30 lma 0x42fdc
Loading section k_mutex_area, size 0xb4 lma 0x4300c
Loading section k_msgq_area, size 0x68 lma 0x430c0
Loading section k_sem_area, size 0x60 lma 0x43128
Loading section net_if_area, size 0x4 lma 0x43188
Loading section net_if_dev_area, size 0x1c lma 0x4318c
Start address 0x00026050, load size 143238
Transfer rate: 6358 KB/sec, 5305 bytes/write.
(gdb)

Many options given to `openocd` here but not an obvious clue from where `openocd` config files are read and parsed.

^ Extras

# Machine-generated file - use "minicom -s" to change parameters.
pu port             /dev/ttyUSB0
pu rtscts           No 

^ References

gdb quick start guide (need to move this to local gdb notes page) . . .

• https://web.eecs.umich.edu/~sugih/pointers/gdbQS.html

Git remove contributors from project history . . .

• https://stackoverflow.com/questions/44563131/removing-contributor-from-github-com