Embedded Linux Debugging Techniques An Overview Hugh O’Keeffe Ashling Microsystems Ltd. www.ashling.com

[email protected] Abstract—this paper provides an overview of fleetingly, quickly disappearing and resisting all debugging techniques for systems based on attempts to “shake them out”; however, be warned, Embedded Linux with a particular focus on they lurk below the surface waiting for their opportunity to emerge fully fledged once the those using MIPS powered architectures. product is released and in the field. The cost of Keywords—Linux, Embedded Linux, Debugging, catching and resolving these defects grows exponentially as a product moves from in-house EJTAG and MIPS (development/test) to the outside world (beta I. INTRODUCTION TO DEBUGGING release/end customers). “Debugging is the process of identifying the root II. EMBEDDED LINUX cause of an error and correcting it. It contrasts with Linux is an open-source, royalty free, mature testing, which is the process of detecting the error operating system with a rich feature set and has initially”. Steven McConnell [1]. been ported to a wide range of different processor “As soon as we started programming, we found to architectures including MIPS, ARM, PowerPC and our surprise that it wasn't as easy to get programs Intel x86 (including Atom). Broad support is also right as we had thought. Debugging had to be available for a wide variety of devices, peripherals discovered. I can remember the exact instant when I and bus and communications standards thus making realized that a large part of my life from then on Linux an ideal candidate for use in embedded was going to be spent in finding mistakes in my own systems. Linux running on an embedded system will essentially use the same APIs (e.g. POSIX [4]) programs.” Maurice Wilkes [2] as Linux running on the desktop (even though the Recent research [3] indicates that approximately embedded system and the desktop may user 40% of embedded development projects run behind different underlying processor architectures), thus schedule. Debugging is generally accepted as the allowing portions of the embedded system most time consuming and costly phase of the development to be carried out on the generally development process with some estimates putting more powerful desktop machine. the debug cost as high as 50% of the total development cost [3]. By its very definition, it is Companies wishing to use Embedded Linux in also difficult if not impossible to predict when their products have a number of options for debugging is complete. Bugs or defects are sourcing it, including: insidious by nature and can take from minutes to • Commercially supported distributions months to resolve. Some bugs never appear during available from companies such as Mentor the development or testing phases; some do

Page 1 of 10

Graphics, MontaVista and Wind River (Intel) • Distributions available from their semiconductor company e.g. Freescale (PowerPC), Texas Instruments (OMAP ARM powered devices) or MIPS Technologies (for MIPS powered devices) • Or, for the very brave directly from the Linux archives which contain the very latest sources at www.kernel.org The Embedded Linux distribution will typically come with full-sources and a cross-compiler (to build Embedded Linux for the targeted architecture on a host PC) Market adoption of Embedded Linux and its use in the last five years has rocketed [5] and it can now be found powering a number of embedded devices including mobile-phones, digital TVs, set-top boxes, tablets and routers from companies such as Samsung, Philips, Logitech, Motorola and Linksys. Given the large amount of software required to run these devices and the amount of development time spent on debugging, it is vital that there are well established and efficient tools and techniques for debugging which is the subject of this paper.

The MIPS architecture provides a dedicated logic block on the core IP which facilitates debugging. This block is known as Enhanced JTAG (EJTAG) [8] and provides a mechanism for external tools (e.g. a software based Debugger and a hardware based Debug Probe, see an example at [9]) to connect and communicate with the core using the standard JTAG [10] interface (which is primarily used for incircuit chip/system testing). The EJTAG block and interface provide a number of very useful features to the external debug tools including: • Core reset (and optional immediate halt) • Core halt/start • PC Sampling (read the current Program Counter value) without halting or intruding on core operation • Breakpoints. The ability to halt the core when a specific address is executed (or a specified data location is accessed) • Memory and register access As we shall see later on, the EJTAG unit provides key functionality needed for effective debug of Linux powered embedded systems based on the MIPS architecture. IV. INTRODUCTION TO THE EMBEDDED LINUX ARCHITECTURE

III. INTRODUCTION TO THE MIPS ARCHITECTURE

The following figure shows the typical software architecture of an Embedded Linux powered system:

MIPS [6] was one of the first Reduced Instruction Set Computing Architectures (RISC) and was initially developed at Stanford University. RISC architectures use a simplified instruction set optimised for pipelining [7] and allows very fast execution of individual instructions. Today, the MIPS architecture is an industry standard which is commercially available from MIPS Technologies. They offer a range of 32-bit and 64-bit cores which are licensed to semiconductor or System-on-Chip (SoC) companies for use in their designs. MIPS Technologies are purely an Intellectual Property (IP) Fig. 1 Embedded Linux Components company who license the core IP to others; they themselves do not make or sell ICs. Examples of The above diagram shows the following companies that license and use MIPS cores include: components: Broadcom, Cisco, Microchip, NXP and Sony. • Bootloader. The bootloader is typically stored in non-volatile memory (e.g. flash) A. MIPS DEBUG TECHNIQUES and is executed after power-up. The

Page 2 of 10

•

•

•

bootloader performs low-level initialisation of the CPU, peripherals and memory subsystems, loads the kernel (typically into RAM) and then executes it. Kernel. This is the core of the Embedded Linux operating system and provides the key features including the invocation and management of applications running in user mode. The kernel is the ultimate authority in the Embedded System. Kernel Modules. Modules are typically dynamically loaded and unloaded by the kernel as needed and provide functionality such as device drivers (e.g. when a USB powered device is connected then its associated driver will be loaded by the kernel; when the device is removed (and there are no other devices using the driver) the driver is unloaded). Modules typically have the same privileges as the kernel. Applications. These run with restricted privileges and typically provide the device specific functionality (e.g. the user interface for a mobile phone). This mechanism provides inherent protection and prevents application crashes from bringing down the complete system; however, the payoff is that user applications will typically run slower than the underlying kernel/modules.

To facilitate the above, Linux based systems use virtual addressing which is a much more robust system for memory management when compared to the “flat” memory management techniques used by other operating systems. A. Virtual Memory

Linux uses a Memory Management Unit (MMU) which allows mapping of virtual addresses to physical addresses. Virtual addresses are the addresses used by the actual kernel/application code and are translated at run-time (via the MMU) to physical addresses which are the actual addresses used to access the physical memory as shown in the following figure:

Fig. 2 Contiguous Virtual Memory Mapped to Discontiguous Physical Memory MIPS architectures implement a MMU using the Translation Lookaside Buffer (TLB) hardware block. The TLB can be thought of as a “table” (maintained by the Linux OS) which maps virtual to physical addresses. On most MIPS CPUs, this translation occurs in 4KB chunks called pages. This memory translation provides many useful features [8] including: • Hiding and protection. User-mode applications can only access memory areas that they have permission for thus preventing a user-mode application from inadvertently over-writing another applications memory, its own memory (for example, code which is marked as read-only) or kernel space. • Allocating contiguous memory. Linux can allocate contiguous virtual memory to an application that actually resides in discontiguous physical memory space. Thus from the applications point-of-view it appears to be located in a contiguous memory region. • Demand paging. Allows programs to run as if all the memory they require has been already allocated. This mechanism allows Linux to manage memory very efficiently and will only allocate memory to an application when it actually accesses it (the access triggers an exception which is trapped/handled by Linux thus allowing it to actually load the appropriate data into

Page 3 of 10

memory and allow the application to this mapping “on-the-fly” during application debug (sometimes called dynamic mapping). continue) 2) Kernel Debugging The Linux kernel also resides in virtual memory, however, unlike applications the kernel uses known, fixed virtual addresses and the pages that make up the kernel are permanently “locked” and cannot be paged out. Because of this fixed translation (usually just an offset and known as static mapping), the kernel is essentially located in physical memory which as we shall see later makes kernel debugging much more straight forward than application debugging.

B. Virtual Memory Impact on Debugging

When source-code is compiled and linked to allow debugging, additional information is included in the binaries to assist the debugger. This information will include: • Address to Source-file/Line number translation. This allows the Debugger to determine the particular source-code file associated with a virtual address. E.g. when the program is halted, the Debugger can read the PC value and show the 3) Bootloader Debugging associated source-file/line-number of the The bootloader resides in a fixed address (i.e. current instruction as shown in the executed after reset), uses physical addressing and following figure: hence does not use or is effected by virtual memory. V. EMBEDDED LINUX DEBUG TECHNIQUES

This section discusses Linux Debug Techniques and outlines the particular areas that they can be used in. Fig. 3 Source-level Debugger showing current PC symbolically (i.e. source-file/line number) •

Symbolic type and address information. This information allows the Debugger to present variables symbolically i.e., to know their type (int, float, struct etc) and evaluate their current value.

Fig. 4 Source-level Debugger showing variables symbolically

A. Print statements

This is probably the oldest debug technique in the world and consists of instrumenting source-code with ‘C’ library calls to printf (or printk for the kernel). At run-time, these library functions execute and typically send a message from the CPU via serial port to a console or terminal which is viewable by the user. Some examples of usage include: • Indicating that a program has reached a particular point • Indicating the value of a particular variable • Measuring the execution time of a particular program section (requires additional time-measurement calls to the run-time library)

1) Application Debugging Although widely used, this mechanism is crude Because debug information is represented using virtual addresses, the Debugger must understand and inflexible. For example: • It may require multiple iterations of insert how the kernel handles the mapping between virtual print statement, rebuild, run and view to and physical spaces and in effect must be able to do find a particular problem

Page 4 of 10

•

As shown in the above figure, a typical GDB It increases program size and intrudes on operation (e.g. slows down program setup consists of: execution) • GDB debugger running on the host PC • GDB server running on the target system

B. GDB Source-level Debugger

Most debugging is now done using a source-level debugger which allows you to load your program to the system-under-test and exercise it. Typical functionality within a debugger includes: • Step your program i.e. execute one ‘C’ or Assembly line/statement at a time • Execute your program in real-time up to a defined point (aka a breakpoint) • View/change memory and registers • View/change symbolic variables in your program Source-level debuggers require that the source-code to be debugged (e.g. kernel, modules or applications) are compiled/linked with debug symbols enabled as follows: • For the kernel, run make menuconfig, select Kernel hacking, enable Kernel debugging and Compile the kernel with debug and run make to rebuild the kernel with debug symbols. • For non-kernel items, add the compiler gcc switch -g (which will generate debug symbols) to your makefile and rebuild.

The GDB server typically communicates back to GDB on the host via an ethernet link and an RS232 link provides a mechanism to run a Terminal shell on the host-side which can display all kernel messages (e.g. printk statements executed during boot) and a Linux console or command-shell. GDB could also be run on the target, however, given GDB’s large memory requirements this is not practical for an embedded system (and would also require a keyboard/display on the target which may not be available), hence, the above architecture is typically used as it minimises the target-side requirements.

GDB debugging is an effective mechanism, however, it is generally only useful for application debugging as it requires that the Bootloader/Kernel are fully functional on the target (i.e. the target must be able to run the GDB server). A modified version of GDB known as KGDB/KDB for Kernel debugging has been integrated into the kernel mainline as of v2.6 [12], however, in the authors view this is a primitive tool which offers no sourcelevel debug, cannot halt the CPU or debug a crashed system and requires that the bootloader and The classic example of a source-level debugger is major parts of the kernel are running and specific the GNU Debugger or GDB for short. This is device drivers are functional (e.g. ethernet). For provided as part of the cross-compiler suite with all these reasons it is not discussed further. Linux distributions and is thus supported on all target architectures. C. Bootloader/Kernel Debugging using EJTAG As previously mentioned, solutions based on a GDB server running on the target will not work in unless the target itself is functional and the bootloader/kernel are running. However, debugging the bootloader and kernel can easily be achieved by using an EJTAG based debugging solution (also know as hardware based debugging) as outlined in the following figure:

Fig. 5 GDB based debugging on an Embedded System

Page 5 of 10

Fig. 6 GDB based debugging on an Embedded System using EJTAG In the above figure, the target connection is provided by an EJTAG compliant Debug Probe (e.g. Ashling Microsystems’ Opella-XD [9]). Typically, the EJTAG Debug Probe is provided with a GDB server software driver that runs on the host PC and allows the GDB debugger to debug via the Debug Probe. EJTAG debugging allows complete bootloader and kernel (including modules/drivers) debug and can effectively take control of the target’s based CPU out of reset. This provides the user with immediate control and is invaluable in rooting out hardware issues (e.g. improper CPU initialisation or memory sub-system setup, boardlevel faults etc.) without the requirement that the kernel is running (or the need for a dedicated communications port (e.g. ethernet) to the host PC). EJTAG based debugging provides powerful features such as hardware based breakpoints and the ability to stop the system at any time including the kernel, applications, threads and interrupts. Once stopped, the user can view the low-level state of the CPU, memory and registers. Higher-level views that show Embedded Linux context (e.g. modules loaded, processes running, thread state etc) are also possible as we shall see in the following section. The system can be restarted at any time either in real-time run mode or via step mode (which will execute either an assembly or ‘C’ level statement before again halting).

Previously, we have looked at solutions for application debugging based on a GDB server running on the target and bootloader/kernel debugging using an EJTAG probe. Effective system level debugging, however, requires that both application and kernel debugging are simultaneously available. For example, an application may require the services of some kernel APIs or a device driver, hence, it is very useful to be able to debug from an application level down to a kernel level, providing full source-code visibility and symbolic access as we transition up or down the “call-stack”. As Linux has become more popular in the embedded space, commercial vendors have implemented solutions to address these problems (which are not effectively addressed by the open-source tools like GDB). We will now take a look at one such solution implemented by Ashling Microsystems [9] whom the author works for. E. Ashling’s PathFinder-XD and Opella-XD

Embedded Linux Debugging Tools Ashling’s PathFinder-XD for MIPS Debugger. PathFinder-XD is a C/C++/Assembly debugger based on the Eclipse framework and supports debugging using the QEMU software simulator [11] or the Ashling Opella-XD Debug Probe connected to MIPS powered target hardware. PathFinder-XD provides a GUI based interface (as opposed to the console-mode interface provided by GDB) hosted on Windows and Linux and supports both “bare-metal” (no target operating system) and Embedded Linux based debugging.

D. Integrated Application and Bootloader/Kernel

Debugging

Fig. 7 Ashling’s Opella-XD EJTAG Debug Probe

Page 6 of 10

Run-mode debugging requires that the kernel is up and running and allows non-intrusive debug of a process (i.e. the kernel will continue to run even when a process is halted). Run-mode also supports thread-aware breakpoints and simultaneous debug of multiple processes and threads.

Fig. 9 Simultaneous Application and Kernel Linux Debug Setup using Ashling’s Opella-XD EJTAG Debug Probe and GDB Server Agent. As well as providing full source-code debug support, PathFinder-XD provides additional windows to show the status of Linux running on Debugging target as follows:

Fig. 8 Ashling’s PathFinder-XD Source-level Debugger

PathFinder-XD Embedded Linux support works in two modes: • Stop-mode: Debugging is done via the onchip debug interface (i.e. hardware based e.g. Opella-XD) and the whole system is halted (e.g. kernel and applications) whenever a breakpoint is taken. • Run-mode: Debugging is done purely in software (i.e. no Opella-XD is required) via a target serial/Ethernet interface and requires a software agent (e.g. GDB server) running on the target. In run-mode, the kernel continues to run when an application breakpoint is taken. Stop-mode debugging is useful for bringing up the kernel as it only requires a functional on-chip debug interface (EJTAG) and allows debug from reset. Stop-mode can also be used for process debugging, however, the kernel/interrupts etc. will not continue to run when halted (unlike run-mode). When stopmode debugging a process, PathFinder-XD automatically scans the kernel MMU mapping for that process and sets up the MIPS core TLB to allow debug access to the process’s memory area.

Fig. 10 PathFinder’s File Browser window showing all kernel source-files

Fig. 11 PathFinder’s Linux Module window showing all currently loaded kernel modules

Page 7 of 10

Fig. 12 PathFinder’s Linux Process window showing all processes

Linux kernel, drivers and applications tend to be written in a mixture of C and target specific assembler. In addition, C++ may be used for application development. C/C++ uses dynamic memory allocation and thus it is the programmer’s responsibility to ensure that memory is freed when no longer required; this contrasts with more modern languages (Java, C#) which use automatic garbage collection. The use of dynamic memory can introduce a whole range of problems including: • Memory leaks due to the programmer not freeing allocated resources • Memory overwrites (e.g. writing past the end of a dynamically allocated array) These problems can be very difficult or impossible to detect using traditional source-level debugging and this has resulted in development of additional tools to address this problem. One such tool is Valgrind [13] which instruments the application code under test with special checking code which can detect illegal memory accesses dynamically at run-time or failure to properly free allocated resources. Valgrind also includes additional tools for heap profiling (i.e. available memory pool), thread-error detection and profiling. B. Real-time Profiling Tools

There are a number of profiling tools for Embedded Linux available which allow gathering of run-time information while the system is running (i.e. you do Fig. 13 Viewing the internal kernel module not have to halt the system). Information shown structures in PathFinder typically includes: These views are populated based on PathFinder’s • Current application/thread knowledge of the key internal Linux Kernel • Time allocated to individual process/threads structures. • Interrupts (occurrence of and execution time) • Kernel/driver status VI. ADVANCED EMBEDDED LINUX DEBUGGING • Status of key kernel objects (e.g. semaphores ). Up to now, we have discussed tools that allow source-level debug of Embedded Linux based systems. This section provides a brief overview of These tools are very useful for tracking problems some additional Linux Embedded Debugging tools related to concurrent processes intercommunicating that can be used to detect memory allocation related (e.g. race conditions) and identification of issues and view real-time performance. In addition, performance bottlenecks (thus showing you the we look at debugging on the Android (Embedded optimum location to focus your optimisation efforts). One popular example is the Linux Trace Linux based) platform. Toolkit (LTT) [14] which allows you to A. Detecting memory related failures view/visualise both kernel and user-space activities. LTT works based on instrumentation (i.e. additional

Page 8 of 10

Unlike traditional Embedded Linux systems, Android applications are executed on the Dalvik Virtual Machine which uses its own proprietary byte-code format called Dex. Applications written in Java are converted to Dex format before execution and each application runs in its own Linux process with its own instance of the Dalvik virtual machine. Debugging of Android systems therefore requires both traditional C/C++ Embedded Linux aware source-level debug tools (as outlined in this paper) for the bottom three layers and Java based source-level debug tools for the upper, application layer (specific Java debug tools are provided by Google). There is however currently a gap in this solution as effective system Fig. 14 Sample LTT output display level debugging requires the ability to seamlessly debug from an application level down to a kernel level, providing full source-code visibility and C. Android Debugging symbolic access as we transition up or down the Android is an open-source software platform “call-stack”. Ashling are currently working on (which is based on Embedded Linux) targeting PathFinder-XD extensions to address this. internet enabled devices. Released by Google and VII. CONCLUSION the Open Handset Alliance (OHA) in 2007, Android consists of an operating system, middle- This paper provided an overview of tools and ware and applications. Android has rapidly techniques for debugging Embedded Linux based established itself in the market to such an extent systems and showed how simultaneous kernel and that is now the best selling Smartphone operating application debug can be achieved using a system in the US [15]. However, Android is not just combination of both hardware based debug (for for Smartphones; OEMs are finding that its flexible kernel/modules) and software agent based debug software stack can be used for many different (for applications). In today’s market, product consumer products including Netbooks, Webpads, development times are shortening and engineering teams are under more and more pressure to deliver. Set-Top Boxes and TVs. The Android architecture is comprehensive, Products have a limited “sales window” and containing all the software needed from a boot- development slippage can mean that the market for loader right up to end-user applications. The that product is lost. Debug is a fundamental, timeAndroid software “stack” consists of four layers as consuming part of the development process and thus Embedded Linux debugging tools that shown in the following figure: seamlessly support both kernel and application debug are an essential part of the embedded Applications engineer’s armoury.

statements are inserted into the code) and generates the relevant data at run-time. LTT is optimised to minimise impact on the system under test and efficiently handle the vast amounts of data that can be generated.

Application Framework Libraries/Android runtime Linux Kernel

Fig. 15 Android Software Architecture

Page 9 of 10

REFERENCES [1] Code Complete by Steven Mc Connell, ISBN: 0735619670 [2] Memoirs of a Computer Pioneer by Maurice Wilkes, ISBN: 0262231220 [3] VDC Volume III: Embedded Systems Market Statistics Report [Online] http://www.vdccorp.com/embedded/reports/06/br06-02.html (last accessed: 6/4/2011) [4] POSIX [Online]. http://en.wikipedia.org/wiki/POSIX (last accessed: 6/4/2011). [5] Linux still top embedded OS [Online]. http://www.linuxfordevices.com/c/a/News/Lin ux-still-top-embedded-OS/ (last accessed: 6/4/2011). [6] MIPS Technologies [Online]. http://www.mips.com/ (last accessed: 6/4/2011). [7] Pipelining [Online]. http://en.wikipedia.org/wiki/Instruction_pipelin e . (last accessed: 6/4/2011). [8] See MIPS Run. Second Edition. By Dominic Sweetman. ISBN 13: 978-0-12-088421-6 [9] Ashling Debugger and Debug Probe for MIPS Devices [Online]. http://www.ashling.com/index.php?option=co m_content&task=view&id=311&Itemid=251& phpMyAdmin=75ccdc70077a5fa2186da48b68 d6daf3 (last accessed: 6/4/2011). [10] JTAG [Online]. http://en.wikipedia.org/wiki/ Joint_Test_Action_Group (last accessed: 6/4/2011). [11] QEMU simulator [Online]. http://qemu.org (last accessed: 6/4/2011). [12] The State of Kernel Debugging Technology [Online]. http://kernel.org/pub/linux/kernel/people/jwess el/dbg_webinar/State_Of_kernel_debugging_L inuxCon2010.pdf (last accessed: 6/4/2011). [13] Valgrind [Online] http://www.valgrind.org (last accessed: 6/4/2011) [14] Linux Trace Toolkit [Online] http://www.lttng.org (last accessed: 6/4/2011) [15] Nielsen: Android top smartphone OS in past 6 months. [Online]. http://www.eetimes.com/electronics-

news/4209352/Nielsen--Android-topsmartphone-OS-in-past-6-months (last accessed: 6/4/2011)).

Page 10 of 10