Seven Lifesaving Tips for Multi-core Debugging

What You Need to Know Before Your First Project

If you haven’t worked on a multi-core project yet, you probably will soon. For decades developers relied on miniaturization, higher clock speeds, and design improvements on single-core chips for software speed-ups. Multi-core processors provide more functionality in a smaller, tightly integrated footprint, which means they draw less power—a big win for makers of embedded systems. As the gains from going multi-core continue to become more competitive with traditional single-core architectures, more companies are making the switch. In fact, according to VDC Research, 61% of embedded engineers expect they will be using multi-core or multiprocessor architectures in their development projects by 2012.

Fully a quarter of multi-core embedded design projects miss meeting their deadlines and their functional targets by at least 50%.

More than 60% of embedded engineers expect to be using multi-core or multiprocessor architectures in their development projects by 2012, according to VDC Research.

1. Breakpoints Can Break Your Debugging

But multi-core is fundamentally different from traditional system design. The things we need to think about as designers or developers are more complex, and while the rewards of migrating to multi-core are great, there are common traps to be aware of. Some are technical, involving new programming complexities and debugging techniques, while others have more to do with project scope and the risks to budgets and timelines. The risks are enough that fully a quarter of multi-core embedded design projects miss meeting their deadlines and their functional targets by at least 50%, according to Embedded Market Forcasters.

2 I Seven Lifesaving Tips for Multi-core Debugging

Knowing about and avoiding these pitfalls can make the difference between a trouble-free development cycle and one that is plagued by problems. So here they are: the top seven issues you need to know about when you make the switch to multi-core.

If you do multithreaded programming, you know that thread-aware debuggers are essential to provide meaningful insight into which thread was executing when a breakpoint was reached and how to stop execution in a synchronized way. On a multi-core system, breakpoints are even more complex because there is no guarantee that the other processor cores will stop simultaneously. In the multi-core world, a debugger needs to have breakpoint monitoring and synchronization across processors or cores, otherwise thousands of instructions may execute before the system is halted. It is very common when halting a core at a breakpoint to lock up the other cores. Of course, you can use a JTAG (Joint Test Action Group) interface, but not all JTAG implementations are equal; and when migrating to a multi-core architecture, it is important to understand the differences. One of the great advantages of JTAG is the ability to truly stop a processor, allowing accurate inspection of its state at a given point in program execution.

With older debugging approaches, low-level operations continue to execute. Even in the JTAG world, debugging becomes more complicated when working with multi-core systems. You can quickly find yourself in a tooling and instrumentation nightmare because multi-core systems can involve several JTAG interfaces or debug agents per core.

SMP: Symmetrical multiprocessing is an environment in which a single operating system determines which core to use for each task. SMP systems use homogenous cores and shared memory. Wind River ICE 2 was designed with these multi-core issues in mind. It provides advanced debugging features such as conditional breakpoints, synchronized breakpoint management, and logical core grouping. For example, let’s say you are consolidating two operating systems on an eight-core device. You are running Linux symmetrical multiprocessing (SMP) on two cores (control plane) and a low-level executive (data plane) on four cores. With the Wind River solution you can represent the cores running the same operating system as a group and synchronize all debug operations for that group. This allows you to represent the system in your debug environment as you designed it. And you have the flexibility to dive deep into any specific core as needed. You just can’t achieve these types of advanced debugging configurations with other toolsets. The Wind River solution supports up to 16 cores simultaneously, so tooling your debugging environment is a simple one-step

3 I Seven Lifesaving Tips for Multi-core Debugging

process. Innovations exclusive to Wind River ICE 2 have refined the debugging experience in multi-core environments.

2. Log Files Can Lead to Logjams Incorrectly synchronized access to shared resources such as memory and cache show up as race conditions, deadlocks, data corruption, and other problems. These issues are notoriously unpredictable and difficult to replicate. The same code processing identical data could mean different results on different passes. Such problems are tricky to debug because they require code-stepping through multiple individual threads. Faced with these limitations, you can resort to logging— peppering the code with printfs to track down bugs—but on a multi-core system, logging presents other problems. The sequencing of information written to logs across cores cannot be guaranteed, making log output unreliable and difficult to understand. In finely tuned architectures, logging can even aggravate unpredictable behavior, affecting memory, registers, cache, and other aspects of the system. A better solution is to use a JTAG-based approach designed to work with the nuances of multi-core chips, providing real-time debugging that is nonintrusive. Wind River Trace extends the value of Wind River ICE 2 by providing an external trace buffer unit for supported processors. This allows developers to monitor program flow to identify and resolve the most challenging issues where the root cause cannot be identified through register and memory access.

3. True Concurrency Can Be Treacherous Resource contention on multi-core systems can lead to subtle

and hard-to-find race conditions. Consider a simple for loop in which two cores modify a global variable in such a way that the loop never exits. Studying and stepping through your source code would reveal nothing out of the ordinary, and it may not be until you gain visibility using hardware watch points that the source of the problem is uncovered. A better approach in the multi-core world is often to use tools that give you rich, detailed insight into code execution for each core. The overhead of a well-designed system-profiling tool can be as low as about 1%. In some cases you may need a completely noninvasive approach. External hardware solutions can give you visibility into hardware/ software interaction, including internal busses, cache resource, and program counters, with the assurance that debugging activities are having no impact on the behavior of the system. A robust external solution will let you record and examine the contents of the program counter. This gives you precise insight into the sequence of instructions being fed to the processor—a powerful tool for detecting problems with non-optimal process allocation, such as an SMP operating system switching a process to another core inefficiently.

Some concurrency problems can be so difficult to reproduce that they only occur after your product has shipped. Even more insidious, some concurrency problems can be so difficult to reproduce that they only occur in the field, after your product has shipped. One way to reduce the risk of this is to deliberately invoke errors during development and testing, such

4 I Seven Lifesaving Tips for Multi-core Debugging

as taking down one core to uncover hidden dependencies across the others. Techniques such as this can help you reduce the risk of these intermittent faults escaping the lab. Wind River has invested in tools to help with visualizing and profiling the dynamic operation of your embedded multi-core system. Tools such as Wind River System Viewer allow you to visualize the interaction of operating objects such as tasks, threads, and semaphores with software that is executing on multiple cores. Wind River Trace is noninvasive external hardware that enables the collection and analysis of historic data from the program counter in addition to other low-level processing details.

Code written for a single-core system and ported to a quad-core processor can expand significantly to handle multi-core requirements. 4. The Wrong Tools Can Cripple Your Project Holistic insight into your multi-core project is critical to achieving scalability and performance, and it’s not enough to think only in terms of point products or individual solution components when making platform choices. You need to take a holistic approach to your development tools as well. Tools for multi-core development must simplify porting and optimization while also providing the level of visibility and assurance you’re accustomed to. And they must deliver new capabilities beyond the traditional edit/compile/debug cycle.

Issues with parallelism, performance, and cache coherency are real—and you need tools that allow you to find these quickly on your platform. Multi-core development and debugging tools also have to address new challenges around configuration, prototyping, and simulation, and they must support diagnosis, analysis, and testing. Having tools that give visibility into the interaction between the application, the operating system, and the chip is essential to achieving your performance and quality goals. You need tools that give you everything you need to effectively prototype, simulate, develop, analyze, and diagnose multi-core software. This applies whether you are working on a single OS single-core design or a single OS SMP design or in the most complex environments involving multi-OS supervised asymmetric multiprocessing (AMP)/SMP configurations with multiple collaborating operating systems and a virtualization layer—all interacting through built-in inter-process communication or inter-core communication.

AMP: Asymmetrical multiprocessing is a collection of independent but interacting operating environments, with separate operating systems running on each core. The ability to do system-level debugging of multi-core systems is critical for in-depth analysis of programming bugs, performance issues, and measuring optimization tweaks. Wind River ICE 2, Wind River Trace, and Wind River Workbench On-Chip Debugging are together an integrated tool solution providing a rich

development environment, multi-core analysis, and on-chip debugging.

5. Projects Can Collapse Under Multi-core Complexity Porting a project from a single to a multi-core environment can challenge the way we work. Without a clear understanding of these issues, you may have problems scaling your development efforts to deal effectively with the size and complexity of your project. Multi-core is so new that terminology such as AMP, SMP, virtualization, supervision, and hyperthreading may be new to many of your team members. They may not understand the issues of running your legacy software on a new multi-core device. And as multi-core is more broadly adopted, more of these developers will be exposed to the concurrency issues that used to be handled by a team specialist. So it’s important to equip your whole team with the best possible tools to ensure that your project doesn’t collapse. Expect your code base to grow significantly. Code written for a single-core system and ported to a quad-core processor expands to accommodate multi-core requirements. Managing and navigating your way around the project files becomes more cumbersome and inefficient. Worse yet, conventional debuggers add to the frustration factor by forcing you to open one monitor window for each core. On a 16-core system, this means hopping between 16 windows to see what is happening. So it’s critical that your debugger gives you the ability to drill down into each core while maintaining a holistic view of your system. Not only do most development environments and emulators not

5 I Seven Lifesaving Tips for Multi-core Debugging

handle these complexities, they also don’t support the sort of distributed team-centric efforts—with bigger workgroups and more sophisticated workflows—that are the hallmarks of multi-core projects. It is quite common today to have engineering and system test teams in different countries, so it makes sense for them all to have access to the same hardware pool. Imagine if you have developers in another country with experience that makes them ideally suited to debug problems on your hardware prototype; you would want to give them seamless access to your debugging environment. If your JTAG tools do not support high-speed remote access, you could be stuck looking for costly, frustrating, and time-consuming workarounds that stretch a 30-minute debugging exercise into days. When you develop embedded applications for multi-core chips, advanced code management becomes more important than ever, and so does distributed team-oriented access to the debugging environment. Wind River Workbench On-Chip Debugging provides advanced code management features, including editing tools such as vi and Emacs emulation, code completion, parameter hinting, and syntax highlighting of source files that improve the edit-compile-debug cycle time and reduce errors. The editor is tightly integrated with the project system, build system, Workbench Source Code Analyzer, and debugger in Workbench, enabling you to move back and forth easily between various source-code debugging

tasks. Wind River ICE 2 supports leading architectures and a multitude of reference boards. With Ethernet connectivity, you can share access among team members throughout the organization. These tools, and Wind River’s training and support services, can help you cross the chasm to multi-core.

6. More Cores Can Slow you Down Getting optimal performance from a multi-core system can be challenging. It might not be obvious, but the higher the number of processing elements, the less of a performance boost per core is likely because more valuable cycles are consumed coordinating tasks across cores. To complicate matters more, Amdahl’s law tells us that even a small portion of a program that cannot be parallelized will limit the maximum speed gains available. If 10% of a program’s execution time is spent in serial code, the best performance possible is a 10-times speed improvement, no matter how many cores are added. In reality, even 5% serialization on an SMP system can be enough to cancel out any benefits you expected from going multi-core. So it’s no wonder that the results are often disappointing when first implementing a multi-core system. A common and costly pitfall is to realize during the integration phase of the product development cycle that the system is not meeting specifications.

“ When a task cannot be partitioned because of sequential constraints, the application of more effort has no effect on the schedule. The bearing of a child takes nine months, no matter how many women are assigned.” — Fredrick P. Brooks Jr. The Mythical Man-Month: Essays on Software Engineering

6 I Seven Lifesaving Tips for Multi-core Debugging

By this point, difficult-to-track-down problems have significant impact on your project timelines and budgets, and the risk of failure amplifies. It is not the time to be heading back to the drawing board. Avoiding these risks involves making the right choices about SMP or AMP early in the project. Often operating systems are relied upon too heavily to deliver SMP services without enough attention to the profiling and customized partitioning that account for the requirements of a specific architecture.

related code for closer inspection or remediation. Workbench Code Coverage Analyzer provides a live analysis of code coverage of a running program and helps to increase your confidence that you have broad coverage of the software that is impacted when running in a multi-core environment. Workbench Data Monitor lets you watch variables, shared variables, data structures, and memory locations in real time. These and other advanced debugging and analysis features provide the insight needed to

Even 5% serialization on an SMP system can be enough to cancel out any benefits you expect from going multi-core.

7. It’s OK to Ask for Help

Wind River Simics lets you create high-performance virtual environments where any system from a single board all the way up to complex, heterogeneous multi-core systems can be defined, developed, and deployed. Your team can use approaches and techniques that are simply not possible on physical hardware to explore architectural alternatives and assess performance, before you’ve invested too many resources to make significant changes practical. Wind River Workbench allows you to play back and examine prerecorded system event information in an intuitive graphical format using System Viewer. You can quickly identify peak activity or sort events by intensity and search for specific events or events of a specific type. Workbench Performance Profiler allows you to examine which aspects of the application are most expensive and jump to the

7 I Seven Lifesaving Tips for Multi-core Debugging

customize your architecture for optimal performance.

A late-night lifeline to professional support services has been the salvation of many multi-core development projects. Consider the example of a company rolling out a project based on a Linux-based SMP kernel that began to encounter random lockups during system integration. The developers tried various debugging approaches—logging, printfs, code inspection— eventually peppering their code with printf statements in an attempt to narrow down the cause. Even after making the issue more reproducible, they were no nearer to finding the cause. Eventually they called in a Wind River support engineer. Using Wind River JTAG tools, he isolated the issue quickly. The problem was not in the project code at all. Rather a bug in the kernel’s translation lookaside buffer (TLB) implementation was causing lockups that would never have occurred on a unicore system. Their product had probably coexisted with this bug for years, only triggering it when they migrated to a multi-core architecture. The Wind River professional support engineer’s deep understanding of how hardware, operating system, and application interact on a multi-core system allowed him to see patterns and read nuances that were not otherwise obvious. The project was saved.

Such engagements with a global award-winning support organization can rescue a floundering project. It’s good risk management to plan for the unexpected and not be afraid to pickup the phone when your project hits a multi-core snag. The benefits of multi-core are numerous, but to realize them you must avoid the common pitfalls by planning carefully and selecting a platform with the required level of optimization, flexibility, and integration. So before you rush into multi-core, take some time to understand these potential pitfalls and make sure that the platform and vendor you choose have the breadth, depth, and quality you’ll need at every level. Doing so will go a long way toward transforming the “potential” of multi-core into solid business results—today and into the future. For more information about Wind River’s multi-core debugging solutions and support capabilities, visit www.windriver.com/ products/OCD.

Wind River is a world leader in embedded and mobile software. We enable companies to develop, run, and manage device software faster, better, at lower cost, and more reliably. www.windriver.com © 2010 Wind River Systems, Inc. The Wind River logo is a trademark of Wind River Systems, Inc., and Wind River and VxWorks are registered trademarks of Wind River Systems, Inc. Other marks used herein are the property of their respective owners. For more information, see www.windriver.com/company/terms/trademark.html. Rev. 10/2010

8 I Seven Lifesaving Tips for Multi-core Debugging