Tools for Software Quality Assurance in Embedded Systems

Ashling Microsystems, Inc.

Ashling Microsystems Inc. 1270 Oakmead Parkway, Suite 208 Sunnyvale, CA 94085 Tel: (408) 732 6490 Fax: (408) 732 6497 Email: [email protected]

Tools for Software Quality Assurance in Embedded Systems by Michael Healy Ashling Microsystems, Inc. Sunnyvale, CA [email protected] Contents 1 2 3

6 7 8

Introduction: Achieving Confidence in Software Quality The Software Life-cycle model Structured Test and Documentation requirements in Software Standards 3.1 ISO9000 - Part 3 3.2 European Space Agency Software Engineering standard 3.2.1 The Software Verification and Validation Plan 3.3 DOD Standard 2167A 3.4 The CMU Capability Maturity Model for Software Test and Documentation Principles in Software Engineering Tools for Software Quality Assurance 5.1 The Microprocessor Emulator 5.2 The Test Macro Script language 5.3 Debugging and Testing at Source-Level 5.4 Performance Analysis: measuring, optimizing and recording software performance 5.5 Function Trace 5.6 Code Coverage 5.7 Path Coverage SQA considerations in embedded systems Summary and conclusions: towards reliable embedded software References:

1

Introduction: Achieving Confidence in Software Quality

4 5

1 2 3 3 4 5 6 6 7 7 8 9 10 11 13 13 15 16 16 18

The drive to improve software quality stems from many sources. Users of software are demanding major improvements in the quality and reliability of the products that they buy; quality managers are seeking a workable set of references against which the quality of software products can be assessed; embeddedproduct vendors are seeking to address the market for software products can be addressed, by offering products that conform to recognized standards; legal experts are being called on to explain the consequences of product liability legislation; and R&D managers are seeking to improve productivity in software production against a background of larger projects, tighter schedules, and more exacting performance requirements. An increasing number of enterprises demonstrate their commitment to customer satisfaction by adopting a defined and audited quality management system. In turn, the software engineering community must select

Ashling: Software Quality Assurance in Embedded Systems, Page 1

and implement the appropriate tools, methods and techniques to confidently assert that their software has been rigorously tested to meet their company's adopted standards and procedures, and the requirements of their customers. This paper details how current and emerging tools and methods for software design, development, debug and test can make an indispensable contribution to an efficient and effective process for software development and maintenance within the requirements of Software Engineering and SQA standards. We'll briefly review some representative standards and guidelines for software quality assurance to identify their requirements for test tools and methods. This will show the changing relationship between software development and quality control, with the quality-standards requirements now being built-in to the development process, rather than providing a gating test at the end of the process. We will introduce Ashling's systems for debug, performance analysis and quality assurance in embedded control, and show how the tool set is used to improve development productivity and to meet quality standards.

2

The Software Life-cycle model

The software life-cycle model provides a block-diagram of all stages of the process used for defining, producing and maintaining software. All software quality standards require that a model be defined and used as the basis for the software project; few standards require the adoption of any specific life-cycle model. Although the terminology is not always uniform, most models identify six phases in the life of a software product:

User requirements

Definition of the job to be done

Software requirements

Analysis of how the job will be done

Architectural Design

The high-level design of the product

Production

Detailed design and implementation

Transfer

Software is put into use

Maintenance

After-sales service on the installed software

In the early days of software engineering, the Waterfall Model was among the first widely adopted descriptions of the development process, and became the basis for many software acquisition standards in government and industry. In summary, the waterfall model defines the documents to be delivered as signifying the end of each development phase; each document provides the basis for implementation of the succeeding phase, and the model provides for feedback of the experience gained and the modifications required to the stage that precedes it.

Ashling: Software Quality Assurance in Embedded Systems, Page 2

> User Requirements

UR
Software Requirements

SR


DD

< TR


OM

Transfer Operations and Maintenance

An example of a traditional Software Life-cycle model: the Waterfall Model By today’s standards, this model has serious limitations; its emphasis on documents as deliverables (probably resulting from its origins in government projects) suggests a philosophy based on "throwing the product over the wall" to a team of quality-control inspectors; the feedback path is limited to the preceding phase, and the model is inappropriate for projects requiring extensive feedback from beta-test users. The waterfall model has been modified and improved in the evolutionary development model (where the development process is repeated, with subsequent passes gaining from field experience), the incremental model (which delivers the software in stages), and the transform model (which defines a process for the automatic re-generation of the software in the light of operational experience). Boehm's Spiral Life-cycle Model is a good example of a modern approach to the software life-cycle. The model can accommodate most of the previously-discussed models as special cases. The spiral model sees the intensity of the development process as spiraling outwards from the requirements plan; the radius of the spiral represents the cumulative cost of the preceding stages, and the angular dimension represents the repeated execution of the same sequence of steps - planning, risk analysis, prototyping, detailed design, test, and validation [Ref. 1]. The choice of model must remain a matter of negotiation between contractor and purchaser; the fundamental requirement on the supplier of software quality assurance tools for development is that the tool set should contribute, in a co-ordinated and manageable process, to as many stages and requirements of the life-cycle model as possible.

3

Structured Test and Documentation requirements in Software Standards

The ISO9000 series of quality provides a framework for an organization’s Quality Management System: an infrastructure in which the organization sets out clear principles on how it will conduct its affairs, and implements those principles through an appropriately-detailed set of procedures, controls and measurements. At an international level, we look to the ISO9000 standards as the means for achieving freedom of trade, on the basis of a clear mutual agreement on the quality criteria on which we buy and sell products. 3.1 ISO9000 - Part 3 In view of the relatively low usage of statistical quality control in software production until now (outside of government and military projects), it has needed a specific guideline document to show how the ISO9001 standard for quality management systems in development and production could be applied to software. Part Three of ISO9000 provides "Guidelines for the application of ISO9001 to the development, supply and maintenance of software". [Ref. 2]

Ashling: Software Quality Assurance in Embedded Systems, Page 3

In identifying specific recommendations within this important new international standard, we must first give the customary warning against selective interpretation of any standard - only the standard-document itself can provide the authoritative reference for its implementation. But on the issues of testing and performance-analysis, ISO9000-3 is very clear in its statement of principles. The supplier carries the responsibility for identifying verification requirements for all software projects, and for providing the necessary resources to meet those requirements; verification is needed at all stages of the chosen life-cycle, from design to post-installation servicing. Testing must be based on a rigorously defined plan: the Test Plan Documents. The test plan must identify and describe the testing at all stages of the design-hierarchy: with test cases, test data, and expected results defined for item, integration, system and acceptance testing, and including functional, boundary, performance and usability tests. The test environment, tools and test software should be identified within the test plan. As in most standards for software engineering and software quality assurance, a lot of attention is given to Configuration Management: the method for identifying, controlling and tracking the versions of each software item. ISO9000-3 recommends that configuration management be applied, not just to the software specifications and files, but to all of the development tools that affect the functional and technical specifications. In describing the supporting activities required during the implementation of the life-cycle, the standard recommends that the supplier's quality system should include quantitative measures of the quality of the development and delivery process; the process metrics should reflect the effectiveness at reducing the probability that faults are introduced, or that they go undetected. In a significant conclusion on the relationship between efficient development procedures, software quality, and the management system, ISO9000-3 includes a comment that tools, facilities and techniques used to implement the SQA guidelines can be effective for management purposes as well as for product development. The commercial importance of ISO9001 is steadily increasing, as software purchasers make certification a mandatory requirement on their suppliers. 3.2 European Space Agency Software Engineering standard To represent a more detailed and more demanding approach to the management of software development, it's useful to examine the European Space Agency's "ESA Software Engineering Standards Issue 2" [Ref. 3]. The document describes the software engineering standards for software supplied to the European Space Agency. Although the source of the standard sounds a little intimidating (suggesting voluminous and expensive paperwork requirements), the document itself isn't. It can be recommended to all concerned with software quality and productivity as a valuable reference for development and quality-assurance procedures based firmly on common sense and experience. There may well be cases where full adoption of this standard is not appropriate for industrial or commercial software, but the inclusion of "how you might do it" as well as "what you must do" elements means that the document can be used to develop a company's software quality assurance and software engineering procedures for certification to more generally-applied requirements such as ISO9001 or DOD-STD-2167A. As in most other standards, the ESA software engineering document requires that the project be planned as a series of distinct phases, from the definition of the user's requirements to the operations and maintenance phase. While it requires that a Software Life-cycle model be adopted, it does not mandate a particular model. It requires that the detailed design and production be based on three principles of software engineering: top-down decomposition, structured programming, and concurrent production and documentation, and it provides an important definition of SQA: the process of verifying that the defined set of software engineering standards have been applied. In clarifying project rôles, it states that project

Ashling: Software Quality Assurance in Embedded Systems, Page 4

management is responsible for selecting methods and tools, and enforcing their use. In complementary fashion, it is an SQA responsibility to check that appropriate tools, techniques and methods are selected, and to monitor their correct application. As in ISO9000-3, the ESA specification makes clear that all software items, including tools, test software and data, must be subject to configuration management procedures. Test software must be produced to the same standards as the delivered product, and test software and data must, therefore, be fully documented and controlled. This is a basic requirement for successful use of regression testing. The Software Verification and Validation (SV&V) Plan is the basic test plan for the process, and it requires that the test plan both confirms that the unit or system satisfies the expected requirements, and identifies differences between expected and actual results. The standard includes a practical and illuminating model for the application of Software verification throughout the software life-cycle, with activities that include

and

Checking the traceability of software requirements to user requirements; Checking that design components are traceable to software requirements; Unit testing; Integration testing; System testing; Acceptance testing,

each of which must be described in the Software Verification and Validation Plan. 3.2.1 The Software Verification and Validation Plan Testing a software product against its requirements is fundamental to assuming its quality; and the ESA standard recognizes that test activities may well be the most time-consuming and expensive part of a project. Testing should include both verification and validation. By the IEEE's definition [Ref 7], Verification is "the process of evaluating a system or component to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase"; Validation is "the process of evaluating a system or component during or at the end of the development process to determine whether it satisfies specified requirements". In simpler terms, verification measures the software product against its written specification; validation confirms that the software product correctly performs the function required of it. To provide an effective and efficient test process, these activities must take place at all stages of the development cycle. This process can be illustrated by a diagram of the Software Verification and Validation Plan, often referred to as the "V-diagram", and shown below.

Ashling: Software Quality Assurance in Embedded Systems, Page 5