Software Quality Assurance Software Engineering 9

1

Software Warranty - An Example Limited Warranty on Software and Media KEYSPAN warrants the disks on which the Software is distributed to be free from defects in materials and workmanship and that the Software will perform substantially in accordance with the Documentation for a period of 90 days from your receipt of the Product. ... If the Product fails to comply with the warranty set forth above, KEYSPAN's entire liability and your exclusive remedy will be replacement of the disk or, at KEYSPAN's option, KEYSPAN's reasonable effort to make the Product meet the warranty set forth above. ...

No Liability for Consequential Damages In no event shall KEYSPAN or its resellers be liable for any damages whatsoever (including, without limitation, damages for loss of profits, business interruption, loss of information, or other pecuniary loss) arising out of the use of or inability to use this KEYSPAN product, even if KEYSPAN has been advised of the possibility of such damages.

2

McCall’s Triangle of Quality Maintainability

Portability

Flexibility Testability

Reusability Interoperability

PRODUCT REVISION

PRODUCT TRANSITION

PRODUCT OPERATION Correctness Usability Efficiency Integrity Reliability 3

Internal and external attributes Number of procedure par ameters Maintainability Cyclomatic complexity Reliability Program size in lines of code Portability Number of error messages Usability Length of user manual

4

Why SQA Activities Pay Off? cost to find and fix a defect 100

60.00-100.00

log scale 10 1

10.00 0.75 1.00

1.50

3.00

testsystem field Design Req. use code test 5

Quality Concepts

General objective: reduce the “variation between samples” ... but how does this apply to software? Quality control: a series of inspections, reviews, tests Quality assurance: analysis, auditing and reporting activities Cost of quality • Appraisal costs • Failure costs • External failure costs 6

Software Quality Assurance

Process Definition & Standards

Formal Technical Reviews

Analysis & Reporting Measurement

Test Planning & Review

7

Organization of SQA - an example

Systematic Software Engineering Dansk Standard

Management AQAP Auditor Finance

Business Process Improvement

Danish Air Force R&D and Consulting

Defense Systems

Danish Air Force Project Group

Command & Control

Industrial Systems

IRIS Interoperability

EC / EDI

Key Accounts

Air: OPUS CCIS

IRIS - Message Format.

EDItoolbox

DSB

Navy: RDN CCIS

IMT - Data Modeling

EDIsec

Superfos

Navy: RDN EWare

X-Post

EC/EDI projects

Realkreditrådet

F-16 Simulator

Post Danmark Tele Danmark Internet Landbr. Rådgiv.center Den Danske Bank

8

SQA Considerations

• SQA organizations are rarely staffed with sufficiently experienced or knowledgeable people. • The SQA management team is often not capable of negotiating with development. • Senior management often backs development over SQA on a large percentage of issues. • Many SQA organizations operate without suitably documented and approved development standards and procedures. • Software development groups rarely produce verifiable quality plans. • Getting good people into SQA is one of the most difficult problems software managers face. One effective solution is to require that all new development mangers be promoted from SQA. 9

Software Quality Assurance CMM Goals 1. Software quality assurance activities are planned. 2. Adherence of software products and activities to the applicable standards, procedures, and requirements is verified objectively. 3. Affected groups and individuals are informed of software quality assurance activities and results. 4. Noncompliance issues that cannot be resolved within the software project are addressed by senior management. 10

Key points

Software quality management is concerned with ensuring that software meets its required standards Quality assurance procedures should be documented in an organisational quality manual Software standards are an encapsulation of best practice Reviews are the most widely used approach for assessing software quality

11

Reviews & Inspections

... there is no particular reason why your friend and colleague cannot also be your sternest critic. Jerry Weinberg

12

Quality management and software development

Software development process

D1

D2

D3

D4

D5

Quality management process

Standards and procedures

Quality plan

Quality review reports

13

What Are Reviews?

• A meeting conducted by technical people for technical people • A technical assessment of a work product created during the software engineering process • A software quality assurance mechanism • A training ground 14

What Reviews Are Not

They are not: • a project budget summary • a scheduling assessment • an overall progress report • a mechanism for reprisal or political intrigue!!

15

The Players review leader

standards bearer (SQA)

producer

maintenance oracle

reviewer

recorder user rep

16

1. be prepared—evaluate product before the review 2. review the product, not the producer

Conducting the Review

3. keep your tone mild, ask questions instead of making accusations 4. stick to the review agenda 5. raise issues, don't resolve them 6. avoid discussions of style—stick to technical correctness 7. schedule reviews as project tasks 8. record and report all review results 17

Reviews: An Effectiveness Scale

Most Effective Inspection (FTR) Walk through (FTR) Formal presentation Formality

Informal presentation Peer group review Casual conversation

18

The inspection process

Planning Overview

Follow-up Individual preparation

Rework Inspection meeting

19

Fault class Data faults

Inspection check Are all program variables initialised before their values are used? Have all constants been named? Should the lower bound of arrays be 0, 1, or something else? Should the upper bound of arrays be equal to the size of the array or Size -1? If character strings are used, is a delimiter explicitly assigned? Control faults For each conditional statement, is the condition correct? Is each loop certain to terminate? Are compound statements correctly bracketed? In case statements, are all possible cases accounted for? Input/output faults Are all input variables used? Are all output variables assigned a value before they are output? Interface faults Do all function and procedure calls have the correct number of parameters? Do formal and actual parameter types match? Are the parameters in the right order? If components access shared memory, do they have the same model of the shared memory structure? Storage management If a linked structure is modified, have all links been faults correctly reassigned? If dynamic storage is used, has space been allocated correctly? Is space explicitly de-allocated after it is no longer required? Exception Have all possible error conditions been taken into management faults account?

Inspection checks 20

Static and dynamic verification Software inspections: Concerned with analysis of the static system representation to discover problems (static verification) • May be supplemented by tool-based document and code analysis Software testing: Concerned with exercising and observing product behaviour (dynamic verification) • The system is executed with test data and its operational behaviour is observed

21

Software inspections

Involve people examining the source representation with the aim of discovering anomalies and defects Do not require execution of a system so may be used before implementation May be applied to any representation of the system (requirements, design, test data, etc.) Very effective technique for discovering errors

22

Inspections and testing

Inspections and testing are complementary and not opposing verification techniques Both should be used during the V & V process Inspections can check conformance with a specification but not conformance with the customer’s real requirements Inspections cannot check non-functional characteristics such as performance, usability, etc.

23

Metrics Derived from Reviews

• • • • • • • •

inspection time per page of documentation inspection time per KLOC or FP inspection effort per KLOC or FP errors uncovered per reviewer hour errors uncovered per preparation hour errors uncovered per SE task (e.g., design) number of minor errors (e.g., typos) number of major errors • (e.g., nonconformance to req.)

• number of errors found during preparation 24

Statistical SQA

Product & Process

• collect information on all defects • find the causes of the defects • move to provide fixes for the process

measurement … an understanding of how to improve quality ... 25

Inspection rate

500 statements/hour during overview 125 source statement/hour during individual preparation 90-125 statements/hour can be inspected Inspection is therefore an expensive process Inspecting 500 lines costs about 40 man/hours effort = £2800

26

Automated static analysis

Static analysers are software tools for source text processing They parse the program text and try to discover potentially erroneous conditions and bring these to the attention of the V & V team Very effective as an aid to inspections. A supplement to but not a replacement for inspections

27