Software Quality Overview: Metrics Duvan Luong Ph.D. Operational Excellence Networks

Software Quality Overview: Metrics Duvan Luong Ph.D. Operational Excellence Networks 4/2/2012 Contents copyright © Operational Excellence Networks 2...
Author: Rosalyn Mason
2 downloads 0 Views 1MB Size
Software Quality Overview: Metrics Duvan Luong Ph.D. Operational Excellence Networks

4/2/2012

Contents copyright © Operational Excellence Networks 2012

1

Famous Quotes “To measure is to know” “If you can not measure it, you can not improve it”

Lord Kelvin

"You can't control what you can't measure" Watts Humphrey, SEI

“In God we trust, everyone else bring data!” Edwards Deming

4/2/2012

Contents copyright © Operational Excellence Networks 2012

2

Table of Contents • • • • •

4/2/2012

Definition The Goals-Questions-Metrics methodology Implement metrics Productivity metrics Summary

Contents copyright © Operational Excellence Networks 2012

3

Definition • “Measure”: to obtain measurement information of an object • “Measurement”: A figure, extent, dimensions, capacity, amount, etc. obtained by measuring • “Metric”: A standard of measurement

4/2/2012

Contents copyright © Operational Excellence Networks 2012

4

How to Use Metrics? “If you don't know where you are, a map won't help.” Watts Humphrey , SEI

• Use Metrics as indicators of progress – Adoption – implementation progress – Result – achievement evaluation

• ...

4/2/2012

Contents copyright © Operational Excellence Networks 2012

5

How to Define Metrics? • The Goals-Questions-Metrics (GQM) methodology – How to identify Goals? – What questions to ask? – What metrics to select?

Goals

Questions Metrics

4/2/2012

Contents copyright © Operational Excellence Networks 2012

6

How to Identify Goals? • Process: – Brainstorm the organizational or product objectives and goals that you are trying to achieve. Break into smaller chunks as needed so that they are: • • • • •

specific and understandable within your power to attain measurable appropriate chunk size with a stretch fixed time period Prioritize and choose a small set of goals

• Best Practices: Ask questions to clarify goals – – – –

4/2/2012

What do we want to achieve? What are our organization objectives? What do they meant to you? Do we have any specific goals? Is there any dependency on company level goals? What are company goals?

Contents copyright © Operational Excellence Networks 2012

7

What Questions to ask? •



4/2/2012

Process: – Brainstorm a list of questions for each objective/goal: What do we need to ask in order to get information we need to manage to this goal? – Synthesize, sort and prioritize the questions to get to a core set of questions whose answers will give you sufficient information to effectively manage toward these goals Best Practices: Ask questions to know more about the requirements for the metrics – What are we going to do to achieve the identified objectives/goals? – What do we want to know about the identified goals? – How do we know we are implementing the planned activities for the identified goals? – How do we know we have done the right things? – How do we know we achieve our objectives? – If we achieve those defined objectives/goals, what are the expected results? How are those results look like? feel like? – When do we need the metrics data? – Etc.

Contents copyright © Operational Excellence Networks 2012

8

What Metrics to select? • Process: Brainstorm metrics that will provide answers to the questions. Reduce the list to a small set of essential metrics based on the following considerations: – Ease of collection – Reliability – Usefulness – easy to understand – Breadth of coverage – Credibility with key targets – Present-ability/communicability • Best Practices: Apply the following tests to the selected metrics, a good metric should satisfy many/most of the tests: – Behavior Tests – Definition Tests – Communication Tests – Formatting Tests

4/2/2012

Contents copyright © Operational Excellence Networks 2012

9

Tests for Good Metrics (1)  Behavior Tests: Use these questions to ensure that the metric will cause the right behavior: • Is this metric consistent with the goals of the organization? • What are the negative behaviors that could result if we encourage this metric? • Are the encouraged behaviors consistent with how individuals and teams are rewarded? • Are the encouraged behaviors consistent with the organizations culture?  Definition Tests • Do we know what 'good' performance is? • Is the metric common with other metrics used elsewhere? • Is the definition clear? • Is the metric objective? • Are the people whom the metric measures able to improve it? • Does the metric promote learning and continuous improvement? • Does the definition and formula for the metric provide visibility to what drives the metric? • Is the metric owned by the decision makers who use it? • Does the metric point people in the right direction for corrective action? • Is the metric easily and objectively verifiable? • Is the metric validly measuring what it was intended to measure?

4/2/2012

Contents copyright © Operational Excellence Networks 2012

10

Tests for Good Metrics (2)  Communication Tests • Will the metric be accepted at all levels where it is used? • Will the metric show favorable results when things improve? • Is the metric presented in both numerical and graphical form? • Can the metric be summarized in an aggregate form for reporting upwards? • Is the metric reported at the level of the organization that can do something about improving what the metric measures?  Formatting Test • Is the metric easy to apply? • Does a numerical formula exist? • Is the data to construct the formula readily available? • Is it cost effective to gather the data and calculate the formula? • Is the formula constructed so that it is easily understood? • Has the type of graph for displaying the metric been identified?

4/2/2012

Contents copyright © Operational Excellence Networks 2012

11

Example - Goals • •



4/2/2012

Project Objectives: Implement Review Process to improve Product Quality Goals: – To reduce the amount of product defects that escape to Customer. – To reduce the amount of product defects that are found by later testing activities. – To increase the defects found early in the development timeframe, before formal testing – Use information from the Review found defects for fine tuning later testing activities and also to improve preventive development practices . – To optimize the costs for defects detection Questions to ask for the identified goals: – Are these goals clear to you? If they are not clear why? – Why do we have these objectives and goals? Are there any company level objectives that link to these objectives? – What are the potential numerical goals/limits for the defect reduction goals? – Do we have any company level goals related to the Review process? – What is the timeframe for the goals? – Do we have a plan for implementing the Review process? – What are the projected costs for the implementation? – Who are the owner of these goals? – Who needs to implement Review? – Etc.

Contents copyright © Operational Excellence Networks 2012

12

Example – Questions for Review Implementation Metrics 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13.

4/2/2012

What is the Review process we are going to implement? How do we know we are implementing the Review process? How do we know people have the necessary skill to do Review? What is the level of adoption for Review in the organization? What is our expectation for the adoption? How do we know we have the necessary structure to support Review implementation? How do we know we are doing the right Review activities? How do we know the Review implementation produces actual improvement? How much investment do we want to spend on Review implementation? Which Review technique is better: inspection or walk-through? How do we link Review results to the overall product Quality picture? What are the Industry standard Review metrics? What are the Industry standard Review metric baselines? Who has to implement Review? Who will provide the support for Review implementation? etc.

Contents copyright © Operational Excellence Networks 2012

13

Example – Potential Metrics for Review Implementation •

Adoption: 1. # of Engineers who performing Review are trained 2. % of trained vs. total Engineers who are involved in Review activities 3. # of Moderators for the organization 4. % trained vs. total moderators 5. Ratio of moderator vs. Engineers who are involved in Review activities 6. # of products with a list of key deliverables that need Review 7. % of products with a list of key deliverables that need Review vs. total products in the organization 8. Time spent in Review activities 9. Amount of code (in KLOC) or documentation (pages) that get Review per month 10. # of Review sessions done per month 11. % of actual Review sessions vs. planned sessions • Result: 12. Problems found by Review activities (breakdown to Major vs. Minor, inspection vs. walkthrough) 13. # Major Review problems resolved 14. % of Major Review problems resolved vs. total Major problems found 15. # of Major Review problem not resolved

4/2/2012

Contents copyright © Operational Excellence Networks 2012

14

Review Questions – Metrics Mapping Metrics

Questions 1. What is the Review process we are going to implement?

1. # of Engineers who are performing Review are trained

2. How do we know we are implementing the Review process?

2. % of trained vs. total Engineers who are involved in Review activities

3. How do we know people have the necessary skill to do Review?

3. # of Moderators for the organization

4. What is the level of adoption for Review in the organization? What is our expectation for the adoption?

4. % trained vs. total moderators 5. Ratio of moderator vs. Engineers who are involved in Review activities

5. How do we know we have the necessary structure to support Review implementation?

6. # of products with a list of key deliverables that need Review

6. How do we know we are doing the right Review activities?

7. % of products with a list of key deliverables that need Review vs. total products in the organization

7. How do we know the Review implementation produce s actual improvement?

8. Time spent in Review activities

8. How much investment do we want to spend on Review implementation?

9. Amount of code (in KLOC) or documentation (pages) that get Review during last month 10. # of Review sessions done during last month

9. Which Review technique is better: inspection or walkthrough?

11% of actual Review sessions vs. planned sessions

10. How do we link Review results to the overall product Quality picture?

12. Problems found by Review activities (breakdown to Major vs. Minor, inspection vs. walkthrough)

11. What are the Industry standard Review metrics?

13. # Major Review problems resolved

12. What are the Industry standard Review metric baselines?

14. % of Major Review problems resolved vs. total Major problems found

13. Who has to implement Review? Who will provide the support for Review implementation?

4/2/2012

15. # of Major Review problem not resolved

Contents copyright © Operational Excellence Networks 2012

15

How to Collect and Track Metrics? •

4/2/2012

In order to be real and useful, the metrics must be easy to collect and track. Ask questions to have enough understanding of how to best collect and track the selected metrics: – Collecting metrics: Do you know • How the data will be collected? • Who will collect the metrics? • How often to collect metrics? • How much metrics data to collect? • Where to keep the metrics data? • Has the person who will collect the data been identified? • Etc. – Tracking/reporting metrics: • Has the person who will calculate and plot the metric been identified? • Has the person who will report the metric been identified? • Can the metric be calculated and reported without delay? • Will the metric be available to those who need it? • Will the metric be reported at a pre-agreed frequency? • Who are the audiences for the metrics report? • Etc.

Contents copyright © Operational Excellence Networks 2012

16

Example – Collecting & Tracking Review Metrics •





4/2/2012

Collecting: – The moderator collects and logs the following information for each Review session: • Time spent (provided by the participants) • Problems found during the Review session (with associated data such as problem type, origin, potential causes, major vs. minor, and inspection vs. walk-through) – Department Manager maintains the Review training information for the Engineers. Tracking: – The Review coordinator for the organization, at the beginning of each month, tracks, analyzes, and reports the following information: • Review training metrics • Information for the availability of the list of key deliverables that need Review • Information about actual vs. planned Review sessions • Time spent for Review activities • Problems found by Review activities • % Major Review problems closed vs. total Major Review problems found • # of open Review major problems Using: – Organization GMs and Staff review the monthly Review metrics – Quality Manager reviews the Review metrics – Review coordinator uses the metrics to plan for Review implementation/process improvement

Contents copyright © Operational Excellence Networks 2012

17

Interpret Metrics for Improvements • Metric data, when compare to the baselines and trends, should provide the information about the progress/status of the measured item. – What is the status of the adoption of the measured item? – What is the result of the measured item? • Heuristics for evaluating the achievement must be defined for the collected metrics so decisions can be made and appropriate actions can be taken.

4/2/2012

Contents copyright © Operational Excellence Networks 2012

18

Sample Release Metrics •



4/2/2012

Release Quality and Effectiveness: – Testing metric  How the product was tested by Developers, Alpha, and Beta tester/users – Defects - Incoming, Fixed and Pending metric  How the product defects were found and fixed – Release-to-release defect containment metric  What is the latest release defect containment in comparing with previous releases – Coding standards compliance metric  How the static analysis tool was used to check the coding standards compliant and what did we do with the analysis results – Code coverage (test coverage) metric  What is the code coverage for the latest release – Defect density metric  What is the defect density for the latest release total code size, the defect density for the new/changed code size Release Efficiency: – Release code size  What is the size of the total release code, the new code created, the code changed and the code deleted. – Release effort  What was the effort spent on the release (new development, rework).

Contents copyright Operational Excellence Networks 2012 Copyright © 2010©by Duvan Luong. All Rights Reserved

19

Sample Release Readiness Metrics Release Critical and Serious Defect Density at 6 month after FCS

XX Release Resources (estimate)

8.7

9.0

Category

FTE's*

R&D Development

33

PV

3.5

PE

15

CM

1.5

Pubs

1

PM

0.5

TOTAL*

54.5

7.7

8.0 7.0

6.0 6.0 5.6 5.0 3.7

4.0

3.3

3.3 2.8

3.0

2.6

2.4

2.4

1.9 2.0 0.8

0.6

1.0

*Rough estimates

Average

D 5.7

C 4.0

B 5.5

A 7.1

ZZZ 5.2

YYY 6.1

XXX 6.1

ZZ 3.1

YY 6.1

XX 3.1.5

Z 5.2

Y 6.1

0.0 X.1.11

Customer Critical and Serious Defects per 100000 LOC

10.0

XX Release Code Size Lines

Cmnts

NCSL

FILES

Rel

7.1

7.2

7.1

7.2

7.1

7.2

7.1

7.2

7.1

7.2

TOTAL

1406733

1516891

187372

202944

200884

212165

854816

924144

2700

2947

DELTA

4/2/2012

Blank

110158

15572

11281

Contents copyright © Operational Excellence Networks 2012

69328

247

20

Sample Release Readiness Metrics Release-to-Release Incoming Defects Metric

Release N-4 # of Defects

Release N-3 Release N-2 Release N-1 Release N

FCS

4/2/2012

Contents copyright © Operational Excellence Networks 2012

21

Sample Release Readiness Metrics Product X Incoming Critical Defects before FCS Weekly incoming

Incoming cumulative

Fixed cumulative

assigned

220

Incoming Critical Defects

200 180 160 140 120 100 80 60 40 20

FCS-2

FCS-4

FCS-6

FCS-8

FCS-10

FCS-12

FCS-14

FCS-16

FCS-18

FCS-20

FCS22

FCS-24

FCS-26

0

Weeks before FCS

4/2/2012

Contents copyright © Operational Excellence Networks 2012

22

Sample Release Readiness Metrics PV Quality Metrics

100%

98%

100%

4/2/2012

Contents copyright © Operational Excellence Networks 2012

23

Sample Release Readiness Metrics R&D Quality Metrics Metric

Product X Status

Key Issues

Action plan

Code Coverage

60% coverage in the R&D unit tests

The coverage is low due to large number of legacy tools

Increase coverage to 70% in Product X automated weekly DFT PureCov reports. Improve process to track newly added tests

Coding std QAC++

633 design issues found 88% of software modules are issue-free

84% of issues in 10 modules, it is mainly C++ virtual destructors decl. in the header. (minor)

Fix all issues in X1 (Feb’XX) Automated daily QAC++ checks. Train all engineers on C++ coding standards by Q1’XX

Unit testing

Daily runs of >3500 tests. 99% passing rate is maintained

Progression testing and low code coverage No large flow tests Monitor failed check-ins

Integrate CUT & check-in reports into CM RB. QoR Meister appointed. Add large flow tests by Q1’XX

4/2/2012

Contents copyright © Operational Excellence Networks 2012

24

Productivity - What is it? •

Definition: – Throughput – the amount of “usable” goods (LOC, pages of Doc, Test cases, etc.) or services (tasks of Project Management, Planning, Testing, etc.) produced per unit of time – Cost to produce a “usable” unit of goods or services – Time required to complete a task – ...



What Productivity Improvement meant to You? – – – – –

4/2/2012

Lower the cost of production Increase the amount of throughputs Shorten the production lifecycle time Increase the value of the produced goods or services Reduce/eliminate the amount of returned goods/rework

Contents copyright © Operational Excellence Networks 2012

25

Productivity - What to Measure? •

Effort: – – –



Process: –

– – – –



Number of line of codes developed per EM (Engineer Month) • Number of new line of codes • Number of modified/changed code • Number of deleted code Number of test case developed per EM Number of line of codes per inspection/review hour Average time to Inspect/review, Test, Detect a bug, Fix a bug ...

Cost – – – –

4/2/2012

% of total effort in Engineer Month (EM) breakdown by type of work: new development, Maintenance, others, … Project effort by development phase, by activity, … ...

Per line of code Per Test case Per bug fix ...

Contents copyright © Operational Excellence Networks 2012

26

Industry Example - % of effort related to project size Project Size (NKLOC)

Management and Support (%) - Project Mgmt, Planning, SQA, etc.

Defect Removal (%) – Test development, execution, report

Paper Work (%) – requirements doc, specifications doc, design doc, etc.

Design and Coding (%)

4096

18

37

33

12

2048

17

36

32

15

1024

16

35

31

18

512

15

34

30

21

256

14

33

29

24

128

14

30

26

30

64

13

28

24

35

32

12

26

22

40

16

12

23

20

45

8

11

20

15

54

4

11

19

12

58

2

11

18

10

61

1

11

17

7

65

.5

11

16

5

68

Capers Jones, Applied SW Measurement

4/2/2012

Contents copyright © Operational Excellence Networks 2012

27

Industry Example – Effort, Schedule, Staff, Assignment, Productivity Project Size (NKLOC)

Effort – Person Month (EM)

Schedule Month

Staff Person

Assignment Scope (NKLOC)

Productivity – LOC per EM

4096

81920

96

853

4.8

50

2048

30118

73

413

5.0

68

1024

8192

55

149

6.9

125

512

2994

42

71

7.2

171

256

1099

32

34

7.8

233

128

405

24

27

7.5

316

64

149

18

8

8.0

430

32

55

14

4

8.0

585

16

20

11

2

8.0

795

8

8.1

6

1.35

6.4

982

4

4.35

4.5

1.0

4.0

919

2

2.56

2.56

1.0

2.0

781

1

1.50

1.5

1.0

1.0

664

.5

.89

.89

1.0

1.0

564

Capers Jones, Applied SW Measurement

4/2/2012

Contents copyright © Operational Excellence Networks 2012

28

Summary • Use the Goals-Questions-Metrics (GQM) methodology to define metrics • Collect and use metrics to track the progress of the release activities • Use metrics to determine the readiness of the release • Use metrics for process improvement effort 4/2/2012

Contents copyright © Operational Excellence Networks 2012

29