Software Quality Overview: Metrics Duvan Luong Ph.D. Operational Excellence Networks
4/2/2012
Contents copyright © Operational Excellence Networks 2012
1
Famous Quotes “To measure is to know” “If you can not measure it, you can not improve it”
Lord Kelvin
"You can't control what you can't measure" Watts Humphrey, SEI
“In God we trust, everyone else bring data!” Edwards Deming
4/2/2012
Contents copyright © Operational Excellence Networks 2012
2
Table of Contents • • • • •
4/2/2012
Definition The Goals-Questions-Metrics methodology Implement metrics Productivity metrics Summary
Contents copyright © Operational Excellence Networks 2012
3
Definition • “Measure”: to obtain measurement information of an object • “Measurement”: A figure, extent, dimensions, capacity, amount, etc. obtained by measuring • “Metric”: A standard of measurement
4/2/2012
Contents copyright © Operational Excellence Networks 2012
4
How to Use Metrics? “If you don't know where you are, a map won't help.” Watts Humphrey , SEI
• Use Metrics as indicators of progress – Adoption – implementation progress – Result – achievement evaluation
• ...
4/2/2012
Contents copyright © Operational Excellence Networks 2012
5
How to Define Metrics? • The Goals-Questions-Metrics (GQM) methodology – How to identify Goals? – What questions to ask? – What metrics to select?
Goals
Questions Metrics
4/2/2012
Contents copyright © Operational Excellence Networks 2012
6
How to Identify Goals? • Process: – Brainstorm the organizational or product objectives and goals that you are trying to achieve. Break into smaller chunks as needed so that they are: • • • • •
specific and understandable within your power to attain measurable appropriate chunk size with a stretch fixed time period Prioritize and choose a small set of goals
• Best Practices: Ask questions to clarify goals – – – –
4/2/2012
What do we want to achieve? What are our organization objectives? What do they meant to you? Do we have any specific goals? Is there any dependency on company level goals? What are company goals?
Contents copyright © Operational Excellence Networks 2012
7
What Questions to ask? •
•
4/2/2012
Process: – Brainstorm a list of questions for each objective/goal: What do we need to ask in order to get information we need to manage to this goal? – Synthesize, sort and prioritize the questions to get to a core set of questions whose answers will give you sufficient information to effectively manage toward these goals Best Practices: Ask questions to know more about the requirements for the metrics – What are we going to do to achieve the identified objectives/goals? – What do we want to know about the identified goals? – How do we know we are implementing the planned activities for the identified goals? – How do we know we have done the right things? – How do we know we achieve our objectives? – If we achieve those defined objectives/goals, what are the expected results? How are those results look like? feel like? – When do we need the metrics data? – Etc.
Contents copyright © Operational Excellence Networks 2012
8
What Metrics to select? • Process: Brainstorm metrics that will provide answers to the questions. Reduce the list to a small set of essential metrics based on the following considerations: – Ease of collection – Reliability – Usefulness – easy to understand – Breadth of coverage – Credibility with key targets – Present-ability/communicability • Best Practices: Apply the following tests to the selected metrics, a good metric should satisfy many/most of the tests: – Behavior Tests – Definition Tests – Communication Tests – Formatting Tests
4/2/2012
Contents copyright © Operational Excellence Networks 2012
9
Tests for Good Metrics (1) Behavior Tests: Use these questions to ensure that the metric will cause the right behavior: • Is this metric consistent with the goals of the organization? • What are the negative behaviors that could result if we encourage this metric? • Are the encouraged behaviors consistent with how individuals and teams are rewarded? • Are the encouraged behaviors consistent with the organizations culture? Definition Tests • Do we know what 'good' performance is? • Is the metric common with other metrics used elsewhere? • Is the definition clear? • Is the metric objective? • Are the people whom the metric measures able to improve it? • Does the metric promote learning and continuous improvement? • Does the definition and formula for the metric provide visibility to what drives the metric? • Is the metric owned by the decision makers who use it? • Does the metric point people in the right direction for corrective action? • Is the metric easily and objectively verifiable? • Is the metric validly measuring what it was intended to measure?
4/2/2012
Contents copyright © Operational Excellence Networks 2012
10
Tests for Good Metrics (2) Communication Tests • Will the metric be accepted at all levels where it is used? • Will the metric show favorable results when things improve? • Is the metric presented in both numerical and graphical form? • Can the metric be summarized in an aggregate form for reporting upwards? • Is the metric reported at the level of the organization that can do something about improving what the metric measures? Formatting Test • Is the metric easy to apply? • Does a numerical formula exist? • Is the data to construct the formula readily available? • Is it cost effective to gather the data and calculate the formula? • Is the formula constructed so that it is easily understood? • Has the type of graph for displaying the metric been identified?
4/2/2012
Contents copyright © Operational Excellence Networks 2012
11
Example - Goals • •
•
4/2/2012
Project Objectives: Implement Review Process to improve Product Quality Goals: – To reduce the amount of product defects that escape to Customer. – To reduce the amount of product defects that are found by later testing activities. – To increase the defects found early in the development timeframe, before formal testing – Use information from the Review found defects for fine tuning later testing activities and also to improve preventive development practices . – To optimize the costs for defects detection Questions to ask for the identified goals: – Are these goals clear to you? If they are not clear why? – Why do we have these objectives and goals? Are there any company level objectives that link to these objectives? – What are the potential numerical goals/limits for the defect reduction goals? – Do we have any company level goals related to the Review process? – What is the timeframe for the goals? – Do we have a plan for implementing the Review process? – What are the projected costs for the implementation? – Who are the owner of these goals? – Who needs to implement Review? – Etc.
Contents copyright © Operational Excellence Networks 2012
12
Example – Questions for Review Implementation Metrics 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13.
4/2/2012
What is the Review process we are going to implement? How do we know we are implementing the Review process? How do we know people have the necessary skill to do Review? What is the level of adoption for Review in the organization? What is our expectation for the adoption? How do we know we have the necessary structure to support Review implementation? How do we know we are doing the right Review activities? How do we know the Review implementation produces actual improvement? How much investment do we want to spend on Review implementation? Which Review technique is better: inspection or walk-through? How do we link Review results to the overall product Quality picture? What are the Industry standard Review metrics? What are the Industry standard Review metric baselines? Who has to implement Review? Who will provide the support for Review implementation? etc.
Contents copyright © Operational Excellence Networks 2012
13
Example – Potential Metrics for Review Implementation •
Adoption: 1. # of Engineers who performing Review are trained 2. % of trained vs. total Engineers who are involved in Review activities 3. # of Moderators for the organization 4. % trained vs. total moderators 5. Ratio of moderator vs. Engineers who are involved in Review activities 6. # of products with a list of key deliverables that need Review 7. % of products with a list of key deliverables that need Review vs. total products in the organization 8. Time spent in Review activities 9. Amount of code (in KLOC) or documentation (pages) that get Review per month 10. # of Review sessions done per month 11. % of actual Review sessions vs. planned sessions • Result: 12. Problems found by Review activities (breakdown to Major vs. Minor, inspection vs. walkthrough) 13. # Major Review problems resolved 14. % of Major Review problems resolved vs. total Major problems found 15. # of Major Review problem not resolved
4/2/2012
Contents copyright © Operational Excellence Networks 2012
14
Review Questions – Metrics Mapping Metrics
Questions 1. What is the Review process we are going to implement?
1. # of Engineers who are performing Review are trained
2. How do we know we are implementing the Review process?
2. % of trained vs. total Engineers who are involved in Review activities
3. How do we know people have the necessary skill to do Review?
3. # of Moderators for the organization
4. What is the level of adoption for Review in the organization? What is our expectation for the adoption?
4. % trained vs. total moderators 5. Ratio of moderator vs. Engineers who are involved in Review activities
5. How do we know we have the necessary structure to support Review implementation?
6. # of products with a list of key deliverables that need Review
6. How do we know we are doing the right Review activities?
7. % of products with a list of key deliverables that need Review vs. total products in the organization
7. How do we know the Review implementation produce s actual improvement?
8. Time spent in Review activities
8. How much investment do we want to spend on Review implementation?
9. Amount of code (in KLOC) or documentation (pages) that get Review during last month 10. # of Review sessions done during last month
9. Which Review technique is better: inspection or walkthrough?
11% of actual Review sessions vs. planned sessions
10. How do we link Review results to the overall product Quality picture?
12. Problems found by Review activities (breakdown to Major vs. Minor, inspection vs. walkthrough)
11. What are the Industry standard Review metrics?
13. # Major Review problems resolved
12. What are the Industry standard Review metric baselines?
14. % of Major Review problems resolved vs. total Major problems found
13. Who has to implement Review? Who will provide the support for Review implementation?
4/2/2012
15. # of Major Review problem not resolved
Contents copyright © Operational Excellence Networks 2012
15
How to Collect and Track Metrics? •
4/2/2012
In order to be real and useful, the metrics must be easy to collect and track. Ask questions to have enough understanding of how to best collect and track the selected metrics: – Collecting metrics: Do you know • How the data will be collected? • Who will collect the metrics? • How often to collect metrics? • How much metrics data to collect? • Where to keep the metrics data? • Has the person who will collect the data been identified? • Etc. – Tracking/reporting metrics: • Has the person who will calculate and plot the metric been identified? • Has the person who will report the metric been identified? • Can the metric be calculated and reported without delay? • Will the metric be available to those who need it? • Will the metric be reported at a pre-agreed frequency? • Who are the audiences for the metrics report? • Etc.
Contents copyright © Operational Excellence Networks 2012
16
Example – Collecting & Tracking Review Metrics •
•
•
4/2/2012
Collecting: – The moderator collects and logs the following information for each Review session: • Time spent (provided by the participants) • Problems found during the Review session (with associated data such as problem type, origin, potential causes, major vs. minor, and inspection vs. walk-through) – Department Manager maintains the Review training information for the Engineers. Tracking: – The Review coordinator for the organization, at the beginning of each month, tracks, analyzes, and reports the following information: • Review training metrics • Information for the availability of the list of key deliverables that need Review • Information about actual vs. planned Review sessions • Time spent for Review activities • Problems found by Review activities • % Major Review problems closed vs. total Major Review problems found • # of open Review major problems Using: – Organization GMs and Staff review the monthly Review metrics – Quality Manager reviews the Review metrics – Review coordinator uses the metrics to plan for Review implementation/process improvement
Contents copyright © Operational Excellence Networks 2012
17
Interpret Metrics for Improvements • Metric data, when compare to the baselines and trends, should provide the information about the progress/status of the measured item. – What is the status of the adoption of the measured item? – What is the result of the measured item? • Heuristics for evaluating the achievement must be defined for the collected metrics so decisions can be made and appropriate actions can be taken.
4/2/2012
Contents copyright © Operational Excellence Networks 2012
18
Sample Release Metrics •
•
4/2/2012
Release Quality and Effectiveness: – Testing metric How the product was tested by Developers, Alpha, and Beta tester/users – Defects - Incoming, Fixed and Pending metric How the product defects were found and fixed – Release-to-release defect containment metric What is the latest release defect containment in comparing with previous releases – Coding standards compliance metric How the static analysis tool was used to check the coding standards compliant and what did we do with the analysis results – Code coverage (test coverage) metric What is the code coverage for the latest release – Defect density metric What is the defect density for the latest release total code size, the defect density for the new/changed code size Release Efficiency: – Release code size What is the size of the total release code, the new code created, the code changed and the code deleted. – Release effort What was the effort spent on the release (new development, rework).
Contents copyright Operational Excellence Networks 2012 Copyright © 2010©by Duvan Luong. All Rights Reserved
19
Sample Release Readiness Metrics Release Critical and Serious Defect Density at 6 month after FCS
XX Release Resources (estimate)
8.7
9.0
Category
FTE's*
R&D Development
33
PV
3.5
PE
15
CM
1.5
Pubs
1
PM
0.5
TOTAL*
54.5
7.7
8.0 7.0
6.0 6.0 5.6 5.0 3.7
4.0
3.3
3.3 2.8
3.0
2.6
2.4
2.4
1.9 2.0 0.8
0.6
1.0
*Rough estimates
Average
D 5.7
C 4.0
B 5.5
A 7.1
ZZZ 5.2
YYY 6.1
XXX 6.1
ZZ 3.1
YY 6.1
XX 3.1.5
Z 5.2
Y 6.1
0.0 X.1.11
Customer Critical and Serious Defects per 100000 LOC
10.0
XX Release Code Size Lines
Cmnts
NCSL
FILES
Rel
7.1
7.2
7.1
7.2
7.1
7.2
7.1
7.2
7.1
7.2
TOTAL
1406733
1516891
187372
202944
200884
212165
854816
924144
2700
2947
DELTA
4/2/2012
Blank
110158
15572
11281
Contents copyright © Operational Excellence Networks 2012
69328
247
20
Sample Release Readiness Metrics Release-to-Release Incoming Defects Metric
Release N-4 # of Defects
Release N-3 Release N-2 Release N-1 Release N
FCS
4/2/2012
Contents copyright © Operational Excellence Networks 2012
21
Sample Release Readiness Metrics Product X Incoming Critical Defects before FCS Weekly incoming
Incoming cumulative
Fixed cumulative
assigned
220
Incoming Critical Defects
200 180 160 140 120 100 80 60 40 20
FCS-2
FCS-4
FCS-6
FCS-8
FCS-10
FCS-12
FCS-14
FCS-16
FCS-18
FCS-20
FCS22
FCS-24
FCS-26
0
Weeks before FCS
4/2/2012
Contents copyright © Operational Excellence Networks 2012
22
Sample Release Readiness Metrics PV Quality Metrics
100%
98%
100%
4/2/2012
Contents copyright © Operational Excellence Networks 2012
23
Sample Release Readiness Metrics R&D Quality Metrics Metric
Product X Status
Key Issues
Action plan
Code Coverage
60% coverage in the R&D unit tests
The coverage is low due to large number of legacy tools
Increase coverage to 70% in Product X automated weekly DFT PureCov reports. Improve process to track newly added tests
Coding std QAC++
633 design issues found 88% of software modules are issue-free
84% of issues in 10 modules, it is mainly C++ virtual destructors decl. in the header. (minor)
Fix all issues in X1 (Feb’XX) Automated daily QAC++ checks. Train all engineers on C++ coding standards by Q1’XX
Unit testing
Daily runs of >3500 tests. 99% passing rate is maintained
Progression testing and low code coverage No large flow tests Monitor failed check-ins
Integrate CUT & check-in reports into CM RB. QoR Meister appointed. Add large flow tests by Q1’XX
4/2/2012
Contents copyright © Operational Excellence Networks 2012
24
Productivity - What is it? •
Definition: – Throughput – the amount of “usable” goods (LOC, pages of Doc, Test cases, etc.) or services (tasks of Project Management, Planning, Testing, etc.) produced per unit of time – Cost to produce a “usable” unit of goods or services – Time required to complete a task – ...
•
What Productivity Improvement meant to You? – – – – –
4/2/2012
Lower the cost of production Increase the amount of throughputs Shorten the production lifecycle time Increase the value of the produced goods or services Reduce/eliminate the amount of returned goods/rework
Contents copyright © Operational Excellence Networks 2012
25
Productivity - What to Measure? •
Effort: – – –
•
Process: –
– – – –
•
Number of line of codes developed per EM (Engineer Month) • Number of new line of codes • Number of modified/changed code • Number of deleted code Number of test case developed per EM Number of line of codes per inspection/review hour Average time to Inspect/review, Test, Detect a bug, Fix a bug ...
Cost – – – –
4/2/2012
% of total effort in Engineer Month (EM) breakdown by type of work: new development, Maintenance, others, … Project effort by development phase, by activity, … ...
Per line of code Per Test case Per bug fix ...
Contents copyright © Operational Excellence Networks 2012
26
Industry Example - % of effort related to project size Project Size (NKLOC)
Management and Support (%) - Project Mgmt, Planning, SQA, etc.
Defect Removal (%) – Test development, execution, report
Paper Work (%) – requirements doc, specifications doc, design doc, etc.
Design and Coding (%)
4096
18
37
33
12
2048
17
36
32
15
1024
16
35
31
18
512
15
34
30
21
256
14
33
29
24
128
14
30
26
30
64
13
28
24
35
32
12
26
22
40
16
12
23
20
45
8
11
20
15
54
4
11
19
12
58
2
11
18
10
61
1
11
17
7
65
.5
11
16
5
68
Capers Jones, Applied SW Measurement
4/2/2012
Contents copyright © Operational Excellence Networks 2012
27
Industry Example – Effort, Schedule, Staff, Assignment, Productivity Project Size (NKLOC)
Effort – Person Month (EM)
Schedule Month
Staff Person
Assignment Scope (NKLOC)
Productivity – LOC per EM
4096
81920
96
853
4.8
50
2048
30118
73
413
5.0
68
1024
8192
55
149
6.9
125
512
2994
42
71
7.2
171
256
1099
32
34
7.8
233
128
405
24
27
7.5
316
64
149
18
8
8.0
430
32
55
14
4
8.0
585
16
20
11
2
8.0
795
8
8.1
6
1.35
6.4
982
4
4.35
4.5
1.0
4.0
919
2
2.56
2.56
1.0
2.0
781
1
1.50
1.5
1.0
1.0
664
.5
.89
.89
1.0
1.0
564
Capers Jones, Applied SW Measurement
4/2/2012
Contents copyright © Operational Excellence Networks 2012
28
Summary • Use the Goals-Questions-Metrics (GQM) methodology to define metrics • Collect and use metrics to track the progress of the release activities • Use metrics to determine the readiness of the release • Use metrics for process improvement effort 4/2/2012
Contents copyright © Operational Excellence Networks 2012
29