PRISMA®

Risk-Based Testing for Agile Projects

Erik van Veenendaal www.erikvanveendaal.nl

1

Erik van Veenendaal www. erikvanveenendaal.nl


Founder and shareholder ImproveQS
In testing since 1989 working for many






different clients and in many different roles Author “TMap”, “ISTQB Foundation” and “TMMi” and many other books and papers Former Vice-President International Software Testing Qualifications Board (ISTQB) Supporting member IREB board Keynote speaker, e.g., EuroSTAR, STAR Winner of the European Testing Excellence Award

Improve IT Services BV

2

Testing Challenges
Increasing importance and size of software in society as a whole − e.g., amount of software in consumer product doubles every 24 months, − number of defects is almost constant


High Competition & Outsourcing −

Time-To-Market, Product Quality, Price Levels are essential for business success


Testing often takes 30 - 40% of project costs Risk-based testing deals with both effectiveness and efficiency Improve IT Services BV

3

What is Risk?
“A factor that could result in a future negative consequence; usually expressed as impact and likelihood” (ISTQB Glossary)

Testers ‘only’ have the responsibility to identify the risks and provide information on their status


“to dare to undertake” − management attitude and style……..

Improve IT Services BV

4

Testing = Risk Management
Objective: most feasible coverage − effective usage of limited resources − optimize test effort


Limited resources staffing infrastructure time ! ..


The right level and type of coverage on the right parts at the right time
Always applicable, whether it is V-model, Agile or .... Improve IT Services BV

5

if only we knew !!

Our Challenge….

Improve IT Services BV

6

Risk--Based Testing Risk
Risk identification looks at ways of establishing what the risks are and where they are


Risk analysis looks into the critical, complex and potential error prone areas


Then we build tests to mitigate the risk
Subsequently we monitor and report regarding the risks Improve IT Services BV

7

Based on Practical Experiences

8

Improve IT Services BV

8

Risk Identification
Split up test basis in functional or technical items
V-model: based on requirements
Agile: backlog items / user stories for next sprint
Can also be based on a brainstorm session
Maximum number of appr. 35 risk items

Improve IT Services BV

Risk item 1

Functionality

Risk item 2

Security

Risk item 3

Functionality

Risk item 4

Interoperability 9

Risk Analyses
Risk = impact x likelihood

Improve IT Services BV

technical risk

Likelihood

− What is the impact for the business ? − What is the likelihood that there are defects ?
Determine factors based on previous projects, e.g., defect patterns

You already know this ! Exercise: Risk Factors

Impact – business risk 10

Factors From Practice defect patterns / history


Likelihood


Impact

− complexity − new development (level of re-uses) − interfacing − size − technology − geographical spread − inexperience (of development team) Improve IT Services BV

− business importance (“selling item”) − financial (or other) damage (e.g. safety) − usage intensity − external visibility − cost of rework Customization required 11

Risk Assessment in a Nutshell (1) 1) Stakeholder analysis: Assigning Factors 2) Individual scoring Business Usage Safety importance intensity they shall make choices

Improve IT Services BV

Item 1

5

Item 2

5 4

Item 3

5 4

Item 4

5 2

Item 5

1 4 12

Risk Assessment in a Nutshell (2) 3) Consensus meeting 4) Position the risk items in a Product Risk Matrix Likelihood Complexity

New development

Interfacing

Technology

Experience level

Business import.

Usage intensity

Safety

Item 1

5

3

2

1

5

16

5

4

1

10

Item 2

2

1

2

1

2

8

3

3

1

7

Item n Improve IT Services BV

Impact

13

MoSCoW priorities

Product Risk Matrix H

Could Test x

Likelihood

Must Test

focus of unit I testing x

III

IV

Should Test

“Won’t Test”

L L

M

Impact Improve IT Services BV

x

focus of acceptance x testing

M

x

II

14

H

Agile - Play the Card Game: Risk Poker
Poker Planning / Wide Band Delphi based

Improve IT Services BV

15

Wide Band Delphi / Poker Planning (1)
Bottom-up estimation techniques - wide variance in estimates - commitment of people involved - accurate estimation we use the collective wisdom of the team









Each estimator has cards with valid estimates A user story (product backlog item) is discussed Each estimator selects an estimate Cards are turned over Discuss differences (especially outliers) Repeat until consensus

Improve IT Services BV

16

Wide Band Delphi / Poker Planning (2) User Story Estimator

Moderator Explain Estimates Collect OK? Share info OK? Final estimate Improve IT Services BV

17

Risk Poker the same, but…
Scoring separately for likelihood and impact
Discussion shall be factor based
Limited number of cards for each value (color)
Agreement, Product owner (impact) and leaddeveloper (likelihood) act as tie breaker

Improve IT Services BV

18

Let’s Play Example User Story As as bank account holder, I want a yearly financial overview of my accounts, so that I can submit my taxes Impact Likelihood

? ? Impact: Likelihood:

Improve IT Services BV

19

Agile Product Risk Matrix

example User Story

Improve IT Services BV

20

See any “issues” with this?

Differentiated Test Approach !!
Test design


Level of independence
More time & effort
Most experienced

techniques
Reviews & inspection
Test design start-up meetings
Reviews of test design
Level of detail of test cases
Exit (DoD) criteria, e.g., coverage Improve IT Services BV

person
Priority setting
Regression testing
Re-testing without this a risk assessment doesn’t make much sense !! 21

Real--Life Agile Example Real Test design techniques Must Test Review design Support module tests Review module tests

Exploratory Testing H Couldtests Test Support module Review module tests

x

I

x x

M Likelihood x

x

III

“Won’t Test”

L

M Impact

Improve IT Services BV

22

IV

Test design techniques Shoulddesign Test Review

Exploratory Testing

L

II

H

Agile “One Page Test Plan” Shall include a definition of “done” Must Test ….. Test Approach ….. Should test …… Test Approach ….. Could Test ….. Test Approach ….. Would Test 1-2 hour session ….. Test ApproachA….. to achieve a one page Sprint Test Plan Improve IT Services BV

23

Communication

Improve IT Services BV

24

Product Risk Matrix – Initial

Likelihood - Technical Risk

US 58

US

I

II US US

III

31

49

IV

Impact - Business risk Improve IT Services BV

27

25

Product Risk Matrix – Progress

Likelihood - Technical Risk

US 58

US

I

II US US

III

31

49

IV

Impact - Business risk Improve IT Services BV

27

26

Key learning points
A structured and practical approach for risk--based testing is available risk


Make it Agile Agile,, but use the formal rules
Efficient, Effective and Attractive
Define a risk risk--based differentiated test approach


Practice riskrisk-based tracking & ““reporting” reporting” Improve IT Services BV

27

Downloads… Downloads … Go to www.erikvanveenendaal.nl for:
A full PRISMA white paper (e-book)
Paper: 10 Recommendations on Risk-Based Testing
Paper: Product Risk Assessments in Agile Projects (Risk-Poker)


To acquire the PRISMA freeware tool sent a request to [email protected]


Book “Practical Risk-Based Testing; The PRISMA Approach”, ISBN9-78490-986070 (www.utn.nl) Improve IT Services BV

28

www.erikvanveenendaal.nl

[email protected]

@ErikvVeenendaal Improve IT Services BV

29