Bug Hunting in practice INTERNATIONAL TESTING CONFERENCE Nordic Testing Days JUNE 1 - 3, 2016 IN TALLINN, ESTONIA •

Szilard Szell, (Marko Kuisma)

1 06/06/2016 Public

© Nokia 2014

1.0 Kuisma_Szell - DocID

Introduction

Marko Kuisma

Szilárd Széll

•2005-2015 working in system testing and performance testing area in Nokia and NSN

•2000 – various Testing Related engineering and management positions in Nokia

•Owner for ~1000 fault reports during testing specialist career for six years in Nokia

•2015 – Test Coach in Central Org.

•Several project manager positions in testing area •Line manager of different test teams for four years •Bachelor of Engineering in Telecommunications, University of Applied Science •Advocate for manual testing with human errors 

2 06/06/2016 Public

© Nokia 2014

1.0 Kuisma_Szell - DocID

•2014 – President of Hungarian Testing Board Studies •MSc in IT Management, Central European University •MSc in Programming and Mathematics, University of Szeged

•Certified Scrum Master, ISTQB CT-AL Test Manager, CT-FL Agile Tester, IREB CPRE, Lean Six Sigma Green Belt

Budapest – Core on Cloud Center Highlights • Core on Cloud: Open MSS, vTAS, TAS-E • SDM Solutions • GSM-R • Cloud Infrastructure • Serve at Once Intelligence, Device Mgmt, Cloud Application Mgmt • All Core Customer Support • T&I: SDN, SON, CEM, Big Data and Network Mgmt 3 © Nokia 2016 Nokia internal use

Ecosystem •Symbiotic cooperation with major ICT Universities in the region •Strategic partnership with Hungarian Government •European ICT hub: Ericsson, Epam Systems, Morgan Stanley, NNG, Lufthansa, SAP, Evosoft, Bosch, Siemens, IBM, Prezi, LogMeIn, BalaBit, TSystems •Partnering with and supporting Hungarian Testing Board

Technology Expertise • Cloud computing and architecture - Core Virtualization • (SDN) Software Defined Networking • 2G/ 3G/4G Technology Expertise • Network Function Virtualization • Self Organizing Networks • Big Data Management

Nokia as a house of Testing Cornerstones of System Verification

• Well defined Testing Processes on Company level • Well defined, standardized Requirements as input • ISO9001 and TL9000 compliant processes

• Lean Six Sigma • Complex, Highly available (99.999%), Robust Products on IT Cloud and Telco HW

4 © Nokia 2016 Nokia internal use

Test Case Design Techniques As used during Product Development Defect-based techniques • Taxonomies

5 06/06/2016 Public

© Nokia 2014

Specificationbased or Black-box techniques

Structure-based or White-box techniques

• EP • BVA • Decision table testing • State transition testing • Orthogonal arrays / all pairs tables • Use case testing

• Statement testing • Decision testing • Condition / Multiple condition / Condition determination testing • Path testing • LCSAJ

1.0 Kuisma_Szell - DocID

Experience-based techniques • Error guessing • Checklist-based testing • Exploratory testing • Attacks

Exploratory Testing (One) Definition • Exploratory Testing can be described as a goal oriented wandering

• There is a mission/goal described in a charter, but there is no planned route you need to take • Create a charter describing what and how and which way you want to test • Describe duration of your test ”Exploratory Testing is an interactive process of concurrent product exploration, test design and test execution. To the extent that the next test we do is influenced by the result of the last we did, we are doing exploratory testing.” James Bach, Satisfice 6 06/06/2016 Public

© Nokia 2014

1.0 Kuisma_Szell - DocID

Exploring Barcelona ET is a goal oriented wandering

7 06/06/2016 Public

© Nokia 2014

1.0 Kuisma_Szell - DocID

8 06/06/2016 Public

© Nokia 2014

1.0 Kuisma_Szell - DocID

Explaining Exploratory test Story Cubes – the game “Once upon a time there was a flash, into who a fish has moved in (to live). The fish has asked the flash - Why is there a pyramid behind you? Why are you striking it? He (the flash) said - because an airplane is just going to fly by 200.000 millimeters below (us). Then, the passengers in the plane has asked the pilot - Why are we turning all around (left and right?)? The pilot said - Because I am reading a book. Then they (the passengers) asked the waitresses - Why don’t we get something to eat? Because (the waitresses said) a tortoise jumped into the soup. And then the tortoise said: Look, the sheep is parachuting out there!” Szonja, 7 years 9 06/06/2016 Public

© Nokia 2014

1.0 Kuisma_Szell - DocID

Explaining Exploratory test Story Cubes – in testing 1.

Gather Developers, Testers, Customers, Business people to the same table

2.

Show them a Story Cube pack

3.

Start collecting 9x6 areas of the system that is important

4.

Make Cubes for your own

5.

Play the game, roll the dice and tell a story by reading the pictograms

6.

You just created an exploratory test case

10 06/06/2016 Public

© Nokia 2014

1.0 Kuisma_Szell - DocID

Explaining Exploratory test Story Cubes – cubes “created” by experts Cube 1

Cube 2

Cube 3

Cube 4

Cube 5

Cube 6

Cube 7

Cube 8

Cube 9

Theme

Unit

State Change

Load Balancer

“CRUD”

Output

Transacti on

Analysis

Attribute Anal.

Interface

Side1

STU

WO RE

H.248 MSS

Create / Add

Log

LocUp

Barring

Routing

A/Iu

Side2

CHU

SP RE

Read / Interrogate

Alarm

HandOve r

User plane

CHA

Mc

Side3

VLRU

CSWO

SIP

Update / Modify

ErrorLog

MO

Digit

EOS

MAP

Side4

CM

FSWO

M3UA

Delete / Remove

ClearCod e

MT

EOS

Service

Nc, BICC/SI P

Side5

SU

DISK

H.248 MGW

Measure ment

Forward

Function

AIF

ISUP

Side6

IPDU

POWER

CHA Ticket

SS

Pre./Ext.

IN

CAMEL

11 06/06/2016 Public

© Nokia 2014

1.0 Kuisma_Szell - DocID

Explaining Exploratory test Story Cubes – a tale by an expert (Balazs Tenyi) • Rolled cubes: CMM -> Faulty SWO -> ClearCode -> A/Iu interaface • Test Case scope: CMM faulty switchover during traffic • Pre-requisite: MSS is running with traffic. WO/SP-CMM are running smoothly. • Execution: Issue faulty CMM switchover for WO-CMM (simulate hw failure)

• ExpRes: CMM is 2N redundant unit, CMM is the heart of the system (main DB copies are stored in its memory). If WO-CMM goes faulty switch over should happen without any traffic outage. SP-CMM should take over the traffic and the tasks smoothly 12 06/06/2016 Public

© Nokia 2014

1.0 Kuisma_Szell - DocID

Explaining Exploratory test from scripted testing exploratory path Abnormal setup to SUT

TEST CASE

Test setup for SUT

misconfigure feature X

traditional path

Test setup for SUT

Use of abnormal conditions in variables Simultaneous test event Distracting the SUT Modify configuration 13 06/06/2016 Public

© Nokia 2014

Test execution

Test execution

Test execution

OK result

NOK result

OK result

minor fault

1.0 Kuisma_Szell - DocID

major fault

critical fault

15

Bug Hunting

14 © Nokia 2016 Nokia internal use

Bug Hunting When can we apply Bug Hunting - Based on “Go on Bug Hunt” – from Klaus Olsen, FiSTB Testing Assembly, 2013 • When a new release of software is ready for test, a Bug Hunt will very clearly read the temperature ~ quality of the software. • As an entry-criteria for new phases. View it as a ”smoke test” executed by people, instead of automated test, if you don´t have any of these. • As team-motivation, when test execution becomes day to day work, and the auto-pilot is taking over, a Bug Hunt can be what you need to add extra adrenalin to your test.

15 06/06/2016 Public

© Nokia 2014

1.0 Kuisma_Szell - DocID

Bug Hunting Ingredients • 4 dissimilar test systems

• 4x test pairs

SUT1

• 4 x 45 minutes testing rounds • Exploratory Testing • Scope of Hunting

SUT4

SUT2

• Chat tool • Referee, Coach • Prize

SUT3

16 06/06/2016 Public

© Nokia 2014

1.0 Kuisma_Szell - DocID

Bug Hunting Steps 1) Define

2) Go for Hunting

3) Inform

Pre-conditions for bug hunting: scope, environments, duration, limitations, known issues

Use all the possible methods and scenarios. Do fault attacks, error guessing, pair working, parallel testing..

Briefly report all findings to referee. Don’t pause hunting for fault investigation

4) Judge

5) Analyze

6) Report

Referee makes notes of all issues and informs if issue is duplicate.

Categorize issues. Investigate more for unclear issues

All clear faults reported

17 06/06/2016 Public

© Nokia 2014

1.0 Kuisma_Szell - DocID

Bug Hunting Go for Hunting - Groups and Environments Performance Test / Robustness Test teams (ESPOO): • ORANGE Team: • BLACK Team:

(SUT86); 2 person – I, I, I, I, I

~ 95 issues found 3 Major, 30 Minor Faults

(SUT33); 2 person – I, I, I

• MAGENTA Team: (SUT64); 4 person – I, I, I, I, I, I, I, I, I, I, I Performance Test / Robustness Test teams (BUDAPEST): • RED Team:

(SUT76): 3 person – I, I, I, I, I, I, I, I, I, I, I, I, I, I, I

• BLUE Team:

(SUT87): 4 person – I, I, I, I, I, I, I

E2E product teams (BUDAPEST): • GREEN Team:

(SUT5/6):3 person – I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I

• Yellow Team:

(SUT7-8): 4 person – I, I, I, I, I, I, I

• PURPLE Team:

(SUT71-72) :4 person – I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I

• Azure Team:

(SUT53-54): 4 person – I, I, I, I, I, I, I

18 06/06/2016 Public

© Nokia 2014

1.0 Kuisma_Szell - DocID

Bug hunting Metrics Different issues detected with 3 hours of bug hunting

Issues detected by most effective pair during 3 hours of bug hunting

90+ Fault reports created after 3 hours testing by whole team

20+ Fault reports done daily by system testing team with traditional way

33 19 06/06/2016 Public

© Nokia 2014

Fault reports created by 6 Product Lines so far with 8 bug hunting events

1.0 Kuisma_Szell - DocID

1-3

100+

Bug Hunting Planning, scheduling and administration

Planning • Plan all Sessions up-front as part of project • Estimate time for the session, preparation and analysis • Communicate to stakeholders 20 06/06/2016 Public

© Nokia 2014

Schedule • As part of Sprint Review • Before Delivering to next stage (or customer) • As quick entry and exit quality check

1.0 Kuisma_Szell - DocID

Administration • Each Session is 1 Test Case in Test Management tool, with fix time (estimate) • Report faults • Create new test case for opened faults as needed

Bug Hunting Misuse – to be avoided

• Negative scoring as a punishment for not following rules

• “Hunting session” takes a month • Using only Pre-Automated test cases • Connect Real Incentive to results

• Collecting Bugs beforehand and keep them in your pocket until the Bug Hunting Event

21 06/06/2016 Public

© Nokia 2014

1.0 Kuisma_Szell - DocID

Lessons Learned

A Bug Hunting event …

• • • • •

…is FUN

helps tester to think out of the box improves team spirit and expertise need to be planned up-front in Project Plan, not an ad-hoc event the event need preparation and time to analyze issues found complements requirement based testing

22 © Nokia 2016 Nokia internal use

23 06/06/2016 Public

© Nokia 2014

1.0 Kuisma_Szell - DocID