MTAT.03.159: Software Testing

Lecture 05: Test Lifecycle and Test Levels (Textbook Ch. 6)

Dietmar Pfahl Spring 2016

MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

email: [email protected]

Exam Dates

Capacity Limit: 75 students

• Exam 1: Thursday, 19 May, 10:15-12:00 (room 405) • Students graduating in June have preference! • Exam 2: Thursday, 26 May, 10:15-12:00 (room 405) You must have at least 20 marks from the homework assignments (labs 1-6) to qualify for the exam. You must receive at least 10 marks in the exam to not fail the course. In total, you need at least 50 marks to not fail the course.

MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

Lab 5 – Static Code Analysis Lab 5 (week 33: Apr 13 – Apr 16) – Static Code Analysis (10%) Lab 5 Instructions & Tools Submission Deadlines: • Monday Lab: • Tuesday Labs: • Wednesday Labs: • Thursday Labs:

Sunday, 17 Apr, 23:59 Monday, 18 Apr, 23:59 Tuesday, 19 Apr, 23:59 Wednesday, 20 Apr, 23:59

Instructions

SUT: HospitalSystem

• Penalties apply for late delivery: 50% penalty, if submitted up to 24 hours late; 100 penalty, if submitted more than 24 Static Code Analyzer: hours late FindBugs (Eclipse plugin)

MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

Lab 5 – Static Code Analysis (cont’d) Instructions Static Code Analyzer: FindBugs (Eclipse plugin)

Static Code Analysis: Find issues and determine whether or not it is a false positive

Report: 10 Issues (Bugs?) with Analysis and discussion MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

SUT: HospitalSystem

Issue detected: - True Positive - False Positive - Undecidable?

List of Defects/Issues (categorized)

Lab 5 – Static Code Analysis (cont’d)  Bug finding is about pointing out programming mistakes: bad practice, coding errors, unexpected behavior.  One interesting example of bugs that static analysis can find is null pointer dereference. int i=0; String s = null; If (i>0) { s = "positive"; } If (s.contains("pos")) { System.out.println(s); } This code will compile but at runtime a null pointer exception will be thrown, the String s being null and calling the method contains. Static analysis tools, for instance FindBugs can find this bug and report it.

MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

Lab 5 – Static Code Analysis (cont’d)  Bug finding is about pointing out programming mistakes: bad practice, coding errors, unexpected behavior.  One interesting example of bugs that static analysis can find is security vulnerabilities. Static analysis can be applied to find security flaws in code. With dataflow analysis, it is possible to follow the propagation of input data, and thus detect possible code injection. Example: public static void main(String args[]) { File f = new File(args[0]); f.open(); //... } This program opens a file with an argument entered in command line. The fact that we use this argument to open a file is just an example, the important fact is that we use directly an input without validation, which constitutes a serious security vulnerability. Dataflow analysis can detect this kind of flaw by finding the source of data inputs, following their propagation until their use in a sensitive instruction (like creating a File object).

Source: http://blog.zenika.com/2012/08/23/static-analysis-part-14/

MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

Structure of Lecture 5 • Test Lifecycle • Test Levels (Ch. 6)

MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

Testing in different Process Types Waterfall model

Agile model(s)

Programmers Tester Customer

Programmer

Programmer

Idea: Testing in collaboration Testers MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

V-model Acceptance testing

Requirements

Functional specification

System testing

Integration testing

Bu ild

Te s

t

Architecture design

Module design

Unit testing

Coding

MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

V-model Acceptance testing

Requirements

Functional specification

System testing

Integration testing

Bu ild

Te s

t

Architecture design

Module design

Bad: •

–

Relies on the existence, accuracy, and timeliness of documentation Asserts testing on each level is designed based on the deliverables of a single design phase

Communicates change poorly –

•

Coding

Document driven –

•

Unit testing

Does not show how changes, fixes, and test rounds are handled (rework!)

Based on simplistic waterfall model (‘Big Bang’) – –

Testing windows get squeezed Difficult to fit into iterative development

MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

Good: •

Intuitive and easy to explain – – –

Matches the familiar waterfall model Makes a good model for training of people Shows how testing is related to other phases/activities of the waterfall process

Testing in Time-Paced Development • Testing is not a phase – But part of each development task

• Software is developed incrementally – It has to be tested incrementally

• Rapid, time-paced development – Time-boxed releases and increments – Deadlines are not flexible

• Scope is flexible – Testing provides information for scoping

MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

Acceptance testing

Requirements

Functional specification

System testing

Architecture design

Integration testing

Module design

Unit testing

Coding

Idea: Time-paced development? – Time is fixed, scope changes, e.g. Scrum • • • •

30 days to complete iteration or sprint 90 days to complete release 1 90 days to complete release 2 180 days for whole project

MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

Approaches to Testing in Time Paced Development • Automated regression testing – Automated unit testing – Test-driven development – Daily builds and automated tests

• Stabilisation phase or increment – Feature freeze – Testing and debugging at the end of the increment or release

• Separate system testing – Independent testing – Separate testers or test team Testing Increments MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

A Combined Testing Approach

MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

TDD

Challenges – … and how to cope with them • Requirements change all the time • Specifications are never final • Code is never ‘finished’, never ready for testing • Not enough time to test • Need to regression test everything in each increment • Developers always break things again • How can we trust? MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

• Let them change, test design is part of each task

• Focus on developing ‘finished’ increments, tracking on the task level • Testing is part of each development task • Trust comes from building-in the quality, not from the external testing ‘safety net’ • Automation is critical

Structure of Lecture 5 • Test Lifecycle • Test Levels (Ch. 6)

MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

Test Levels Unit/Component Level System Level

MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

Test Level vs Programming Paradigm Level Unit Module

OO Method Class

Integration

Cluster of classes System

System

MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

Procedural Function or procedure Group of functions or procedures Subsystem System

Test levels vs programming paradigm Level Unit Module

OO Method Class

Integration

Cluster of classes System

System

MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

Today often called: Procedural -Function Intra-method testing or procedure -Group Inter-method of functionstesting or -procedures Intra-class (Class) testing -Subsystem Inter-class testing System

Class Testing Basics • Considers the encapsulated class as smallest testable unit • Tests each operation as part of a class hierarchy (because the class hierarchy defines the operation’s context of use) • Approach: • Test each method (and constructor) within a class • Test the state behavior (attributes) of the class between methods

• How is class testing different from conventional testing? • Conventional testing focuses on input-process-output (black-box testing), whereas class testing focuses on each method, then designing sequences of methods to exercise states of a class • White-box testing can still be applied

MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

Class Test Case Design 1. Identify each test case uniquely Associate test case explicitly with the class and/or method to be tested

2. State the purpose of the test What behavior is in focus of test? Coverage criteria?

3. Each test case should contain: a. A list of messages and operations that will be exercised as a consequence of the test b. A list of exceptions that may occur as the object is tested c. A list of external conditions for setup (i.e., changes in the environment external to the software that must exist in order to properly conduct the test) d. Supplementary information that will aid in understanding or implementing the test

Automated unit testing tools help facilitate these requirements

MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

Random Class Testing 1.

Identify methods applicable to a class

2.

Define constraints on their use – e.g. the class must always be initialized first

3.

Identify a minimum test sequence – an operation sequence that defines the minimum life history of the class (object)

4.

Generate a variety of random (but valid) test sequences – this exercises more complex class instance life histories

Example: 1.

An account class in a banking application has open, setup, deposit, withdraw, balance, summarize and close methods

2.

The account must be opened first and closed on completion

3.

Open – setup – deposit – withdraw – close

4.

Open – setup – deposit –* [deposit | withdraw | balance | summarize] – withdraw – close. Generate random test sequences using this template

MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

Challenges of Class Testing • Encapsulation:  Difficult to obtain a snapshot of a class without building extra methods which display the classes’ state Scaffolding

• Inheritance and polymorphism:  Each new context of use (subclass) requires re-testing because a method may be implemented differently (polymorphism).  Other unaltered methods within the subclass may use the redefined method and need to be tested

• White-box tests:  Basis path, condition, data flow and loop tests can all apply to individual methods, but don’t test interactions between methods  Need to construct CFGs connecting methods MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

Scaffolding

Test Harness (automated test framework) In Labs 2 and 3 you wrote drivers for unit tests. You did not make any stubs

MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

Scaffolding

Tool example: JUnit Driver

Classes to be tested Tool example: JMockIt Stubs (c) 2008 Mauro Pezzè & Michal Young

MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

import org.junit.*; import static org.junit.Assert.*; import java.util.*;

Random Class Testing public class JunitTest1 {

private Collection collection;

@BeforeClass public static void oneTimeSetUp() { // one-time initialization code System.out.println("@BeforeClass - oneTimeSetUp"); } @AfterClass public static void oneTimeTearDown() { // one-time cleanup code System.out.println("@AfterClass - oneTimeTearDown"); }

@Before public void setUp() { collection = new ArrayList(); System.out.println("@Before - setUp"); } @After public void tearDown() { collection.clear(); System.out.println("@After - tearDown"); } @Test public void testEmptyCollection() { assertTrue(collection.isEmpty()); System.out.println("@Test - testEmptyCollection"); } @Test public void testOneItemCollection() { collection.add("itemA"); assertEquals(1, collection.size()); System.out.println("@Test - testOneItemCollection"); } }

MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

import org.junit.*; import static org.junit.Assert.*; import java.util.*;

Random Class Testing public class JunitTest1 {

private Collection collection;

@BeforeClass public static void oneTimeSetUp() { // one-time initialization code System.out.println("@BeforeClass - oneTimeSetUp"); } @AfterClass public static void oneTimeTearDown() { // one-time cleanup code System.out.println("@AfterClass - oneTimeTearDown"); }

@Before public void setUp() { collection = new ArrayList(); System.out.println("@Before - setUp"); } @After public void tearDown() { collection.clear(); System.out.println("@After - tearDown"); } @Test public void testEmptyCollection() { assertTrue(collection.isEmpty()); System.out.println("@Test - testEmptyCollection"); } @Test public void testOneItemCollection() { collection.add("itemA"); assertEquals(1, collection.size()); System.out.println("@Test - testOneItemCollection"); } }

MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

@BeforeClass - oneTimeSetUp @Before - setUp @Test - testEmptyCollection @After - tearDown @Before - setUp @Test - testOneItemCollection @After - tearDown @AfterClass - oneTimeTearDown

OO-Testing Approaches

MTAT.03.159 / Lecture 02 / © Dietmar Pfahl 2015

(c) 2008 Mauro Pezzè & Michal Young

Intra-class State Machine Testing Basic idea: • The state of an object is modified by operations

• Methods can be modeled as state transitions • Test cases are sequences of method calls that traverse the state machine model

• State machine model can be derived from specification (functional testing), code (structural testing), or both MTAT.03.159 / Lecture 02 / © Dietmar Pfahl 2015

Class Testing and State push(s, elem1) Show_top(s) -> elem1 pop(s, x) -> x=elem1 pop(s, x) -> x=? The result of a method call depends on the state of the stack

MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

Stack Example

How many test cases needed to cover all feasible branches?

MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

Stack Example

How many test cases needed to cover all feasible branches?

4 decisions MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

Stack Example

How many test cases needed to cover all feasible branches?

4 decisions MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

Stack Example

2 test cases are enough:

MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

Stack Example D1 D2 2 test cases are enough:

D3 D4 MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

Stack Example D1 D2 2 test cases are enough:

D3 D4 MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

D3: false

Stack Example D1 D2 2 test cases are enough:

D3 D4 MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

D1: false D2: true D1: false D2: true D1: false D2: true D1: true D4: true ... false D3: true

Stack Example – CFGs

true

true

if false if false

if true

for true

MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

false

false

Stack Example – CFG (Class)

pop()

Stack()

if true

Stack

false

push(x)

if

true

false

resize() true for true

MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

false

if false

State Diagram for Stack Example

non_exist

Stack()

pop()

empty

error

(size = 0)

push(x)

c

[size = 1] pop() / size-not_empty

[size < length] push(x) / size++

(0 < size 1] pop() / size--

Customer

Account

1

0..*

1

USAccount

Order

OtherAccount

1

*

*

1

*

*

CustomerCare

JPAccount

EUAccount

Package *

LineItem

UKAccount SimpleItem

CompositeItem

*

*

*

*

Class diagram of a more complex example program ...

Model

*

1

Component

PriceList

1

*

0..1 *

Slot

1 ModelDB

* 1

1

SlotDB

ComponentDB

(c) 2008 Mauro Pezzè & Michal Young CSVdb

(c) 2008 Mauro & Michal MTAT.03.159 / Lecture 02 /Pezzè © Dietmar PfahlYoung 2015

 Ch 15, slide 65

Informal State-full Specifications

Computer

Model Slot1

Comp1

Slot2

Comp2

Example: Informal description of a simple class Slot taken from the specification of the web-shop of some example company selling configurable computer models. Slot: represents a slot of a computer model. .... slots can be bound or unbound. Bound slots are assigned a compatible component, unbound slots are empty. Class Slot offers the following services:

• Install: slots can be installed on a model as required or optional. ...

• Bind: slots can be bound to a compatible component. ... • UnBind: bound slots can be unbound by removing the bound component. • IsBound: returns the current binding, if bound; otherwise returns the special value empty. MTAT.03.159 / Lecture 02 / © Dietmar Pfahl 2015

Identifying States and Transitions •

•

From the informal specification we can identify three states: •

Not_installed

•

Unbound

•

Bound

and four transitions •

install: from Not_installed to Unbound

•

bind: from Unbound to Bound

•

unBind: from Bound to Unbound

•

isBound: does not change state

MTAT.03.159 / Lecture 02 / © Dietmar Pfahl 2015

Deriving an FSM and test cases isBound

incorporate install

unBind 0 Not present Not_installed

1

2 Unbound

Bound bind

unBind

TC-1: install, isBound, bind, isBound TC-2: install, unBind, bind, unBind, isBound

MTAT.03.159 / Lecture 02 / © Dietmar Pfahl 2015

isBound

Testing with State Diagrams • A statechart (called a “state diagram” in UML) may be produced as part of a specification or design • May also be implied by a set of message sequence charts (interaction diagrams), or other modeling formalisms

• Two options: •

Convert (“flatten”) into standard finite state machine (FSM), then derive test cases

•

Use state diagram model directly (-> hierarchically nested state machine (HSM)

MTAT.03.159 / Lecture 02 / © Dietmar Pfahl 2015

class model

Statechart specification noModelSelected

super-state or selectModel(model) _________________ “OR-state” send modelDB: getModel(modelID,this)

deselectModel()

method of class Model

modelSelected addComponent(slot, component) _________________________ send mopdelDB: findComponent() send slot:bind() addComponent(slot, component) _________________________ send Component_DB: get_component() send slot:bind

MTAT.03.159 / Lecture 02 / © Dietmar Pfahl 2015

workingConfiguration isLegalConfiguration() [legalConfig = true]

validConfiguration

removeComponent(slot) _________________________ send slot:unbind()

removeComponent(slot) _________________________ send slot:unbind()

called by class Model

From Statecharts to FSMs noModelSelected

selectModel(model) addComponent(slot, component)

addComponent(slot, component)

deselectModel()

workingConfiguration isLegalConfiguration() [legalConfig=true] validConfiguration

MTAT.03.159 / Lecture 02 / © Dietmar Pfahl 2015

removeComponent(slot) deselectModel() removeComponent(slot)

Finite State Machine (FSM) State and transition explosion due to frequently repeated transitions (clear, off)

Hierarchically nested State Machine (HSM) Cf. statecharts and UML state diagrams

(c) 2008 Mauro & Michal MTAT.03.159 / Lecture 02 /Pezzè © Dietmar PfahlYoung 2015

 Ch 15, slide 72

Summary: Statechart-based testing • In some cases, “flattening” a Statechart to a finitestate machine may cause “state explosion” • Particularly for super-states with “history”

• Alternative: Use the statechart directly • Simple transition coverage: execute all transitions of the original Statechart • incomplete transition coverage of corresponding FSM • useful for complex statecharts and strong time constraints (combinatorial number of transitions)

MTAT.03.159 / Lecture 02 / © Dietmar Pfahl 2015

Intra-class data flow testing • Exercise sequences of methods • From setting or modifying a field value

• To using that field value

• We need a control flow graph that encompasses more than a single method ...

MTAT.03.159 / Lecture 02 / © Dietmar Pfahl 2015

The intra-class control flow graph Model()

Control flow for each method + Method node for class addComponent + edges from node class to the start nodes of the methods from the end nodes of the methods to node class

1.1

void selectModel(String modelID)

boolean legalConfig = false

1.2

openDB()

ModelDB modelDB = null

1.3

modelDB.getModel(modelID, this)

modelID = NoModel

exit Model

1.4

2.1

2.2

exit selectModel

2.4

3.1

modelID = NoModel

3.2

Method selectModel 2.3

longName = “No ...selected.”

slot = null

1.5

3.5

class Model

void removeComponent(int slotIndex) 5.1 void addComponent(int slotIndex, String sku) 4.1 if (slots[slotIndex].isBound() (componentDB.contains(sku))

5.2

4.2 True

True

slots[slotIndex].unbind() Component comp = new Component(order, sku)

4.5

legalConfig = false; slot.unbind();

False

legalConfig = false

5.4

exit removeComponent

5.5

4.4

True

False

5.3

4.3

Method checkConfiguration (comp.isCompatible(slot.slotID))

slot.unbind();

slot.bind(comp)

4.7

4.6

4.8

void checkConfiguration()

legalConfig = false;

legalConfig = true

4.9

exit addCompoment

int i = 0

4.10

True

checkCongfiguration()

if (!isLegalConfig) 7.3

7.1

6.2

6.4

True

7.2

Slot slot = slots[i]

6.5

False

++i return legalConfig

6.1

6.3

i < slot.length boolean isLegalConfiguration()

class Model

3.3

3.4

exit deselectModel

False

=> control flow through sequences of method calls

void deselectModel()

False

6.6

False

7.4 if (slot.required && ! slot.isBound()

6.7

True

legalConfig = false

MTAT.03.159 / Lecture 02 / © Dietmar Pfahl 2015

6.8

exit checkConfiguration 6.9

Definition-Use (DU) pairs instance variable legalConfig Each pair corresponds to a test case note that some pairs may be infeasible to cover pairs we may need to find complex sequences

MTAT.03.159 / Lecture 02 / © Dietmar Pfahl 2015

Definition-Use (DU) pairs Model()

instance variable legalConfig

Boolean legalConfig = false

1.1

void selectModel(String modelID)

boolean legalConfig = false

1.2

openDB()

ModelDB modelDB = null

1.3

modelDB.getModel(modelID, this)

modelID = NoModel

exit Model

1.4

2.1

2.2

exit selectModel

2.3

void deselectModel()

3.1

modelID = NoModel

3.2

longName = “No ...selected.”

slot = null

2.4

1.5

3.4

exit deselectModel

3.5

class Model

void removeComponent(int slotIndex) 5.1 void addComponent(int slotIndex, String sku) 4.1 if (slots[slotIndex].isBound() (componentDB.contains(sku))

5.2

4.2 True

True

slots[slotIndex].unbind() Component comp = new Component(order, sku)

(comp.isCompatible(slot.slotID))

4.5

legalConfig = false; slot.unbind();

False

legalConfig = false

5.4

exit removeComponent

5.5

4.4

True

False

slot.unbind();

5.3

4.3

False

slot.bind(comp)

4.7

4.6

4.8

void checkConfiguration()

legalConfig = false;

legalConfig = true

4.9

exit addCompoment

int i = 0

4.10

True

checkCongfiguration()

if (!isLegalConfig) 7.3

7.1

6.2

6.4

True

7.2

Slot slot = slots[i]

6.5

False

++i return legalConfig

6.1

6.3

i < slot.length boolean isLegalConfiguration()

If (!isLegalConfig)

3.3

False

6.6

False

7.4 if (slot.required && ! slot.isBound()

6.7

True

legalConfig = false

MTAT.03.159 / Lecture 02 / © Dietmar Pfahl 2015

6.8

exit checkConfiguration 6.9

Inspectors and modifiers • Classify methods (execution paths) as •

inspectors: use, but do not modify, instance variables

•

modifiers: modify, but not use instance variables

•

inspector/modifiers: use and modify instance variables

• Example – class Slot: •

Slot()

modifier

•

bind()

modifier

•

unBind() modifier

•

isBound() inspector

MTAT.03.159 / Lecture 02 / © Dietmar Pfahl 2015

Note: This is different to the previous slide where only methods of the same class were involved

Definitions from modifiers Definitions of instance variable slot in class model

void addComponent(int slotIndex, String sku) 4.1

(componentDB.contains(sku))

addComponent (4.5)

True

addComponent (4.7) addComponent (4.8)

Component comp = new Component(order, sku)

(comp.isCompatible(slot.slotID))

slot.unbind();

4.5

legalConfig = false; slot.unbind();

4.6

4.9

exit addCompoment

MTAT.03.159 / Lecture 02 / © Dietmar Pfahl 2015

slot.bind(comp)

4.8

legalConfig = false;

4.4

True

False

removeComponent (5.3)

modifier modifier modifier inspector

4.3

False

selectModel (2.3)

Slot() bind() unBind() isBound()

4.2

4.10

4.7

Uses from inspectors void checkConfiguration()

Uses of instance variable slot in class model

6.1

Slot slot =slots[slotIndex]; legalConfig = true

removeComponent (5.2) checkConfiguration (6.4)

int i = 0

6.2

6.3

checkConfiguration (6.5) i < slot.length

checkConfiguration (6.7)

True

Slot slot = slots[i]

Slot() bind() unBind() isBound()

6.4

modifier modifier modifier inspector

++i

6.5 False

6.6

False

if (slot.required && ! slot.isBound()

6.7

True

legalConfig = false

MTAT.03.159 / Lecture 02 / © Dietmar Pfahl 2015

6.8

exit checkConfiguration 6.9

Integration Testing • More than one (tested) unit • Detecting defects – On the interfaces of units – Communication between units • Helps assembling incrementally a whole system • Non-functional aspects if possible • Done by developers/designers or independent testers – Preferably developers and testers in collaboration • Often omitted due to setup difficulties – Time is more efficiently spent on unit and system tests MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

Integration Testing – Procedural

MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

Top-down testing Level 1

Testing sequence

Level 2 Le vel 2 stubs

Le vel 3 stubs

MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

Level 1

Level 2

Le vel 2

. ..

Level 2

Bottom-up testing Test drivers Level N

Test drivers

Level N

Level N–1

MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

Le vel N

Level N–1

Level N

Level N

Level N–1

Testing sequence

Incremental integration testing A

T1

T1

A T1

T2

A

T2 T2

B

B

T3 T3

B

C T4

T3 C

T4 T5

D Test sequence 1

Test sequence 2

MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

Test sequence 3

Integration Testing – OO

MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

Summary: Inter-Class Testing • OO does not have a hierarchical control structure so conventional top-down and bottom-up integration tests have little meaning • Integration applies three different incremental strategies:  Thread-based testing: integrates classes required to respond to one input or event  Use-based testing: integrates classes required by one use case  Cluster testing: integrates classes required to demonstrate one collaboration

MTAT.03.159 / Lecture 02 / © Dietmar Pfahl 2015

Interactions in Inter-Class Tests • Consider all combinations of interactions •

example: a test case for class Order includes a call to a method of class Model, and the called method calls a method of class Slot, exercise all possible relevant states of the different classes

•

problem: combinatorial explosion of cases

•

so select a subset of interactions: • arbitrary or random selection • plus all significant interaction scenarios that have been previously identified in design and analysis: sequence + collaboration diagrams

MTAT.03.159 / Lecture 02 / © Dietmar Pfahl 2015

sequence diagram O:Order

C20:Model

ChiMod:ModelDB

C20Comp:Compoment

C20slot:Slots

ChiSlot:SlotDB

ChiComp:ComponentDB

selectModel() getmodel(C20)

select()

extract(C20)

addCompoment(HD60) contains(HD60) found isCompatible(HD60) incompatible fail

addCompoment(HD20) contains(HD20) found

isCompatible(HD20) compatible bind success

(c) 2008 Mauro Pezzè & Michal Young

MTAT.03.159 / Lecture 02 / © Dietmar Pfahl 2015

Ch 15, slide 92

Summary: Inter-Class structural testing • Working “bottom up” in dependence hierarchy • Dependence is not the same as class hierarchy; not always the same as call or inclusion relation. • May match bottom-up build order

•

Starting from leaf classes, then classes that use leaf classes, ...

• Summarize effect of each method: Changing or using object state, or both •

Treating a whole object as a variable (not just primitive types)

MTAT.03.159 / Lecture 02 / © Dietmar Pfahl 2015

System testing • Testing the system as a whole • Functional – Functional requirements and requirements-based testing

• Non-functional – Performance, stress, configuration, security, ... – As important as functional requirements – Often poorly specified – Must be tested

• Often done by independent test group – Collaborating developers and testers MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

Types of system testing

MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

Regression testing • Testing changed software – Retest all – Selective regression testing • Regression test suite • Types of changes – Defect fixes – Enhancements – Adaptations [Skoglund, Runeson, ISESE05] – Perfective maintenance MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

Retest all • Assumption: – Changes may introduce faults anywhere in the code • BUT: expensive, prohibitive for large systems • Reuse existing test suite • Add new tests as needed • Remove obsolete tests

MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

Selective regression testing • Impact analysis • Only code impacted by change needs to be retested • Select tests that exercise such code • Add new tests if needed • Remove obsolete tests MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

Acceptance testing (α, β) • Final stage of validation – Customer (user) should perform or be closely involved – Customer can perform any test they wish – Final user sign-off

• Approach – Mixture of scripted and unscripted testing – Performed in real operation environment

MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

• Project – Contract acceptance – Customer viewpoint – Validates that the right system was built

• Product – Final checks on releases – User viewpoint throughout the development; validation

Recommended Textbook Exercises • Chapter 6 – 1, 3, 6, 7, 8, 9, 12

MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016

Next 2 Weeks • Lab 5: – Static Code Analysis

• Lecture 6: – Test Tools, Metrics, Documentation, Organisation and Process Improvement (Test Maturity Model)

• In addition to do: – Read textbook chapters

MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2016