Quality Assurance in Software Development

Institute for Software Technology t Quality Assurance in Software Development Qualitätssicherung in der Softwareentwicklung A.o.Univ.-Prof. Dipl.-I...
Author: Marylou Ford
0 downloads 0 Views 422KB Size
Institute for Software Technology

t

Quality Assurance in Software Development Qualitätssicherung in der Softwareentwicklung

A.o.Univ.-Prof. Dipl.-Ing. Dr. Bernhard Aichernig Institute for Software Technology Graz University of Technology Austria

Summer Term 2016

B. Aichernig

Quality Assurance in Software Development 1 / 47

Institute for Software Technology

t

Agenda

1

The Challenge of Software Testing

2

Definitions

3

Functional testing strategies Boundary Value Testing Random Testing Equivalence Class Testing

B. Aichernig

Quality Assurance in Software Development 2 / 47

Institute for Software Technology

t

The Challenge of Software Testing

I

Correctness

I

Requirements

I

Development Process

B. Aichernig

Quality Assurance in Software Development 3 / 47

Institute for Software Technology

t

A silly joke

I

A mathematician, a physicist, and an engineer are told: “All odd numbers are prime.”

I

The mathematician says, “That’s silly, nine is a non-prime odd number.”

I

The physicist says, “Let’s see, 3 is prime, 5 is prime, 7 is prime — looks like it’s true.”

I

The engineer says, “Let’s see, 3 is prime, 5 is prime, 7 is prime, 9 is prime, 11 is prime — looks like it’s true.”

B. Aichernig

Quality Assurance in Software Development 4 / 47

Institute for Software Technology

t

Becoming more serious

The little joke illustrates us the following challenges in testing: I

oracle problem: How to predict the outcome of a test-case?

I

incompleteness: exhaustive testing is usually infeasable. How much testing is needed?

I

test case selection: Are the chosen test cases adequate?

I

correctness of test cases: Are the test cases correctly designed?

B. Aichernig

Quality Assurance in Software Development 5 / 47

Institute for Software Technology

t

Testing and Correctness

I

“Testing can only demonstrate the presence of errors, not their absence” (E. Dijkstra, 1972)

I

Reason: discrete nature of software

I

Testing of all possible inputs would be necessary

I

Exhaustive testing is infeasible

I

Representative test cases have to be selected (testing strategy)

B. Aichernig

Quality Assurance in Software Development 6 / 47

Institute for Software Technology

t

Testing and Requirements I

Quality of testing strongly depends on the quality of requirement documents

I

Requiremens that are not documented cannot be tested systematically

I

60 % of all bugs in the requirements phase are due to missing requirements Requirements written in natural language are often

I

I I I

I

B. Aichernig

incomplete ambiguous inconsistent

Formal specifications and formal models help!

Quality Assurance in Software Development 7 / 47

Institute for Software Technology

t

Testing in the Development Process

I

In the classical waterfall model testing is the last phase.

I

Danger: if project is delayed, testing gets cut

I

Dramatic: most of the large SW projects overrun their deadline

I

Consequence: the test cases have to be designed as early as possible, as soon as the SW requirements are known.

I

this leads to agile software processes

I

High testing costs!

I

Test automation is needed!

B. Aichernig

1/3 – 1/2 of the resources in big projects

Quality Assurance in Software Development 8 / 47

Institute for Software Technology

t

Self test

Write a set of test cases that you feel would adequately test this program (Myers, The Art of Software Testing): I

The program reads three integer values from an input stream.

I

The three values are interpreted as representing the lengths of the sides of a triangle.

I

The program prints a message that states whether the triangle is scalene, isosceles, or equilateral.

B. Aichernig

Quality Assurance in Software Development 9 / 47

Institute for Software Technology

t

Self test: questions Do you have ... 1

a test case that represents a valid scalene triangle? Test cases (1, 2, 3) and (2, 5, 10) are not valid.

2

a test case that represents a valid equilateral triangle?

3

a test case that represents a valid isosceles triangle?

4

at least three test cases that represent valid isosceles triangles such that you have tried all three permutations of two equal sides (e.g., (3, 3, 4), (3, 4, 3), and (4, 3, 3))?

5

a test case in which one side has a zero value?

6

a test case in which one side has a negative value?

7

a test case with three integers greater than zero such that the sum of two of the numbers is equal to the third?

B. Aichernig

Quality Assurance in Software Development 10 / 47

Institute for Software Technology

t

Self test: questions Do you have ... 1

a test case that represents a valid scalene triangle? Test cases (1, 2, 3) and (2, 5, 10) are not valid.

2

a test case that represents a valid equilateral triangle?

3

a test case that represents a valid isosceles triangle?

4

at least three test cases that represent valid isosceles triangles such that you have tried all three permutations of two equal sides (e.g., (3, 3, 4), (3, 4, 3), and (4, 3, 3))?

5

a test case in which one side has a zero value?

6

a test case in which one side has a negative value?

7

a test case with three integers greater than zero such that the sum of two of the numbers is equal to the third?

B. Aichernig

Quality Assurance in Software Development 10 / 47

Institute for Software Technology

t

Self test: questions Do you have ... 1

a test case that represents a valid scalene triangle? Test cases (1, 2, 3) and (2, 5, 10) are not valid.

2

a test case that represents a valid equilateral triangle?

3

a test case that represents a valid isosceles triangle?

4

at least three test cases that represent valid isosceles triangles such that you have tried all three permutations of two equal sides (e.g., (3, 3, 4), (3, 4, 3), and (4, 3, 3))?

5

a test case in which one side has a zero value?

6

a test case in which one side has a negative value?

7

a test case with three integers greater than zero such that the sum of two of the numbers is equal to the third?

B. Aichernig

Quality Assurance in Software Development 10 / 47

Institute for Software Technology

t

Self test: questions Do you have ... 1

a test case that represents a valid scalene triangle? Test cases (1, 2, 3) and (2, 5, 10) are not valid.

2

a test case that represents a valid equilateral triangle?

3

a test case that represents a valid isosceles triangle?

4

at least three test cases that represent valid isosceles triangles such that you have tried all three permutations of two equal sides (e.g., (3, 3, 4), (3, 4, 3), and (4, 3, 3))?

5

a test case in which one side has a zero value?

6

a test case in which one side has a negative value?

7

a test case with three integers greater than zero such that the sum of two of the numbers is equal to the third?

B. Aichernig

Quality Assurance in Software Development 10 / 47

Institute for Software Technology

t

Self test: questions Do you have ... 1

a test case that represents a valid scalene triangle? Test cases (1, 2, 3) and (2, 5, 10) are not valid.

2

a test case that represents a valid equilateral triangle?

3

a test case that represents a valid isosceles triangle?

4

at least three test cases that represent valid isosceles triangles such that you have tried all three permutations of two equal sides (e.g., (3, 3, 4), (3, 4, 3), and (4, 3, 3))?

5

a test case in which one side has a zero value?

6

a test case in which one side has a negative value?

7

a test case with three integers greater than zero such that the sum of two of the numbers is equal to the third?

B. Aichernig

Quality Assurance in Software Development 10 / 47

Institute for Software Technology

t

Self test: questions Do you have ... 1

a test case that represents a valid scalene triangle? Test cases (1, 2, 3) and (2, 5, 10) are not valid.

2

a test case that represents a valid equilateral triangle?

3

a test case that represents a valid isosceles triangle?

4

at least three test cases that represent valid isosceles triangles such that you have tried all three permutations of two equal sides (e.g., (3, 3, 4), (3, 4, 3), and (4, 3, 3))?

5

a test case in which one side has a zero value?

6

a test case in which one side has a negative value?

7

a test case with three integers greater than zero such that the sum of two of the numbers is equal to the third?

B. Aichernig

Quality Assurance in Software Development 10 / 47

Institute for Software Technology

t

Self test: questions Do you have ... 1

a test case that represents a valid scalene triangle? Test cases (1, 2, 3) and (2, 5, 10) are not valid.

2

a test case that represents a valid equilateral triangle?

3

a test case that represents a valid isosceles triangle?

4

at least three test cases that represent valid isosceles triangles such that you have tried all three permutations of two equal sides (e.g., (3, 3, 4), (3, 4, 3), and (4, 3, 3))?

5

a test case in which one side has a zero value?

6

a test case in which one side has a negative value?

7

a test case with three integers greater than zero such that the sum of two of the numbers is equal to the third?

B. Aichernig

Quality Assurance in Software Development 10 / 47

Institute for Software Technology

t

Self test: questions (cont.) 8

at least three test cases in Category 7 such that you have tried all three permutations where the length of one side is equal to the sum of the lengths of the other two sides (e.g., (1, 2, 3), (1, 3, 2), and (3, 1, 2))?

9

a test case with three integers greater than zero such that the sum of two of the numbers is less than the third (e.g., (1, 2, 4))

10

at least three test cases in Category 9 such that you have tried all three permutations (e.g., (1, 2, 4), (1, 4, 2), and (4, 1, 2))?

11

a test case in which all sides are zero?

12

at least one test case specifying noninteger values?

13

at least one test case specifying the wrong number of values?

14

for each test case the expected outputs specified?

B. Aichernig

Quality Assurance in Software Development 11 / 47

Institute for Software Technology

t

Self test: questions (cont.) 8

at least three test cases in Category 7 such that you have tried all three permutations where the length of one side is equal to the sum of the lengths of the other two sides (e.g., (1, 2, 3), (1, 3, 2), and (3, 1, 2))?

9

a test case with three integers greater than zero such that the sum of two of the numbers is less than the third (e.g., (1, 2, 4))

10

at least three test cases in Category 9 such that you have tried all three permutations (e.g., (1, 2, 4), (1, 4, 2), and (4, 1, 2))?

11

a test case in which all sides are zero?

12

at least one test case specifying noninteger values?

13

at least one test case specifying the wrong number of values?

14

for each test case the expected outputs specified?

B. Aichernig

Quality Assurance in Software Development 11 / 47

Institute for Software Technology

t

Self test: questions (cont.) 8

at least three test cases in Category 7 such that you have tried all three permutations where the length of one side is equal to the sum of the lengths of the other two sides (e.g., (1, 2, 3), (1, 3, 2), and (3, 1, 2))?

9

a test case with three integers greater than zero such that the sum of two of the numbers is less than the third (e.g., (1, 2, 4))

10

at least three test cases in Category 9 such that you have tried all three permutations (e.g., (1, 2, 4), (1, 4, 2), and (4, 1, 2))?

11

a test case in which all sides are zero?

12

at least one test case specifying noninteger values?

13

at least one test case specifying the wrong number of values?

14

for each test case the expected outputs specified?

B. Aichernig

Quality Assurance in Software Development 11 / 47

Institute for Software Technology

t

Self test: questions (cont.) 8

at least three test cases in Category 7 such that you have tried all three permutations where the length of one side is equal to the sum of the lengths of the other two sides (e.g., (1, 2, 3), (1, 3, 2), and (3, 1, 2))?

9

a test case with three integers greater than zero such that the sum of two of the numbers is less than the third (e.g., (1, 2, 4))

10

at least three test cases in Category 9 such that you have tried all three permutations (e.g., (1, 2, 4), (1, 4, 2), and (4, 1, 2))?

11

a test case in which all sides are zero?

12

at least one test case specifying noninteger values?

13

at least one test case specifying the wrong number of values?

14

for each test case the expected outputs specified?

B. Aichernig

Quality Assurance in Software Development 11 / 47

Institute for Software Technology

t

Self test: questions (cont.) 8

at least three test cases in Category 7 such that you have tried all three permutations where the length of one side is equal to the sum of the lengths of the other two sides (e.g., (1, 2, 3), (1, 3, 2), and (3, 1, 2))?

9

a test case with three integers greater than zero such that the sum of two of the numbers is less than the third (e.g., (1, 2, 4))

10

at least three test cases in Category 9 such that you have tried all three permutations (e.g., (1, 2, 4), (1, 4, 2), and (4, 1, 2))?

11

a test case in which all sides are zero?

12

at least one test case specifying noninteger values?

13

at least one test case specifying the wrong number of values?

14

for each test case the expected outputs specified?

B. Aichernig

Quality Assurance in Software Development 11 / 47

Institute for Software Technology

t

Self test: questions (cont.) 8

at least three test cases in Category 7 such that you have tried all three permutations where the length of one side is equal to the sum of the lengths of the other two sides (e.g., (1, 2, 3), (1, 3, 2), and (3, 1, 2))?

9

a test case with three integers greater than zero such that the sum of two of the numbers is less than the third (e.g., (1, 2, 4))

10

at least three test cases in Category 9 such that you have tried all three permutations (e.g., (1, 2, 4), (1, 4, 2), and (4, 1, 2))?

11

a test case in which all sides are zero?

12

at least one test case specifying noninteger values?

13

at least one test case specifying the wrong number of values?

14

for each test case the expected outputs specified?

B. Aichernig

Quality Assurance in Software Development 11 / 47

Institute for Software Technology

t

Self test: questions (cont.) 8

at least three test cases in Category 7 such that you have tried all three permutations where the length of one side is equal to the sum of the lengths of the other two sides (e.g., (1, 2, 3), (1, 3, 2), and (3, 1, 2))?

9

a test case with three integers greater than zero such that the sum of two of the numbers is less than the third (e.g., (1, 2, 4))

10

at least three test cases in Category 9 such that you have tried all three permutations (e.g., (1, 2, 4), (1, 4, 2), and (4, 1, 2))?

11

a test case in which all sides are zero?

12

at least one test case specifying noninteger values?

13

at least one test case specifying the wrong number of values?

14

for each test case the expected outputs specified?

B. Aichernig

Quality Assurance in Software Development 11 / 47

Institute for Software Technology

t

Naming the Wrong IEEE Standard Glossary of Software Engineering Terminology: Mistake: a human action that produces an incorrect result. Example: an incorrect action taken by the programmer. Fault: an incorrect step, process, or data definition in a computer program (also defect, bug). Error: the difference between a computed, observed, or measured value or condition and the true, specified, or theoretically correct value or condition. Failure: the inability of a system or component to fulfill its required functions within specified performance requirements. Dijkstra was against calling it a “bug”.

B. Aichernig

Quality Assurance in Software Development 12 / 47

Institute for Software Technology

t

Naming the Wrong: Example

I

Consider a statement x = y + x;

I

By a mistake a programmer changes it to x = y − x, thus characterising a fault.

I

Executing the fault with x = 0 does not lead to an error,

I

and consequently not to an observable failure.

B. Aichernig

Quality Assurance in Software Development 13 / 47

Institute for Software Technology

t

Testing, Test, Test Case

testing: the act of designing, debugging, and executing test cases.. test: A test is the act of exercising software with test cases. A test has two distinctive goals: to find failures or to demonstrate correct execution. test case: A test case has an identity and is associated with program behaviour. A test case also has a set of inputs and a list of expected outputs. The output portion of a test case is frequently overlooked, which is unfortunate, because it is often the hard part.

B. Aichernig

Quality Assurance in Software Development 14 / 47

Institute for Software Technology

t

Black & White

Two fundamental approaches are used to identify test cases: functional testing: functional testing is based on the view that any program can be considered to be a function that maps values from its input domain to values in its output range (black box testing). Test cases are identified using the function’s specification. structural testing: based on how the function is actually implemented (white box testing).

B. Aichernig

Quality Assurance in Software Development 15 / 47

Institute for Software Technology

t

Why We Test validation: the process of evaluating an object to demonstrate that it meets the user requirements. verification: in the ANSI/IEEE Std 729-1983 three possible definitions of verification are given: 1 the process of reviewing, inspecting, testing, etc., if objects, processes, services, documents satisfy the specified requirements. 2 the process of evaluating if an object in a given phase in the software lifecycle, meets the requirements established in the previous phase. 3 formal correctness proofs of programs. falsification: the process of evaluating an object to demonstrate that it does not meet requirements.

B. Aichernig

Quality Assurance in Software Development 16 / 47

Institute for Software Technology

t

When We Test The following kinds of tests can be distinguished during sw-lifecycle: unit tests: testing of units: modules, classes, components integration tests: tests that explore the interaction and consistency of successfully tested components. system tests: testing of the whole system in order to explore behaviors that can’t be done by unit or integration testing: performance, data integrity, storage management, security, reliability. acceptance tests: does the system satisfy the requirements? Part of the contract. regression tests: tests after changes to prevent unintended changes.

B. Aichernig

Quality Assurance in Software Development 17 / 47

Institute for Software Technology

t

The V-model

B. Aichernig

Quality Assurance in Software Development 18 / 47

Institute for Software Technology

t

Functional Testing: Programs as Functions

I

In functional testing, a program is considered as a function p : Input → Output I I

mapping values from its domain (Input) to values in its range (Output).

I

Types permit the definitions of domain and range.

I

Preconditions restrict the domain further.

I

Postconditions restrict the range.

B. Aichernig

Quality Assurance in Software Development 19 / 47

Institute for Software Technology

t

Boundary Value Analysis

I

Focuses on the boundary of input space (domain) to identify test cases.

I

Rationale: errors tend to occur near the extreme values of an input variable. Strategy: input values at

I

1 2 3 4 5

B. Aichernig

minimum just above minimum a nominal value just below maximum maximum

Quality Assurance in Software Development 20 / 47

Institute for Software Technology

t

Boundary Value Analysis: 2 Input Variables Boundary value test cases for a function of two variables i 1 and i 2: i2 d

c

i1 a

b

Single fault assumption! (no relation between parameters) B. Aichernig

Quality Assurance in Software Development 21 / 47

Institute for Software Technology

t

Generalizing Boundary Value Analysis

I I

Number of test cases: 4n + 1 for a function with n input variables. Different ranges: I I

I

I

Limitations: only good when program is a function of several independent variables that represent bounded physical variables. I I

B. Aichernig

Triangle problem: e.g. min = 1, max = MAXINT Date input: date corresponds to 3 input variables; e.g. min = 1.3.2003, max = 31.3.2003 Boolean input: does not make sense!

good: temperature, coordinates bad: date, pin code, telephone number

Quality Assurance in Software Development 22 / 47

Institute for Software Technology

t

Robustness Testing Exceeding the limits slightly: i2 d

c

i1 a

b

Most interesting with expected outputs not inputs! B. Aichernig

Quality Assurance in Software Development 23 / 47

Institute for Software Technology

t

Robustness Testing Examples

Robustness testing forces attention on exception handling: I

Exceeding angle of attack of a plane: Will it stall?

I

Exceeding load capacity of an elevator: Hopefully only a warning!? With strongly typed languages robustness testing may be very awkward:

I

I I

C leads to run-time errors. Exception handling (C#, Java) mandates robustness testing.

I

Specifications: test cases such that ¬ pre(in) holds.

I

Automated: fuzz testing or fuzzing

B. Aichernig

Quality Assurance in Software Development 24 / 47

Institute for Software Technology

t

Worst-Case Testing Rejecting the single-fault assumption: i2 d

c

i1 a

b

Number of test cases: 5n , for n input variables! B. Aichernig

Quality Assurance in Software Development 25 / 47

Institute for Software Technology

t

Special Value Testing

I

tester uses his I I I

domain knowledge, experience with similar programs, and information about “soft spots” to devise test cases.

I

most widely practiced form.

I

also called ad hoc testing.

I

very dependent on the abilities of the tester!

B. Aichernig

Quality Assurance in Software Development 26 / 47

Institute for Software Technology

t

Example: Commission I

Rifle1 salesman sales rifle locks2 , stocks3 , and barrels4 .

I

Locks costs $45, stocks $30, barrels $25.

I

Has to sell at least one complete rifle per month.

I

Has to sell at most 70 locks, 80 stocks, and 90 barrels per month. The gunsmith computes the salesman’s commission:

I

I I I

10% on sales up to (and including) $1000, 15% on the next $800, and 20% on any sales in excess of $1800.

1 Gewehr 2 Verschluss 3 Schaft 4 Lauf B. Aichernig

Quality Assurance in Software Development 27 / 47

Institute for Software Technology

t

Commission: Output Boundary Analysis

Case 1 2 3 4 5 6 7 8 9 10 11 12 13

B. Aichernig

Locks 1 1 1 2 5 10 10 9 10 10 10 11 14

Stocks 1 1 2 1 5 10 9 10 10 10 11 10 14

Barrels 1 2 1 1 5 9 10 10 10 11 10 10 14

Sales ($) 100 125 130 145 500 975 970 955 1000 1025 1030 1045 1400

Comm ($) 10 12.5 13 14.5 50 97.5 97 95.5 100 103.75 104.5 106.75 160

Comment output minimum output minimum + output minimum + output minimum + midpoint border point border point border point border point border point + border point + border point + midpoint

Quality Assurance in Software Development 28 / 47

Institute for Software Technology

t

Commission: Output Boundary Analysis (cont.)

Case 14 15 16 17 18 19 20 21 22 23 24 25

B. Aichernig

Locks 18 18 17 18 18 18 19 48 70 70 69 70

Stocks 18 17 18 18 18 19 18 48 80 79 80 80

Barrels 17 18 18 18 19 18 18 48 89 90 90 90

Sales ($) 1775 1770 1755 1800 1825 1830 1845 4800 7775 7770 7755 7800

Comm ($) 216.75 215.5 213.25 220 225 226 229 820 1415 1414 1411 1420

Comment border point border point border point border point border point + border point + border point + midpoint output maximum output maximum output maximum output maximum

Quality Assurance in Software Development 29 / 47

Institute for Software Technology

t

Random Testing I

Idea: Random number generator to pick test case values.

I

Motivation: avoiding a form of bias in testing.

I

Question: How many random test cases are sufficient? Answer by 1 2

coverage criteria reliability models

I

Pro: fully automatic test case generation.

I

Con: more test cases needed to reach structural coverage.

I

At least 2 decades of literature in academia (statistical testing methods).

Many popular tools use random testing, e.g. ScalaCheck or FsCheck.

B. Aichernig

Quality Assurance in Software Development 30 / 47

Institute for Software Technology

t

Random Testing of Commission

Test Cases 91 27 72 176 48 152 125 avg.

B. Aichernig

10% Commission 1 1 1 1 1 1 1 1.01%

15% Commission 6 1 1 6 1 6 4 3.62%

20% Commission 84 25 70 169 46 145 120 95.37%

Quality Assurance in Software Development 31 / 47

Institute for Software Technology

t

Random Testing of Triangle

Test Cases 1289 15436 17091 2603 6475 5978 9008 avg.

NoTriangle 663 7696 8556 1284 3197 2998 4447 49.83%

Scalene 593 7372 8164 1252 3122 2850 4353 47.87%

Isosceles 32 367 367 66 155 129 207 2.29%

Equilateral 1 1 1 1 1 1 1 0.01%

I

Program generates random test cases until at least one of each output occurs.

I

Here an upper limit of 200 has been chosen.

B. Aichernig

Quality Assurance in Software Development 32 / 47

Institute for Software Technology

t

Software Reliability Engineering

failure rate =

number of failures logical unit

Logical unit: time, number of printed pages etc. Reliability: propability that system will have no failure for a given time. 1

Decide target failure rate or reliability

2

Operational profile: What use cases (functions) are often executed?

3

More tests for highly frequent use cases.

4

Additional tests for high-risk use cases.

5

Statistical failure models, and the failures detected provide answers, when the target reliability is reached.

Book: John Musa, Software Reliability Engineering, 2nd. edition, 1998.

B. Aichernig

Quality Assurance in Software Development 33 / 47

Institute for Software Technology

t

Equivalence Class Testing Main idea: I

Test hypothesis: classes of inputs show equal behaviour

I

Select one test case from each equivalence class.

Main motivation: I

adequate set of test cases (having a sense of completeness)

I

avoiding redundancy!

We distinguish: I

weak / strong equivalence class testing (single vs. multiple faults)

I

normal / robust equivalence class testing

B. Aichernig

Quality Assurance in Software Development 34 / 47

Institute for Software Technology

t

Equivalence Classes

I

form a set partition.

I

Definition: (Partition) Given a set A and a set of n subsets A1 , A2 , . . . , An of A, the subsets form a partition of A iff A1 ∪ A2 ∪ · · · ∪ An = A and ∀i , j ∈ {1, . . . , n} · i 6= j ⇒ Ai ∩ Aj = {}

I

1st property provides a form of completeness,

I

2nd property ensures a form of nonredundancy.

B. Aichernig

Quality Assurance in Software Development 35 / 47

Institute for Software Technology

t

Choosing Equivalence Classes

I

Key to Strategy: Choice of equivalence relation.

I

Deduced from the problem specification Triangle: one test case for equilateral (5, 5, 5), we don’t expect much more from test cases, like (6, 6, 6), (7, 7, 7).

I

often by guessing a likely implementation

I

Formal interface specifications (contracts): choice of equivalence classes can be automated.

B. Aichernig

Quality Assurance in Software Development 36 / 47

Institute for Software Technology

t

Weak Normal Equivalence Class Testing

i2 d

c

i1 a

B. Aichernig

b

Quality Assurance in Software Development 37 / 47

Institute for Software Technology

t

Strong Normal Equivalence Class Testing

i2 d

c

i1 a

B. Aichernig

b

Quality Assurance in Software Development 38 / 47

Institute for Software Technology

t

Weak Robust Equivalence Class Testing

i2 d

c

i1 a

B. Aichernig

b

Quality Assurance in Software Development 39 / 47

Institute for Software Technology

t

Strong Robust Equivalence Class Testing

i2 d

c

i1 a

B. Aichernig

b

Quality Assurance in Software Development 40 / 47

Institute for Software Technology

t

Triangle Example

I

Note that four possible outpus can occur.

I

We can use these to identify output (range) equivalence classes: 1 2 3 4

B. Aichernig

{(a, b, c) {(a, b, c) {(a, b, c) {(a, b, c)

| | | |

isTriangle(a, b, c) = NoTriangle} isTriangle(a, b, c) = Scalene} isTriangle(a, b, c) = Isosceles} isTriangle(a, b, c) = Equilateral}

Quality Assurance in Software Development 41 / 47

Institute for Software Technology

t

Triangle: Normal Equivalence Class Testing

The four weak normal equivalence class test cases are: Test Case WN1 WN2 WN3 WN4

a 4 3 2 5

b 1 4 2 5

c 2 5 3 5

Expected Output NoTriangle Scalene Isosceles Equilateral

No valid subintervals of variables a, b, and c exists ⇒ strong normal equivalence testing does not add test cases.

B. Aichernig

Quality Assurance in Software Development 42 / 47

Institute for Software Technology

t

Triangle: Robust Equivalence Class Testing

I

Let’s assume the valid input data has been restricted to [0, 200], and

I

error messages indicate if input is out of its range,

I

then the additional weak robust equivalence class test cases are: Test Case WR1 WR2 WR3 WR4 WR5 WR6

B. Aichernig

a -1 5 5 201 5 5

b 5 -1 5 5 201 5

c 5 5 -1 5 5 201

Expected Output Value a out of range! Value b out of range! Value c out of range! Value a out of range! Value b out of range! Value c out of range!

Quality Assurance in Software Development 43 / 47

Institute for Software Technology

t

Triangle: Robust Equivalence Class Testing

We show one corner of the cube of strong robust equivalence class test cases: Test Case WS1 WS2 WS3 WS4 WS5 WS6 WS7

B. Aichernig

a -1 5 5 -1 5 -1 -1

b 5 -1 5 -1 -1 5 -1

c 5 -5 -1 5 -1 -1 -1

Expected Output Value a out of range! Value b out of range! Value c out of range! Value a, b out of range! Value b, c out of range! Value a, c out of range! Value a, b, c out of range!

Quality Assurance in Software Development 44 / 47

Institute for Software Technology

t

Triangle: Input Equivalence Class

I

I

Equivalence class testing is sensitive to the equivalence relation chosen. Equivalence classes based on the input domain: 1 2 3 4 5

B. Aichernig

{(a, b, c) {(a, b, c) {(a, b, c) {(a, b, c) {(a, b, c)

| | | | |

a = b ∧ b = c} a = b ∧ a 6= c} a 6= b ∧ a = c} b = c ∧ a 6= b} a 6= b ∧ a 6= c ∧ b 6= c}

Quality Assurance in Software Development 45 / 47

Institute for Software Technology

t

Triangle: Input Equivalence Class (cont.)

I

Furthermore, we can apply the triangle property to see if they constitute a triange: 6 7 8

B. Aichernig

{(a, b, c) | a ≥ b + c} {(a, b, c) | b ≥ a + c} {(a, b, c) | c ≥ a + b}

Quality Assurance in Software Development 46 / 47

Institute for Software Technology

t

Triangle: Input Equivalence Class (cont.)

I

We could be even more thorough and distinguish between x = y and x > y in the properties above: 6 7 8 9 10 11

B. Aichernig

{(a, b, c) {(a, b, c) {(a, b, c) {(a, b, c) {(a, b, c) {(a, b, c)

| | | | | |

a = b + c} a > b + c} b = a + c} b > a + c} c = a + b} c > a + b}

Quality Assurance in Software Development 47 / 47