Software Testing Challenges
Structural Testing
Regression Testing
Mutation Testing
Future Work
Conclusi
The Theory and Practice of Software Testing: Can we Test it? Yes we Can! Gregory M. Kapfhammer† Department of Computer Science Allegheny College, Pennsylvania, USA http://www.cs.allegheny.edu/~gkapfham/
SGT Global, February 2008 † In Conjunction with Mary Lou Soffa, Kristen Walcott (UVa/CS) Suvarshi Bhadra, Joshua Geiger, Adam Smith, Gavilan Steinman, Yuting Zhang (Allegheny/CS)
Featuring images from Embroidery and Tapestry Weaving, Grace Christie (Project Gutenberg)
The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 1 / 23
Software Testing Challenges
Structural Testing
Regression Testing
Mutation Testing
Future Work
Conclusi
Presentation Outline 1
Software Testing Challenges
2
Structural Testing
3
Regression Testing
4
Mutation Testing
5
Future Work
6
Conclusion
The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 2 / 23
Software Testing Challenges
Structural Testing
Regression Testing
Mutation Testing
Future Work
Conclusi
The Challenge of Software Testing I shall not deny that the construction of these testing programs has been a major intellectual effort: to convince oneself that one has not overlooked “a relevant state” and to convince oneself that the testing programs generate them all is no simple matter. The encouraging thing is that (as far as we know!) it could be done. Edsger W. Dijkstra, Communications of the ACM, 1968
Important Question: What are your software development and testing challenges? What are your best solutions? The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 3 / 23
Software Testing Challenges
Structural Testing
Regression Testing
Mutation Testing
Future Work
Conclusi
Modern Software is Complex
Complex source code, database, files, and network communication Can we increase reliability by simplifying software? The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 4 / 23
Software Testing Challenges
Structural Testing
Regression Testing
Mutation Testing
Future Work
Conclusi
Defect Locations P Input
Output
a 5 print ... exit
Final Result: 45 Byte Code
Database
Virtual Machine File System
Graphical Interface Operating System
Defects may exist in the individual components or the interactions The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 5 / 23
Software Testing Challenges
Structural Testing
Regression Testing
Mutation Testing
Future Work
Conclusi
Approaches to Software Testing
Structural Testing
Regression Testing
Mutation Testing
Specification Testing
Testing isolates defects and establishes a confidence in the correctness of a software application The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 6 / 23
Software Testing Challenges
Structural Testing
Regression Testing
Mutation Testing
Future Work
Conclusi
Presentation Outline 1
Software Testing Challenges
2
Structural Testing
3
Regression Testing
4
Mutation Testing
5
Future Work
6
Conclusion
The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 7 / 23
Software Testing Challenges
Structural Testing
Regression Testing
Mutation Testing
Future Work
Conclusi
What is a Test Case? Expected Output
Set Up
Input
Method Under Test
Output
Test Oracle
Verdict
Tear Down
Overview Test suite executor (JUnit) runs each test case independently Each test invokes a method within the program and then compares the actual and expected output values The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 8 / 23
Software Testing Challenges
Structural Testing
Regression Testing
Mutation Testing
Future Work
Conclusi
Test Coverage Monitoring Program Adequacy Criterion Residual Test Requirements
Instrumentation
Instrumented Program Test Suite Execution
Cummulative Adequacy Calculator
Covered Requirements
Overview Structural adequacy criteria focus on the coverage of nodes, edges, paths, and definition-use associations Instrumentation probes track the coverage of test requirements The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 9 / 23
Software Testing Challenges
Structural Testing
Regression Testing
Mutation Testing
Future Work
Conclusi
Calculating the Coverage of a Test Test Suite
Calculating Coverage
Adequacy Criterion Program
Use instrumentation probes to capture and analyze a test suite’s coverage of the program
Instrumented Program
Instrumentation Instrumented Test Suite
Database
Test Coverage Monitoring
Coverage Results
Test Requirements
Adequacy Calculation Adequacy Measurements
Regression Testing The adequacy measurements can be used to support both test suite reduction and prioritization
The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 10 / 23
Software Testing Challenges
Structural Testing
Regression Testing
Mutation Testing
Future Work
Conclusi
Presentation Outline 1
Software Testing Challenges
2
Structural Testing
3
Regression Testing
4
Mutation Testing
5
Future Work
6
Conclusion
The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 11 / 23
T11 Software Testing Challenges R4
Structural Testing
Regression Testing
Mutation Testing
Future Work
Conclusi
Finding the Overlap in Coverage
R4 R4 R4 T5 R4 R4 T10
R
R5 R2
R1
R3
R4
T1
T3
T11
R6
R7
R5
R6 T10 R7 R7
T8
T2
T4
T12
T7
T9
T10
T5
T6
Rj → Ti means that requirement Rj is covered by test Ti T = hT2 , T3 , T6 , T9 i covers all of the test requirements May include the remaining tests so that they can redundantly cover the requirements The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 12 / 23
Software Testing Challenges
Structural Testing
Regression Testing
Mutation Testing
Future Work
Conclusi
Reducing and Prioritizing the Tests Original Test Suite GRT Repeat Begin
Coverage Report
Reduction or Prioritization
Modified Test Suite
Test Suite Execution
Testing Results
End
Program VSRT Repeat
Regression Testing Overview Reduction creates a smaller test suite that covers the same requirements as the original suite. Prioritization re-orders the tests so that they cover the requirements more effectively. Techniques use heuristics to solve NP-complete problems. The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 13 / 23
Software Testing Challenges
Structural Testing
Regression Testing
Mutation Testing
Future Work
Conclusi
Evaluating a Test Prioritization Cover R(T1 ) Cover
i=1
R(Ti ) Cover R(T )
Covered Test Reqs C(T ,t)
acements
Sn−1
Tn Done
... Area Testing Time
T1 Done
R t(n) 0
C(T , t)
(t) Tn−1 Done
Prioritize to increase the CE of a test suite CE = Actual ∈ [0, 1] Ideal The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 14 / 23
Software Testing Challenges
Structural Testing
Regression Testing
Mutation Testing
Future Work
Conclusi
Characterizing a Test Suite Test Information Test Case T1 T2 T3
Cost (sec) 5 10 4
R1 X X X
Requirements R2 R3 R4 R5 X X X X X X
Total Testing Time = 19 seconds
Formulating the Metrics CE considers the execution time of each test while CE u assumes that all test cases execute for a unit cost The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 15 / 23
Software Testing Challenges
Structural Testing
Regression Testing
Mutation Testing
Future Work
Conclusi
Coverage Effectiveness Values Calculating CE and CEu Ordering T1 T2 T3 T1 T3 T2 T2 T1 T3 T2 T3 T1 T3 T1 T2 T3 T2 T1
CE .3789 .5053 .3789 .4316 .5789 .5789
CEu .4 .4 .5333 .6 .4557 .5333
Observations Including test case costs does impact the CE metric Depending upon the characteristics of the test suite, we may see CE = CEu , CE > CEu , or CE < CEu The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 16 / 23
Software Testing Challenges
Structural Testing
Regression Testing
Mutation Testing
Future Work
Conclusi
Presentation Outline 1
Software Testing Challenges
2
Structural Testing
3
Regression Testing
4
Mutation Testing
5
Future Work
6
Conclusion
The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 17 / 23
Software Testing Challenges
Structural Testing
Regression Testing
Mutation Testing
Future Work
Conclusi
Mutation Testing Techniques Program
Mutant Creation
Mutant Generator
A mutation testing tool (e.g., µJava or Jumble) inserts defects into the program under test
Mutant Programs
Question: Why are we inserting faults into the the programs that we are testing?
Test Suite Execution Mutation Score
Test Quality Goal: measure the quality of the test suite by determining whether or not it can differentiate between faulty and non-faulty programs The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 18 / 23
Software Testing Challenges
Structural Testing
Regression Testing
Mutation Testing
Future Work
Conclusi
Average Percentage of Faults Detected Program
Fault Seeding Fault Seeding
Use known faults or a mutation testing tool (e.g., µJava or Jumble) to insert defects into the program
Faulty Program
Test Suite
Determine which test(s) are able to detect the seeded faults and construct a fault table
Test Suite Execution
Test Results
Fault Table
APFD Calculator
APFD Score
APFD Calculation A test ordering has a higher APFD score if it rapidly detects the faults
The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 19 / 23
Software Testing Challenges
Structural Testing
Regression Testing
Mutation Testing
Future Work
Conclusi
Presentation Outline 1
Software Testing Challenges
2
Structural Testing
3
Regression Testing
4
Mutation Testing
5
Future Work
6
Conclusion
The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 20 / 23
Software Testing Challenges
Structural Testing
Regression Testing
Mutation Testing
Future Work
Conclusi
Search-Based Test Suite Prioritization
0.70 0.65 0.60 0.55 0.50
Coverage Effectiveness Value
0.75
Coverage Effectiveness Values
0
20
40
60
80
100
120
Test Prioritization Index
Use heuristic search (HC, SANN, GA) to prioritize the test suite The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 21 / 23
Software Testing Challenges
Structural Testing
Regression Testing
Mutation Testing
Future Work
Conclusi
Detailed Empirical Evaluations
New Testing Techniques
Real World Programs
Systematically study the efficiency and effectiveness trade-offs of different software testing techniques
The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 22 / 23
Software Testing Challenges
Structural Testing
Regression Testing
Mutation Testing
Future Work
Conclusi
Conclusions Concluding Remarks Software development and testing is fun and exciting! There are many new developments in research and practical tools – some of which are ready for use today! What are your favorite software testing tools and techniques?
Resources Conferences: ICSE, FSE, ISSTA, ASE, ICSM, ISSRE Journals: TSE, TOSEM, IST, JSS, JSME Many articles are available online from Google Scholar http://www.cs.allegheny.edu/~gkapfham/ The Theory and Practice of Software Testing: , Can we Test it? Yes we Can! 23 / 23
