Testing Techniques. Functional and NonFunctional Testing CSCI E - 19 SOFTWARE TESTING AND TEST -DRIVEN DEVELOPMENT DR . A L I N E YUR I K

HARVAR D UN I V E RSI TY E X T ENSION S CHOOL

COPYRIGHT © 2015 DR. ALINE YURIK

General Testing Techniques ◦ General/High-Level Techniques ◦ Positive and Negative Testing ◦ White-box and Black-box Testing ◦ Experience-Based Testing ◦ Error Guessing ◦ Automated Software Testing

COPYRIGHT © 2015 DR. ALINE YURIK

Functional Testing Techniques ◦ Functional Testing ◦ Equivalence Partitioning ◦ Boundary and Value Analysis ◦ Intrusive Testing ◦ Random Testing ◦ State Transition Testing ◦ Static Testing ◦ Thread Testing ◦ Pairwise Testing

COPYRIGHT © 2015 DR. ALINE YURIK

Non-Functional Testing Techniques ◦ Non-Functional Testing ◦ Configuration/Installation Testing ◦ Compatibility and Interoperability Testing ◦ Documentation and Help Testing ◦ Fault Recovery Testing ◦ Performance Testing ◦ Reliability Testing ◦ Security Testing ◦ Stress Testing ◦ Usability Testing ◦ Volume Testing COPYRIGHT © 2015 DR. ALINE YURIK

General: Positive/Negative Testing ◦ Positive Testing ◦ Does the system conform to stated requirements? ◦ Derived from analysis of requirements documents ◦ Required as part of application acceptance for delivery

◦ Negative Testing ◦ Does the system not do what it is not supposed to do? ◦ Testing parts of the applications that have not been documented or poorly documented ◦ This could be open-ended as there may be countless negative testing possibilities ◦ A representative selection of negative test cases can be included but it may not be possible to be have complete negative testing COPYRIGHT © 2015 DR. ALINE YURIK

Retrieved from: http://www.softwaretestingclass.com/positive-and-negative-testing-in-software-testing/

General: White-Box/Black-Box Testing ◦ White-Box Testing ◦ Takes into account how a system is constructed ◦ May be testing specific known logic or system interactions ◦ Design documents can be used as a foundation of white-box testing ◦ Very useful during unit testing and early testing phases

◦ Black-Box Testing ◦ Tests are designed with no knowledge of how the system is constructed ◦ Tests are based on the external behavior of the system ◦ May be used with legacy system testing or 3rd party system testing ◦ Can be used during system testing and integration testing phases, as well as during end-to-end testing COPYRIGHT © 2015 DR. ALINE YURIK

General: Experience-Based Testing ◦ Experience-Based Testing or Error Guessing – Ability to Find Defects Based on: ◦ Knowledge of the System: ◦ Design and implementation knowledge ◦ Knowledge of Earlier Testing Results: ◦ Useful in Regression Testing ◦ Experience with Testing Similar Systems ◦ Knowledge of Typical Implementation Issues

COPYRIGHT © 2015 DR. ALINE YURIK

General: Automated Testing ◦ Test Automation Tools ◦ Can run regression test suites for functional testing ◦ Can perform non-functional testing: performance, load, volume, stress ◦ Very useful with iterative development approaches

◦ Automation Tools Are Useful for Testing Applications with: ◦ ◦ ◦ ◦ ◦ ◦

Quick build/release cycles Delivered across different platforms Complex GUI and architecture Mission-Critical and Strict/Thorough Testing Requirements Need for Reduce Testing Time/Manual Effort Need to Do More Testing without Adding Time

COPYRIGHT © 2015 DR. ALINE YURIK

Class Checkpoint: General Testing Techniques ◦ Your team is in charge of testing a self-checkout cash-resister system for a major supermarket ◦ Consider General Testing Techniques you will be using on this project and provide 1 example of a test scenario using each technique: ◦ Positive Testing ◦ Negative Testing ◦ White Box Testing ◦ Black Box Testing ◦ Experience Based Testing ◦ Automated Testing COPYRIGHT © 2015 DR. ALINE YURIK

Functional Testing Techniques Functional Testing

Boundary Testing

Strong Equivalence Class Testing

Weak Equivalence Class Testing Weak Normal Equivalence Class Testing

State Transition Testing

Equivalence Class Testing

Weak Robust Equivalence Class Testing

Strong Normal Equivalence Class Testing COPYRIGHT © 2015 DR. ALINE YURIK

Strong Robust Equivalence Class Testing

Functional: Equivalence Partitioning ◦ Equivalence Partitioning: ◦ Approach where inputs and outputs are grouped into classes, so that each class is treated in the same manner by the system ◦ We can test one/several representative member(s) of an equivalence class and that would be similar to testing the entire class ◦ How can we effectively divide inputs/outputs into equivalence classes?

◦ Equivalence Class Testing Example - Shopping Cart in Online Store: ◦ ◦ ◦ ◦ ◦ ◦

Equivalence Class A: 1 item in a shopping cart Equivalence Class B: Multiple items in a shopping cart Equivalence Class C: Credit card payment Equivalence Class D: PayPal payment Equivalence Class E: Standard Shipping Equivalence Class F: Express Shipping COPYRIGHT © 2015 DR. ALINE YURIK

Source: http://www.testnbug.com/2015/01/equivalence-class-partitioning-and-boundary-value-analysis-black-box-testing-techniques/

Weak Normal Equivalence Class Testing ◦ Select 1 value from each equivalence class for a test case: ◦ Shopping cart with 1 item, Payment by Credit Card, Standard Shipping

COPYRIGHT © 2015 DR. ALINE YURIK

Strong Normal Equivalence Class Testing ◦ Select all possible combinations of values from equivalence classes ◦ In the online store program with the following equivalence classes: ◦ Number of items in the shopping cart: 1 or more than one ◦ Method of payment: credit card or PayPal ◦ Shipping method: standard shipping, overnight shipping, free ground shipping

◦ Test cases would test all possible combinations of these value selections ◦ Test Case 1: 1 item in the shopping cart, pay by credit card, standard shipping ◦ Test Case 2: multiple items in the shopping cart, pay by credit card, standard shipping ◦ Test Case 3: 1 item in the shopping cart, pay by PayPal, standard shipping ◦… COPYRIGHT © 2015 DR. ALINE YURIK

Weak Robust Equivalence Class Testing ◦ Testing both valid and invalid inputs ◦ Select 1 valid value from valid equivalence classes and one invalid value from invalid equivalence classes (the rest are valid) for a test case: ◦ Shopping cart with 1 item, Standard Shipping, try to pay by credit card with invalid account number

COPYRIGHT © 2015 DR. ALINE YURIK

Strong Robust Equivalence Class Testing ◦ Select all possible combinations of values from valid and invalid equivalence classes ◦ Test cases would test all possible combinations of these value selections ◦ Test Case 1: no items in the shopping cart, trying to pay by credit card and standard shipping ◦ Test Case 2: 1 item in the shopping cart, trying to pay by credit card with invalid number, standard shipping ◦ Test Case 3: 1 item in the shopping cart, trying to pay by credit card, overnight shipping selected, but the product is too big to ship by overnight shipping ◦… COPYRIGHT © 2015 DR. ALINE YURIK

Boundary Value Testing ◦ Look for specific values that may be on the boundary of accepted value ranges ◦ Examples: ◦ trying to withdraw $0 at an ATM, ◦ checking for correct date ranges (can we pay an online bill today? Yesterday?), ◦ how does a program behave with very large test values ◦ Related Testing ◦ Robustness testing - testing with values going beyond the boundaries and making sure the program handles these error conditions ◦ Worst case testing - testing with multiple values at boundary conditions ◦ Special value testing – using test values that the tester thinks may cause problems COPYRIGHT © 2015 DR. ALINE YURIK

State Transition Testing ◦ Tests the system from the perspective of different states the system can be in, and actions that can take system from one state to another ◦ Test cases can be created based on the requirements and design artifacts such as a state diagram ◦ Test cases exercise transitions between states, and specify: ◦ Initial states, inputs, expected outputs, expected final state ◦ Can be useful for positive and negative testing

COPYRIGHT © 2015 DR. ALINE YURIK

Class Checkpoint: Functional Testing Techniques ◦ What equivalence classes are appropriate for a self-checkout cashresister system for a major supermarket? ◦ Provide 1 example of a test scenario using each technique: ◦ Weak Normal Equivalence Testing ◦ Weak Robust Equivalence Testing ◦ Boundary Value Testing ◦ State Transition Testing

COPYRIGHT © 2015 DR. ALINE YURIK

Non-Functional Testing Techniques Non-Functional Testing

Configuration and Installation Testing

Compatibility and InteroperabilityTesting

Documentation and Help Testing

Fault Recovery Testing

Performance Testing

Usability Testing

Reliability Testing

Security Testing

Stress Testing

COPYRIGHT © 2015 DR. ALINE YURIK

Volume Testing

Non-Functional Testing ◦ Configuration and Installation Testing ◦ Is installation and configuration correct, is appropriate data loaded? ◦ Compatibility and Interoperability Testing ◦ Is the system coexisting well with other systems? Is it able to interoperate/communicate with appropriate systems? ◦ Documentation and Help Testing ◦ Fault Recovery Testing ◦ How well does a system recover after a crash? Testing rollback and recovery of data ◦ Performance Testing ◦ Testing user response time, system response time, external interface response time, CPU and memory utilization COPYRIGHT © 2015 DR. ALINE YURIK

Non-Functional Testing - continued ◦ Reliability Testing ◦ Testing resistance to failure (integrity) under typical usage ◦ Example: ensuring resources are released and there are no memory leaks ◦ Structural testing ◦ Example: ensuring all web links are connected and correct content is displayed ◦ Security Testing ◦ Stress Testing (peak loads) ◦ Usability Testing ◦ Volume Testing (large volumes of data) COPYRIGHT © 2015 DR. ALINE YURIK

Test-Driven Development: Software Engineer in Test ◦ Software Engineer in Test (SET) focuses on writing test cases to ”break” code – ◦ to find scenarios and data values that will cause a program to break ◦ SET also writes test frameworks to assist SWEs with creating small and medium test cases ◦ SETs work closely with SWEs and jointly own testing of the product

COPYRIGHT © 2015 DR. ALINE YURIK

Software Engineer in Test: Responsibilities ◦ SET Responsibilities: ◦ Identify where small tests need to be written ◦ Write test build targets that combine unit tests for program components ◦ Integrate smaller component test build targets into a larger application test target (test framework) ◦ Integrate smaller test build targets into a regression test suite

COPYRIGHT © 2015 DR. ALINE YURIK

SET Involvement in Development ◦ SETs are collocated with features developers ◦ SETs are active collaborators in design and coding phases ◦ SETs participate in code reviews ◦ SET is added to a project once the project went through initial prototyping phase and is considered viable ◦ Design phase: ◦ SET can bring perspective on code reuse and component interaction design ◦ SET reviews design documents for: ◦ Completeness, consistency, design issues, interfaces and protocols, testing issues COPYRIGHT © 2015 DR. ALINE YURIK

SET Leadership in Testing ◦ Testing of Interfaces and Protocols ◦ Early integration testing through mocks/fakes ◦ Automation testing planning ◦ Lightweight automation framework for continuous testing of the mocked system and checked-in code ◦ Testability ◦ Focus on program structure and code reviews

COPYRIGHT © 2015 DR. ALINE YURIK

Resources  

John Watkins, Simon Mills, Testing IT: An Off-the-Shelf Software Testing Process, 2nd edition, 2011, Cambridge University Press, Chapter 3 James Whittaker, Jason Arbon, Jeff Carollo, How Google Tests Software, 2012, Addison-Wesley, Chapter 2