Part I Quality Assurance 1. Introduction to Quality 1.1 Introduction 3 1.2 Historical Perspective of Quality 4 1.3 What is Quality? (Is it a fact or perception?) 4 1.4 Definitions of Quality 6 1.5 Core Components of Quality 7 1.6 Quality View 10 1.7 Financial Aspect of Quality 15 1.8 Definition of Quality 17 1.9 Customers, Suppliers and Processes 18 1.10 Total Quality Management (TQM) 18 1.11 Quality Principles of ‘Total Quality Management’ 19 1.12 Quality Management Through Statistical Process Control 1.13 Quality Management Through Cultural Changes 23 1.14 Continual (Continuous) Improvement Cycle 24 1.15 Quality in Different Areas 25 1.16 Benchmarking and Metrics 26 1.17 Problem Solving Techniques 26 1.18 Problem Solving Software Tools 27 2. Software Quality 2.1 Introduction 30 2.2 Constraints of Software Product Quality Assessment 2.3 Customer is a King 31
SOFTWARE TESTING: Principles, Techniques and Tools Quality and Productivity Relationship 32 Requirements of a Product 34 Organisation Culture 36 Characteristics of Software 38 Software Development Process 38 Types of Products 43 Some Other Schemes of Criticality Definitions 44 Problematic Areas of Software Development Life Cycle Software Quality Management 50 Why Software Has Defects? 50 Processes Related to Software Quality 51 Quality Management System Structure 52 Pillars of Quality Management System 53 Important Aspects of Quality Management 54
45
Part II Basic Concepts of Software Testing 3. Fundamentals of Software Testing 3.1 Introduction 63 3.2 Historical Perspective of Testing 64 3.3 Definition of Testing 65 3.4 Approaches to Testing 66 3.5 Popular Definitions of Testing 70 3.6 Testing During Development Life Cycle 73 3.7 Requirement Traceability Matrix 75 3.8 Essentials of Software Testing 77 3.9 Workbench 78 3.10 Important Features of Testing Process 80 3.11 Misconceptions About Testing 82 3.12 Principles of Software Testing 83 3.13 Salient Features of Good Testing 84 3.14 Test Policy 85 3.15 Test Strategy or Test Approach 85 3.16 Test Planning 85 3.17 Testing Process and Number of Defects Found in Testing 3.18 Test Team Efficiency 87 3.19 Mutation Testing 88 3.20 Challenges in Testing 89 3.21 Test Team Approach 90 3.22 Process Problems Faced by Testing 94 3.23 Cost Aspect of Testing 95 3.24 Establishing Testing Policy 99 3.25 Methods 99 3.26 Structured Approach to Testing 100 3.27 Categories of Defect 101 3.28 Defect, Error, or Mistake in Software 101
63–111
87
Contents 3.29 3.30 3.31 3.32 3.33 3.34 3.35 3.36
xi
Developing Test Strategy 102 Developing Testing Methodologies (Test Plan) 102 Testing Process 105 Attitude Towards Testing (Common People Issues) 107 Test Methodologies/Approaches 107 People Challenges in Software Testing 111 Raising Management Awareness for Testing 111 Skills Required by Tester 112
4. Configuration Management 4.1 Introduction 116 4.2 Configuration Management 116 4.3 Cycle of Configuration Management 117 4.4 Configuration Management Process 119 4.5 Auditing Configuration Library 122 4.6 Configurable Item 123 4.7 Baselining 123 4.8 Few More Concepts About Configuration Library 4.9 Storage of Configurable Items in Library 124 4.10 Using Automated Configuration Tools 126 4.11 Configuration Management Planning 127
116–127
123
5. RiskAnalysis 5.1 Introduction 132 5.2 Advantages of Automated System 132 5.3 Disadvantages of Automated System 133 5.4 Risk 133 5.5 Constraints (Caveats) 134 5.6 Project Risks (Project Management Risks) 134 5.7 Product Risks 136 5.8 Risks Faced Due to Software Systems 137 5.9 Software Implementation Risks 147 5.10 Identification of Risks 148 5.11 Types of Software Risks 150 5.12 Handling of Risks in Testing 151 5.13 Types of Actions for Risk Control Management 152 5.14 Risks and Testing 154 5.15 Assumptions in Testing 155 5.16 Testing as a Risk Reduction Program (Prioritisation in Testing) 5.17 Risks of Testing 156 6. Software Verification and Validation 6.1 Introduction 161 6.2 Verification 162 6.3 Verification Work Bench 163 6.4 Methods of Verification 164 6.5 Types of Review on the Basis of Stage/Phase
SOFTWARE TESTING: Principles, Techniques and Tools Examples of Entities Involved in Verification 174 Reviews in Testing Lifecycle 175 Coverage in Verification (Test Designing) 177 Concerns of Verification 179 Validation 180 Validation Work Bench 181 Levels of Validation 182 Coverage in Validation (Prioritisation/Slice Based Testing) Acceptance Testing 185 Management of Verification and Validation (V & V) 186 Software Development Verification and Validation Activities
183
188
7. V-Test M odel 7.1 Introduction 191 7.2 V Model For Software 191 7.3 Testing During Proposal Stage 194 7.4 Testing During Requirement Stage 194 7.5 Testing During Test-Planning Phase 195 7.6 Testing During Design Phase 196 7.7 Testing During Coding 197 7.8 VV Model 198 7.9 Critical Roles and Responsibilities 200
191–200
8. Defect M anagement 8.1 Introduction 203 8.2 Defect Classification 204 8.3 Defect Management Process (Approach) 206 8.4 Defect Life Cycle 207 8.5 Defect Template 209 8.6 Defect Management Process (Fixing and Root Cause of Defect) 8.7 Estimate Expected Impact of a Defect 214 8.8 Why Defect Management Needs a Risk Discussion? 215 8.9 Techniques for Finding Defects 216 8.10 Reporting a Defect 216
203–216
212
Part III TestingTechniques and Tools 9. Levels of Testing 9.1 Introduction 221 9.2 Proposal Testing 221 9.3 Requirement Testing 222 9.4 Design Testing 224 9.5 Code Review 225 9.6 Unit Testing 225 9.7 Module Testing 226 9.8 Integration Testing 226 9.9 Big-Bang Testing 230
221–234
Contents 9.10 9.11 9.12 9.13 9.14
xiii
Sandwich Testing 231 Critical Path First 232 Subsystem Testing 232 System Testing 232 Testing Stages 233
10. Acceptance Testing 10.1 Introduction 235 10.2 Acceptance Testing Criteria 236 10.3 Importance of Acceptance Criteria 237 10.4 Some Famous Acceptance Criteria 237 10.5 Alpha Testing 241 10.6 Beta Testing 242 10.7 Gamma Testing 243 10.8 Acceptance Testing During Each Phase of Software Development 244 10.9 Consideration of Alpha and Beta Acceptance Testing Process 245 10.10 What Does Software Acceptance Enable? 246 10.11 Customer’s Responsibilities in Acceptance Testing 246 10.12 Fits for Acceptance Testing 246 10.13 Define Acceptance Criteria 247 10.14 Criticality of Requirements 248 10.15 Factors Affecting Criticality of the Requirements 249 10.16 Developing Acceptance Test Plan 251 10.17 Software Acceptance Plan 251 10.18 User Responsibilities in Acceptance Test plan 253 10.19 Executing Acceptance Plan 255
SOFTWARE TESTING: Principles, Techniques and Tools Execution Testing 281 Operations Testing 281 Compliance Testing 282 Usability Testing 282 Decision Table Testing (Axiom Testing) 283 Documentation Testing 284 Training Testing 285 Rapid Testing 285 Control Flow Graph 286 Generating Tests on the Basis of Combinatorial Designs State Graph 289
12. Special Tests (Part II) 12.1 Introduction 293 12.2 Risk Associated With New Technologies 294 12.3 Process Maturity Level of Technology 296 12.4 Testing Adequacy of Control in New Technology Usage 12.5 Object-Oriented Application Testing 298 12.6 Testing of Internal Controls 299 12.7 ‘COTS’ Testing 302 12.8 Client-Server Testing 306 12.9 Web Application Testing 307 12.10 Mobile Application Testing (PDA Devices) 309 12.11 eBusiness/eCommerce Testing 310 12.12 Agile Development Testing 312 12.13 Data Warehousing Testing 313
288
293–313
297
13. TestingTools 13.1 Introduction 317 13.2 Features of Test Tool 318 13.3 Guidelines for Selecting a Tool 318 13.4 Tools and Skills of Tester 319 13.5 Static Testing Tools 319 13.6 Dynamic Testing Tool 319 13.7 Advantages of Using Tools 320 13.8 Disadvantages of Using Tools 320 13.9 When to Use Automated Test Tools 320 13.10 Testing Using Automated Tools 321 13.11 Difficulties While Introducing New Tools 323 13.12 Process of Procurement of COTS (Readily Available Tool from Market) 13.13 Procurement of Tools from Contractor 326 13.14 Advantages of Tools Developed By External Organisations 327 13.15 Contracting a Software 327 13.16 Process of Procurement of Tools from Contractor 328
317–328
324
Contents
xv
Part IV TestingProcess 14. Test Planning 14.1 Introduction 333 14.2 Test Policy 334 14.3 Content of Test Policy in General 335 14.4 Test Strategy 336 14.5 Content of Test Strategy in General 336 14.6 Test Planning 338 14.7 Test Plan 338 14.8 Quality Plan and Test Plan 340 14.9 Quality Plan Template 341 14.10 Test Plan Template (System Testing) 345 14.11 Guidelines for Developing the Test Plan 353 14.12 Test Administration Definition 354 14.13 Test Estimation 354 14.14 Test Standards 356 14.15 Building Test Data and Test Cases 357 14.16 Test Scenario 358 14.17 Test Cases 361 14.18 Essential Activities in Testing 363 14.19 Template for Test Cases 363 14.20 Test Scripts 366 14.21 Test Management Software 368 14.22 Test Log Document 369 14.23 Effective Test Cases 369 14.24 Test File 370 14.25 Building Test Data 371 14.26 Generation of Test Data 378 14.27 Tools Used to Build Test Data 380 14.28 Roles and Responsibilities in Testing Life Cycle 14.29 Test Progress Monitoring 382
333–382
380
15. Test M etrics and Test Reports 15.1 Introduction 384 15.2 Testing Related Data 385 15.3 Defect Data 386 15.4 Efficiency/Productivity Data 388 15.5 Categories of the Product/Project Test Metrics 391 15.6 Estimated, Budgeted, Approved and Actual 392 15.7 Resources Consumed in Testing 393 15.8 Effectiveness of Testing 394 15.9 Defect Density 395 15.10 Defect Leakage Ratio (Defect Life) 396 15.11 Residual Defect Density (RDD) 397 15.12 Test Team Efficiency 397
SOFTWARE TESTING: Principles, Techniques and Tools Test Case Efficiency 397 Rework 398 MTTF/MTTR 398 Implementing Measurement Reporting System in an Organisation Test Reports 402 Project Test Status Report 403 Test Reports 406 Integration Test Report 407 System Test Report 407 Acceptance Test Report 408 Guidelines for Writing and Using Report 408 Final Test Reporting 408 Test Status Report 409 Benchmarking 419
16. Qualitative and Quantitative Analysis 16.1 Introduction 425 16.2 Types of Data 425 16.3 Qualitative and Quantitative Data 426 16.4 Types of Qualitative Data 427 16.5 Types of Quantitative Data 428 16.6 Quality Tools 431 16.7 Non-Statistical Tools 431 16.8 Statistical Tool Background 440 16.9 Statistical Tools 441
399
425–441
Part V Test Process M anagement 17. Test Process Improvement 17.1 Introduction 455 17.2 Change in Perception About Testing 455 17.3 Problems Concerning Testing Process 456 17.4 The Need for Test Process Improvement 457 17.5 Test Process Maturity 457 17.6 Test Process Improvement Model 458 17.7 Test Process Improvement Model Stages 460 17.8 Graphical Representation of Improvements 461 Appendix FrequentlyAskedQuestions Glossary List of Abbreviations References Index
