School of Innovation, Design and Engineering

A Model for Estimating the Execution Cost of Test Cases

Author: Morteza Hosseini May-2015 Västerås-Sweden

Supervisor: Dr. Mehrdad Saadatmand Examiner: Prof. Kristina Lundqvist

Abstract Indubitably, testing plays a pivotal role in the software-developing life cycle through validating and verifying the quality of the software. In the testing process, however, estimating the test cost is an important aspect that needs to be taking into consideration. Due to a large volume of test cases and resource restrictions (budget, people, time-to-market, etc.), care must be taken in the process of selecting the most appropriate test case. Then, the costs for each individual test case must be determined based upon the standards and limitations of the organization to which it belongs. There are two main phases in estimating the test effort; namely, test case creation effort and test execution effort. The focus of the present study is, however, on estimating the latter one, i.e., test execution effort. It is commonly agreed that having a good command of test execution effort in early stages of the software development life cycle can help the practitioners of the field execute the most essential test cases. Accordingly, we attempted to estimate the test execution effort by developing a new approach in which we assigned a cost value to each test case based on some criteria that can be varying regarding target domain. The target domain can be any software system. Having done that, we evaluated our approach by applying it to both desktop and web applications. Keywords: Testing, test effort, test execution, test effort estimation, test execution effort, software testing.

Seyed Morteza Hosseini – Master Thesis – 2015

I

Acknowledgment This thesis work has been conducted with the great help and support of thesis supervisor Mehrdad Saadatmand who provided with his useful guidelines and suggestions throughout this thesis work. Special thanks also go to the examiner, Professor Kristina Lundqvist for her support and contribution in this thesis work. I also wish to offer my sincere gratitude to Mr. Majid Dehghan executive manager and Mr. Kian Nasr project manager at Fara Andish Company, in addition Mr. Saeed Salehi executive manager and Mr. Tavakolnia developer at Partak Company for their faithfully collaboration for doing case studies at their companies. Finally yet importantly, I would like to give thanks to my parents for their inmost support.

Seyed Morteza Hosseini Västerås, May 2015

Seyed Morteza Hosseini – Master Thesis – 2015

II

Table of Contents Abstract .....................................................................................................................................I Acknowledgment ..................................................................................................................... II 1. Introduction ........................................................................................................................ 1 1.1

Problem Statement .............................................................................................. 2

1.2

Research Goals ................................................................................................... 3

1.3

Contribution ....................................................................................................... 3

2. Research Methodology ........................................................................................................ 4 3. Systematic Review Protocol ................................................................................................ 6 3.1

Question Formulation ......................................................................................... 6

3.2

Question Quality ................................................................................................. 6

3.3

Sources Selection ................................................................................................ 8

3.4

Execution Evaluation .......................................................................................... 8

3.5

Information Extraction ........................................................................................ 9

3.6

Results Summarization ..................................................................................... 10

4. Related Work .................................................................................................................... 11 4.1

Software Cost Estimation .................................................................................. 11

4.2

Prioritizing Test Case Methods ......................................................................... 12 4.2.1 Historical-Based Methods ........................................................................... 12 4.2.1.1 Test Case Prioritization for Black Box Testing ............................... 12 4.2.2 Statement-Based Techniques ....................................................................... 14 4.2.2.1

Total Branch Coverage ................................................................ 14

4.2.2.2

Additional Branch Coverage ........................................................ 14

4.2.2.3

Total Statement Coverage Prioritization ....................................... 14

4.2.2.4

Additional Total Statement Coverage Prioritization ...................... 15

4.2.2.5

Total Fault Exposing Potential (FEP) Prioritization ..................... 15

4.2.2.6 Additional Total Fault Exposing Potential (FEP) Prioritization....... 15 4.2.3 Function Level Techniques .......................................................................... 16 4.2.3.1 Total Function Coverage Prioritization .......................................... 16 4.2.3.2 Additional Function Coverage Prioritization ................................... 16

Seyed Morteza Hosseini – Master Thesis – 2015

III

4.2.3.3 Total FEP Prioritization ................................................................. 16 4.2.3.4 Additional Total FEP Prioritization ............................................... 16 4.2.3.5 Total Fault Index (FI) Prioritization ............................................... 16 4.2.3.6 Additional Fault-Index (FI) Prioritization ...................................... 17 4.2.3.7 Total FI with FEP Coverage Prioritization ..................................... 19 4.2.3.8 Additional FI with FEP Coverage Prioritization ............................. 19 4.2.3.9 Total DIFF Prioritization ................................................................ 20 4.2.3.10 Additional DIFF Prioritization ...................................................... 20 4.2.3.11 Total DIFF with FEP Prioritization .............................................. 20 4.2.3.12 Additional DIFF with FEP Prioritization ...................................... 20 4.2.4 System Level Techniques ............................................................................ 21 4.2.4.1 Prioritization Using Relevant Slice .................................................. 21 4.2.5 Coverage-Based Techniques ........................................................................ 22 4.2.5.1 Fault Localization Technique .......................................................... 22 4.2.5.2 Fault Aware Test Case Prioritization (FATCP) ............................... 22 4.2.6 Model-Based Techniques ............................................................................ 25 4.2.6.1 Test Prioritization Using System Model ............................................. 25 4.2.6.2 Selective Test Prioritization............................................................. 26 4.2.6.3 Model Dependence-Based Test Prioritization .................................. 26 4.2.7

Random Ordering ....................................................................................... 27

4.2.8

Optimal Ordering ....................................................................................... 27

4.2.9

Average Percentage of Fault Detected (APFD) ........................................... 27

4.2.10

Average Percentage of Fault Detected Cost Cognizant/Aware (APFDc) .... 29

4.2.11 Case-Based Ranking Test Case Prioritization.............................................. 30 4.2.12 Bayesian Network-based Prioritization ....................................................... 31 4.2.13 Test Case Prioritization Techniques Summary ............................................ 32 4.3 Test Execution Effort Estimation Methods .......................................................... 36 4.3.1

Use Case Points (UCP) ............................................................................. 38

4.3.2 Conventional Methods................................................................................. 39 4.3.2.1 Ad-hoc Method................................................................................ 39 4.3.2.2 Percentage of Development Time .................................................... 39 4.3.2.3 Function Points Estimation .............................................................. 39 4.3.3 Estimation Model for Test Execution Effort based on Test Specifications ... 40 4.3.4 Early Test Effort Estimation by Using Use Case .......................................... 41

Seyed Morteza Hosseini – Master Thesis – 2015

IV

4.3.5 Cognitive Information Complexity Measure ................................................ 42 4.3.6 New, Search, Modify/ Management Information System Functionality ........ 43 4.3.7 Simplified Method based on the NSM/MIS method ..................................... 43 4.3.8 Software Requirement Specification (SRS).................................................. 43 4.3.9 Accumulated Efficiency Method ................................................................. 43 4.3.10 Case-Based Reasoning (CBR) Method ...................................................... 44 4.3.11 Cuckoo Search Method ............................................................................. 44 5. Our Effort Estimation Model ........................................................................................... 45 5.1

Metrics Descriptions ......................................................................................... 48

6. Case Study ......................................................................................................................... 51 6.1 Case Study 1 .......................................................................................................... 51 6.2 Case Study 2 .......................................................................................................... 74 7. Discussion .......................................................................................................................... 87 8. Summary and Conclusion ................................................................................................. 89 9. References ......................................................................................................................... 90 10. Appendix A........................................................................................................................ 94

Seyed Morteza Hosseini – Master Thesis – 2015

V

List of Figures Figure 1. Rela on Matrix R ............................................................................................................. 13 Figure 2. Addi onal Fault-Index Process .......................................................................................... 19 Figure 3. Relevant Slice Technique Example .................................................................................... 21 Figure 4. FATCP Algorithm ............................................................................................................. 23 Figure 5. EFSM Sample Model ......................................................................................................... 25 Figure 6. CBR High Level View ......................................................................................................... 30 Figure 7. BN Model Process ............................................................................................................ 31 Figure 8. The Es ma ng Process of Execution Effort ....................................................................... 40 Figure 9. Early Test Effort Estimation Approach .............................................................................. 42 Figure 10. Our Approach View ......................................................................................................... 46 Figure 11. Conversion Factor Calcula on ........................................................................................ 50 Figure 12. High Level View of Fara Andish ERP System ..................................................................... 52 Figure 13. Sale Module Communication of Fara Andish ERP System ................................................ 55 Figure 14. Modules Involved in TC1 ................................................................................................. 55 Figure 15. In-house Purchasing Module Communication of Fara Andish ERP System ....................... 59 Figure 16. Modules Involved in Test Case 2...................................................................................... 59 Figure 17. Payroll Module Communication of Fara Andish ERP System ............................................ 63 Figure 18. Modules Involved in Test Case 3...................................................................................... 63 Figure 19. Accounting Module Communication of Fara Andish ERP System ..................................... 67 Figure 20. Modules Involved in Test Case 4...................................................................................... 67 Figure 21. Personnel Module Communication of Fara Andish ERP System ....................................... 70 Figure 22. Modules Involved in Test Case 5...................................................................................... 70 Figure 23. High Level View of Partak CRM ........................................................................................ 74 Figure 24. High Level View of Web.Controllers Layer ....................................................................... 75 Figure 25. TC1 Layers Communica ons............................................................................................ 77 Figure 26. TC2 Layers Communica ons............................................................................................ 80 Figure 27. TC3 Layers Communica ons............................................................................................ 83

Seyed Morteza Hosseini – Master Thesis – 2015

VI

List of Tables Table 1. Keywords and Synonyms ...................................................................................................... 7 Table 2. Escalate Order ................................................................................................................... 13 Table 3. Deescalate Order ............................................................................................................... 13 Table 4. Total FI with FEP ................................................................................................................ 19 Table 5. APFD Rate of Fault ............................................................................................................. 28 Table 6. Test Case Priori za on Techniques .................................................................................... 34 Table 7. Statement Level & Func on Level Priori za on Techniques ............................................... 35 Table 8.Outset and Result Techniques ............................................................................................ 35 Table 9. UUCW ............................................................................................................................... 38 Table 10. Technical and Environmental Factors ............................................................................... 38 Table 11. Importance Degree Guideline........................................................................................... 47 Table 12. Modules Ranking .............................................................................................................. 53 Table 13. Registra on Sale Dra Test Case ...................................................................................... 54 Table 14. Registra on Sale Dra Test Record................................................................................... 54 Table 15. Test Case 1 Sale Module Dependencies ............................................................................ 56 Table 16. Test Case 1 Storehouse Module Dependencies................................................................. 56 Table 17. Test Case 1 Importance Degree ........................................................................................ 57 Table 18. Purchase Invoice Issuance Test Case................................................................................. 58 Table 19. Purchase Invoice Issuance Test Record ............................................................................. 58 Table 20. Test Case 2 In-house Purchasing Module Dependencies ................................................... 60 Table 21. Test Case 2 Storehouse Module Dependencies................................................................. 60 Table 22. Test Case 2 Importance Degree ........................................................................................ 61 Table 23. Salary Calculation Test Case ............................................................................................. 62 Table 24. Salary Calculation Test Record .......................................................................................... 62 Table 25. Test Case 3 Payroll Module Dependencies........................................................................ 64 Table 26. Test Case 3 Accoun ng Module Dependencies ................................................................. 64 Table 27. Test Case 3 Importance Degree ........................................................................................ 65 Table 28. Accounting Code Registration Test Case ........................................................................... 66

Seyed Morteza Hosseini – Master Thesis – 2015

VII

Table 29. Accoun ng Code Registra on Test Record ....................................................................... 66 Table 30. Accounting Module Dependencies ................................................................................... 68 Table 31. Test Case 4 Importance Degree ........................................................................................ 68 Table 32. Personnel Registration Test Case ...................................................................................... 69 Table 33. Registration of Personnel Test Record .............................................................................. 69 Table 34. Personnel Module Dependencies ..................................................................................... 71 Table 35. Test Case 5 Importance Degree ........................................................................................ 71 Table 36. Case Study Results............................................................................................................ 72 Table 37. Layers Ranking ................................................................................................................. 75 Table 38. Inserting New News Test Case .......................................................................................... 76 Table 39. Inserting New News Test Record ...................................................................................... 76 Table 40. Test Case 1 Layers Dependencies ..................................................................................... 78 Table 41. Test Case 1 Importance Degree ........................................................................................ 78 Table 42. Inserting New Image in Image Gallery Test Case ............................................................... 79 Table 43. Inserting New Image in Image Gallery Test Record ........................................................... 79 Table 44. Test Case 2 Layers Dependencies ..................................................................................... 80 Table 45. Test Case 2 Importance Degree ........................................................................................ 81 Table 46. Creating New Training Class Test Case .............................................................................. 82 Table 47. Creating New Training Class Test Record .......................................................................... 82 Table 48. Test Case 3 Layers Dependencies ..................................................................................... 83 Table 49. Test Case 3 Importance Degree ........................................................................................ 84 Table 50. Case Study 2 Results ......................................................................................................... 85 Table 51. Actual & Estimated Effort Difference ................................................................................ 87 Table 52. Acronyms ......................................................................................................................... 93

Seyed Morteza Hosseini – Master Thesis – 2015

VIII

List of Charts Chart 1. Number of Papers in each Source ......................................................................................... 4 Chart 2. Number of Papers in each Group .......................................................................................... 5 Chart 3. Papers Selection Procedure .................................................................................................. 6 Chart 4. Comparison of Total Actual Execu on Effort & Total Es mated Execu on Effort ................ 73 Chart 5. Actual & Estimated Execution Effort ................................................................................... 73 Chart 6. Comparison of Total Actual Execution Effort & Total Estimated Execution Effort ................ 85 Chart 7. Actual & Estimated Execution Effort ................................................................................... 86

Seyed Morteza Hosseini – Master Thesis – 2015

IX

List of Equations Equa on 1. FEP Award Value .......................................................................................................... 15 Equa on 2. Ordering Score ............................................................................................................. 23 Equa on 3. APFD Rate of Fault ....................................................................................................... 27 Equa on 4. APFD Formula ............................................................................................................... 28 Equa on 5. APFDC Formula ............................................................................................................. 29 Equa on 6. UUCP ........................................................................................................................... 38 Equa on 7. TEF Formula .................................................................................................................. 39 Equa on 8. AUCP ............................................................................................................................ 39 Equation 9. Es ma on Rela on ...................................................................................................... 41 Equation 10. Test Execu on Complexity ......................................................................................... 42 Equation 11. Accumulated Efficiency .............................................................................................. 44 Equa on 12. Conversion Factor ...................................................................................................... 72 Equa on 13. Es ma on Execu on Effort Formula ........................................................................... 72 Equa on 14. Case Study 2 Conversion Factor.................................................................................. 84

Seyed Morteza Hosseini – Master Thesis – 2015

X

1. Introduction In modern software industry, having a high quality and trustable software is considered a key factor in developing a given company. The company needs its customers to have full confidence in the software products so, building this confidence is a big deal, but even bigger is maintaining it. One of the procedures on which the companies can rely to sort out this problem and produce products that are more reliable, is testing. In fact, the more accurate and earlier testing is done, the higher the quality of the software would be. In order to achieve such a high level of confidence and quality and to reduce the costs of the software, testing should be administered in early stage of the software development life cycle. As testing is a costly method in assuring the quality of software [34] and considering that testing can take 35% of total effort of software development [35], we should select the most appropriate ones among the huge number of possible test cases. Selecting an appropriate test case is, however, a challenging task as executing all test cases is impossible, time consuming, and costly. There are some methods by which test cases can be prioritized. Some of these methods have utilized average percentage of fault detection metric in order to prioritize test cases [1] [6] [5]. Some others can be classified in the system level category meaning that they use a high level architecture of the system to prioritize test cases [8] [3], but prior to prioritization of test cases we need to know the cost of each single test case including cost of test case creation and cost of test case execution. There with test execution effort, the effort on developing tests, creating test cases, getting prepared for test execution, and evaluating results should also be taken into consideration for better estimation [31]. However, the focus of this study would be on the estimation of test execution effort. The estimation of the test execution effort is a difficult task [30]. Several categories have been developed to estimate the cost of test execution effort, such as system level category [12] [19] [14] that belongs to this group, the second category is model-based that [28] can be considered in this category, and the third category is the group which utilizes requirements to estimate the execution effort, [14] [18] belongs to this group. In this study, we do a survey on these methods and also provide a new approach to estimate test effort execution by assigning a cost value to each test case based on some criteria that my vary regarding the target domain. The approach is evaluated through applying it to a desktop application and a web application. The rest of the study is organized as follow: Section 1 presents problem statement, our contribution in the mentioned problem, and research question. Section 2 describes research methodology, a new method that we employed in this research study. In Section 3, the systematic review protocol is described. Section 4 presents related works including the software cost estimation methods, the methods for prioritizing test case, and the methods for estimating test execution effort. Section 5 describes our new model designed for estimating test case execution effort. In Section 6, two case studies are presented to validate our model. Section 7 presents the discussion about the achieved results for case study 1 and case study 2. Finally, in Section 8 summaries and conclusions are presented.

Seyed Morteza Hosseini – Master Thesis – 2015

1

1.1

Problem Statement

Time and budget predictions are the most difficult tasks in software development process. Among the development phases, testing is the time consuming and costly process [15] due to the fact that it should be applied to all stages of the software development process. Static testing and dynamic testing, two different types of testing [15], can be considered as the main solutions in testing the system in order to build a high level of confidence for the system under test (SUT). In this regard, dynamic testing is believed to be more expensive compared to static testing [15] in that it requires SUT to be run. As such, one needs to predict the requirements of testing such as test preparation and test execution prior to the commencement of the testing phase in order to decrease the cost of testing. Test preparation incorporates such stages as test plan, test case creation, and so forth, which are not considered in this thesis work. Test execution, on the other hand, is a costly activity. The advantage of being informed about test execution effort is that one would be able to prioritize test cases with respect to the project limitation. To classify and order the test cases based on resources (budget, time, tools, and people), one needs to be familiar with the test cases effort. Actually, the problem is that we want to be able to estimate the execution costs of test cases so that we can prioritize and select them based on the amount of resources (time and/or money) that we have for testing. The root of the problem in another word is that we cannot execute all possible test cases. Estimating test execution effort is important in order to know the execution cost for reducing the cost of test process. There is not much investigation conducted on the estimation of test execution effort. In most existing methods that have been developed in code level, the estimation procedure could not have been accomplished in the early stages of the development process because it required the code. In this regard, few methods have been developed which have made use of architecture of the SUT such as utilizing use case and functional and non-functional requirements.

Seyed Morteza Hosseini – Master Thesis – 2015

2

1.2

Research Goals

Considering the aforementioned facts, the present study sets out to investigate the following research goals: 1. A survey on methods for prioritizing test cases. 2. A new approach for test case execution effort estimation. Test case prioritization area and test case execution effort estimation more specifically, are the main areas on which the research questions evolve. However, the second question is considered as the main research question to be addressed in this thesis work. It intends to seek methods for assigning cost/effort value for execution of test cases. As such, it prioritizes test cases based on the efforts required to execute them.

1.3

Contribution

In this study we have done a survey of literature on software cost execution, methods for prioritizing test cases, and methods for estimating the execution cost of test cases. The test case prioritization techniques that we have used in present survey determine the order of test case execution and do not estimate the execution effort of test cases. As these methods work in line of code level, does not have any estimation of test execution effort in early stage of software development life cycle. Hence, from the existing methods we concluded that estimating test execution effort later on software development or even after execution of test cases could not be useful rather than estimating the execution cost of test cases prior to test case execution. The main consequence of our survey bodes that knowing the execution cost of test cases in early stages of software development life cycle assists us in order to decide which test case should execute first with consideration of our limitation and priorities. We also utilized directly some criteria such as finish-to-finish and finish-to-start parameters from previous works or used indirectly some other criteria as a pattern in order to achieve new parameters. We also provide a new approach to estimate test effort execution by assigning a cost value to each test case based on some criteria that may vary regarding the target domain. In order to estimate the test effort in a way that none of the drawbacks of existing methods (the drawbacks of existing methods will be mentioned in the Related Work Section) is experienced, we have developed a new approach through which the effort in early stage of the SUT would be estimated. To this end, our approach incorporates the entire system rather than Line of Code (LOC) or use cases. Moreover, the model is based on the modules/components subsumed within each test case. As such, it uses the architecture of the system and operates in a high level. In fact, our model tactfully estimates the effort in the early stage of the system life cycle. Ultimately, the purpose of this study and the test case execution estimation is to prioritize test cases based on their costs. The operationalization of this model is arranged in the following steps: 1. Extracting the modules, those are involved in each test case. 2. Assigning values to each module based on the criteria that are listed in Table 11 (these criteria are flexible and can be updated based on the target domain). 3. Calculating a degree for executed test case based on the values assigned to the involved modules. In the worst case, it would be needed to execute a test case actually, while the projects are not similar or there is no historical data available for the test case. 4. Computing the test case effort.

Seyed Morteza Hosseini – Master Thesis – 2015

3

2. Research Methodology The first phase in conducting the study was to consult some systematic review guide papers [36] [37] in order to find the proper methodology and protocol for our research. Focusing on the guidelines suggested in these papers [38] [39] [40], we designed our protocol, which is completely elaborated in the Systematic Review Protocol Section. The main research question (the second one) is formulated based on the designed protocol. The papers were selected from the following sources based on the reviewing protocol with search terms and combination of them with AND, OR logical operation:    

ACM digital library IEEE Springer Link Google Scholar

The test execution effort estimation and test case prioritization were found in a total of 176 papers. The contribution of each source is shown in Chart 1.

Number of papers found in each source 6; 3%

19; 11% IEEE Springer Link

38; 22%

Google Scholar 113 ; 64%

ACM Digital Library

Chart 1. Number of Papers in each Source

In addition, four more thesis projects were added to the selected sources. Two groups of studies, i.e., test case prioritization methods group and test case execution effort estimation methods group, were considered in a way so as to manage the final sources.

Seyed Morteza Hosseini – Master Thesis – 2015

4

The first step was to read the titles of all appropriate papers. In cases in which the titles were clear enough to recognize, they could be transferred to the relevant group of study. Alternatively, in the case where the title was not applicable, the study area was excluded from the sources. Reading the abstract of the selected papers was considered as the next step. To put it another way, the clarity, and applicability of the abstract was determined as the criterion for its transferring to the relevant group of study. Similarly, this selection process was employed for the Introduction and Conclusion Sections of the papers. Finally, 148 papers were opted as the target sources, out of which 36 papers were ultimately chosen. The number of papers related to the test case prioritization methods and test execution effort estimation methods is illustrated in Chart 2 and the selection procedure for papers is shown in Chart 3.

Number of Papers in each Group 140 120 100 80 124

60 40 20

28

24 0 Estimation group

Prioritization group

Excluded

Chart 2. Number of Papers in each Group

Seyed Morteza Hosseini – Master Thesis – 2015

5

Papers Selection Procedure 200 180

Number of papers

160 140 120 100 80

176

60 40 20

36

3

3

0 Reading Title

Reading Abstract

Reading Introduction Reading Conclusion

Chart 3. Papers Selection Procedure

3. Systematic Review Protocol In this section, we will describe the systematic review protocol, which deals with performing a plan to achieve review goal. Actually, the research question, research keywords and their synonyms, as well as their abbreviation and definition, are included in the review protocol. We would also describe the source selection procedure, search strings, and source selection criteria.

3.1

Question Formulation

Question Focus: To identify test case prioritization techniques and assign cost/effort value to test case execution.

3.2

Question Quality

Problem: Software testing needs test case execution. It is necessary to verify, prior to the test execution, to see if the selected set of test cases for execution is suitable. The purpose of such verification is, therefore, to prevent improper selection of the test cases for execution, which can affect the quality of the final product. Question: What cost/effort value would be determined for the test cases?

Seyed Morteza Hosseini – Master Thesis – 2015

6

Keywords and Synonyms Search Term

Synonym

Abbreviation

Test Case

TC

Selection

Choose, Pick

Prioritization

Rate , Rank, Range, Order, Grade, Place, Estimate, Set

Software

Cost

Estimation

Assign Methods Execution Time Static analysis Model base analysis Non-functional analysis Testing Test script

SW

Effort, Attempt Approximation, Measure, Quantify, Determination, Evaluation Allocate, Portion, Give, Specify, Put, Apply, Set Technique, Strategy, Plan, Planning, Scenario, Schema Running Schedule Study, Examine

Evaluate, Examine Table 1. Keywords and Synonyms

Seyed Morteza Hosseini – Master Thesis – 2015

7

Definition A case should try something in order to find out about it. The act of choosing or selecting. Ordering test cases to catch the best execution arrange. A computer written program including documentation of a computer system. Value must be assign to test case by which will be decided to run test case in which order. An approximate calculation of quantity or degree or worth.

3.3

Sources Selection

The Criteria Considered in Selecting Sources:  Availability of articles on the web.  Presence of search mechanisms using keywords.  Availability of papers in the electronic form. Studies Language: English. Sources Search Method: The applied method is searching through web search engines.

3.4

Execution Evaluation

In order to execute search strings in web search engines, due to different criteria employed in each individual web engine, we had to adopt our search strings with the limitations existing in each web search engine because they did not operate equally with logical operators. Search Strings 

(test case OR test script) AND (selection OR choose OR pick)



(test case OR test script) AND (selection OR choose OR pick) AND (methods OR techniques OR strategy OR plan OR planning OR scenario OR schema)



(test case OR test script) AND (prioritize OR prioritization OR rating OR ranking OR ranging OR order OR grading OR place OR setting)



(test case OR test script) AND (effort OR cost OR attempt)



(prioritize OR prioritization OR rating OR ranking OR ranging OR order OR grading OR place OR setting) AND (methods OR techniques OR strategy OR plan OR planning OR scenario OR schema)



(effort OR cost OR attempt) AND (assign OR allocate OR portion OR give OR specify OR put OR apply OR set)



(effort OR cost OR attempt) AND (estimation OR approximation OR measure OR quantify OR determination OR evaluation)



(effort OR cost OR attempt) AND (estimation OR approximation OR measure OR quantify OR determination OR evaluation) AND (methods OR techniques OR strategy OR plan OR planning OR scenario OR schema)



(effort OR cost OR attempt) AND (assign OR allocate OR portion OR give OR specify OR put OR apply OR set) AND (methods OR techniques OR strategy OR plan OR planning OR scenario OR schema)



(test case execution) AND (prioritize OR prioritization OR rating OR ranking OR ranging OR order OR grading OR place OR setting)

Seyed Morteza Hosseini – Master Thesis – 2015

8

Sources List The following sources have been considered based on their credibility.    

ACM Digital Library Scholar.google.com IEEE Springer Link

Sources Selection after Evaluation As the listed sources are among the most popular sources, all of which can be claimed to enjoy high quality criteria.

References Checking: All the references selected for this study are from reliable sources.

Studies Definition Studies Inclusion and Exclusion Criteria Definition  

The sources must be in English. Present techniques of test case prioritization and test execution effort estimation processes prior to their execution.

Studies Type: All kinds of studies selected for the purpose of investigation are in line with the research topic. Procedures for Studies Selection First, the search strings were administered in the selected sources. To select an initial set of studies, the titles of all selected studies were read and if a title was in line with the scope of the study, the study could be opted for investigation. In the next stage, the abstracts of all selected studies were read. In cases where an abstract was related to the research field, the Introduction and Conclusion Sections were then read. To refine this initial set of studies, their full texts were also read.

3.5

Information Extraction

Information Inclusion and Exclusion Criteria Definition: The extracted information from the corpus data contains techniques, methods, and strategies for describing and evaluating the test case prioritization and test execution effort estimation processes.

Seyed Morteza Hosseini – Master Thesis – 2015

9

3.6

Results Summarization

The results summarization Section will describe the total overview of the study results including the procedure of presenting the results, comments about number of studies, search selection bias, result application and finally recommendations (if any).

Results Presentation in Tables and Charts: Relevant tables and charts have been used in order to reveal the results. Final Comments Number of Studies: initially, a total of 176 studies were taken into consideration, out of which 36 were selected. Search, Selection, and Extraction Bias: None of them has been defined. Publication Bias: has not been defined. Inter-Reviewers Variation: There was no variation. Results Application: The results of this study suggest that assigning cost/effort values to test cases and verifying test case execution processes must be determined prior to their execution. Recommendations: No one.

Seyed Morteza Hosseini – Master Thesis – 2015

10

4. Related Work Many methods and researches have been conducted on the software testing area during past several years, some of which have been done on the test case prioritization, e.g., these methods have utilized average percentage of fault detection metric in order to prioritize test cases [1] [6] [5] and other ones can be classified in system level category[8] [3]. The others have been carried out on the test execution effort estimation among them [12] [19] [14] are in system level category (the system level methods are the ones that can be used in early stages of SDLC to provide high-level estimates of the execution cost of test cases [19]). System level and function level categories have been defined in [4], [28] is in model-based category, and [14] [18] utilize requirements to estimate the execution effort. In continuance, we have described a concise explanation of estimation methods that have generally been applied in the software settings. Following that, test case prioritization techniques and methods of estimation test case execution effort have been presented.

4.1 Software Cost Estimation Oziegbe [29] believes that the methods utilized in the software cost estimation process are classified into two groups, i.e., algorithm and non-algorithm methods. Oziegbe in [29] further explains that algorithm methods are the ones, which use mathematic while non-algorithm methods are those, which make use of human judgments. According to Oziegbe [29], the following methods are considered as non-algorithm methods: 1. 2. 3. 4. 5. 6.

Analogy costing Expert judgment Parkinson’s law Price-to-win Bottom-up (Engineering-build) Top-down

In addition, Jayakumar and Abran have classified the techniques [41] based on the: 1. 2. 3. 4. 5.

Judgment and rules of thumb Analogy and work breakdown Factors and weights Size Fuzzy and other models

The process of comparing current projects to the previous ones proceeds in analogy costing method. One of the drawbacks for analogy costing is that a similar project should be available and the previous project and the new one should be in the same domain in order for them to be compared and to estimate their efforts. However, in such cases, the estimation process is easy to be accomplished as we can use historical data of the old project. Oziegbe [29] does not recommend the Parkison’s law as its estimation is not realistic.

Seyed Morteza Hosseini – Master Thesis – 2015

11

4.2 Prioritizing Test Case Methods In this Section, we explain 31 methods employed on the test case prioritization process including the code base (code coverage) methods, fault detection methods, system-level methods, and functionlevel methods. The code-base methods are the ones, which use the code or require the code in order to prioritize test cases, the fault detection methods are the methods that use average percentage of faults that are covered by each single test case, and function level methods use functions, which are covered by the test cases. The important point that should be considered before going forward is that two different concepts of test case reduction and test case prioritization should not confuse us. Evidently, test case reduction is a method through which we can reduce the number of test cases, while in test case prioritization we do not omit any test cases; but rather, we just pick them up in the desired order.

4.2.1

Historical-Based Methods

In historical-based methods group we present the methods, which use the information of the similar or previous projects in order to prioritize test cases.

4.2.1.1 Test Case Prioritization for Black Box Testing Test case prioritization for black box testing [10] uses the historical information and executed information. Actually, it classifies the test cases into different categories based on the kind of fault they reveal and, consequently, adjust their priority upon their given categories. Grouping the test cases is based on the relation matrix R, which predicts the fault detection relation between the test cases. The algorithm for producing matrix R is illustrated in [10] to which you can refer for more details. When a test case reveals a fault, the test cases, which are in the same group (i.e., related test cases), will be given higher priority. The prioritization process is illustrated in the following sample (example is given from [10]): Adjusting range α=2 Original order=t2, t3, t4, t5, t6

Seyed Morteza Hosseini – Master Thesis – 2015

12

t1, t2, t3 are in one group and t2, t4 in the other group

Figure 1. Rela on Matrix R [10]

Escalate algorithm: (detail about escalate algorithm can be found in [10]) When test case tm is tested, the next test case tm+1 will be given the highest priority; hence no need to escalate its priority. In our example, since the test case t1 is executed, the next test case, which is t2 does not need to escalate (t1 and t2 ignored). As t1, t3, and t6 are in the same group (the solid lines in relation matrix), t3 and t6 would be adjusted. t3 adjusted to highest priority, t4 and t5 are not in this group, so they do not need to be adjusted. The t6 escalates to higher priority (as adjusting range value is 2). Step 1 2

t3 t3

t2 t2

Order t4 t5 t6 t4

t6 t5

Table 2. Escalate Order [10]

Deescalate algorithm: (detail about deescalate algorithm can be found in [10]) Deescalate algorithm involves all test cases in the reverse order. As the test case t6 has the lower priority, it is ignored. t5, t4, t2 do not need to be deescalated as they are not in t1’s group. t3 is adjusted to lower priority (two positions lower as the adjusting rang (α) is 2). Step 1 2

t2 t2

t3 t4

Order t4 t5 t5 t3

t6 t6

Table 3. Deescalate Order [10]

Seyed Morteza Hosseini – Master Thesis – 2015

13

4.2.2

Statement-Based Techniques

In statement-based methods group, on the other hand, we present the methods, which use number of statements covered by each single test case that will be considered in order to prioritize test cases. Six methods will be presented in this group.

4.2.2.1

Total Branch Coverage

Total branch coverage [2] [7] uses the information of unmodified version of program (i.e., the version of program before execution) and does not need prior knowledge of faults, which is considered an advantage of this technique. It also uses the information of outset. Total branch coverage technique classifies the test cases in decreasing order in terms of the number of decisions (branches) each test case covers. A disadvantage of this technique, however, is that some branches may not be dealt with by any test case after executing the test cases. The amount of time this technique needs to run n branches is O (n log n) [2].

4.2.2.2

Additional Branch Coverage

Additional branch coverage [2] [7] recalculates coverage information for each test case that is not prioritized. This technique classifies test cases in the decreasing order in terms of the number of additional decisions (branches) each test case covers, which might have not been covered already. It deals with the process of choosing the test case that covers the greatest branch and set the coverage information. This process is reiterated several times until all branches are covered by at least one test case. This latter merit of the technique, i.e., covering all branches by at least one test case, is actually, what makes the difference between additional branch coverage and total branch coverage. The cost for running this technique for n branches is O ( ). [2]

4.2.2.3

Total Statement Coverage Prioritization

Total statement coverage [2] [4] [7] does not consider fault probability to cause a failure in relevant branch/statement for the related test case. In the same vein as total branch coverage technique, this technique also uses the information of unmodified version of program as well as the information of outset. That is why it does not need prior knowledge of faults, which is considered as one of its advantages. As mentioned above, total statement coverage technique is somehow similar to total branch coverage, but the only difference between two techniques is that the former covers statements rather than branches/decisions. The time duration required to execute this technique is O (mn+m log m) where m is the number of test cases and n is the number of statements. mn signifies the time needed in counting the statements covered by each test case, and m log m signifies the time required for classifying the test cases [7]. Assumption is that n is greater than m, so the time is O (mn).

Seyed Morteza Hosseini – Master Thesis – 2015

14

4.2.2.4

Additional Total Statement Coverage Prioritization

Additional total statement coverage [2] [4] [7] uses the feedback in order to focus on the statements not covered yet. Additional statement coverage is the same as additional branch coverage in that it covers statements rather than branches/decisions. A disadvantage of both branch and statement coverage techniques is that there is no guaranty to expose a fault by executing a test case and cover faulty statement because these techniques only take into account the coverage of a branch or statement by a test case. Consideration of fault ability/probability to cause a failure in relevant branch/statement for related test case is an important issue, which is ignored. The cost of additional statement coverage is O ( n) where m stands for the number of test cases and n for the number of statements.

4.2.2.5

Total Fault Exposing Potential (FEP) Prioritization

The fact that a fault can be exposed by a test case depends not only on the test case to be able to execute the relevant component but also on the ability of the fault to cause a failure in relevant component regarding that test case. This fault’s ability to expose a failure is named fault exposing potential (FEP), which should be determined in approximation. Total fault exposing potential [2] [4] [7] is an expensive method to be administered. The cost of measuring ms(s;t), which is mutation score, for each statement is high. It uses the information of the program before execution (unmodified version of program). It also makes use of the outset information. Actually, it is more expensive compared to the code coverage techniques. Total fault exposing potential is concerned about exposing a fault by a test case, which is based upon both executing test case of the relevant statement and the probability of fault to cause a failure in that statement. Total fault exposing potential technique orders the test cases based on the following calculations [2]: for given program P, statement s € P, test suite T, test case t € T, and ms(s; t) which is mutation score for each statement, s as the ratio of mutants of s exposed by test case t [2].



=

(

( ; ))

Equation 1. FEP Award Value [7]

A disadvantage of FEP is that the cost of calculating ms(s;t) is high. Total fault exposing potential prioritization classifies the test cases in order of these award values. The cost of FEP for m test cases and n statements is O(mn+mlogm) in time. In cases where n is greater than m, the cost is O(mn).

4.2.2.6 Additional Total Fault Exposing Potential (FEP) Prioritization One of the prerequisites of this technique is the prior knowledge of the faults. It uses the feedback. Expanding total fault exposing potential technique leads to additional total fault exposing potential

Seyed Morteza Hosseini – Master Thesis – 2015

15

[2] [4] [7] technique. The main difference between these two techniques is the assigning of the lower award values for all test cases t that cover statements.

4.2.3

Function Level Techniques

In function-level methods group, we present the methods that use the number of functions, which are covered by each single test case considered in order to prioritize test cases. We will present 12 methods in this group.

4.2.3.1 Total Function Coverage Prioritization Total function coverage [4] is less expensive compared to the total statement coverage technique. Total function coverage prioritization orders the test cases based on the number of functions executed by each test case. The cost of this technique is O(mn+m log m) where m is number of test cases and n is number of functions. Typically n is smaller than m.

4.2.3.2 Additional Function Coverage Prioritization Additional function coverage prioritization [4], classifies the test cases with respect to the total number of additional functions each test case covers. When covering all the statements, reset the coverage vectors and applies additional function coverage on remaining test cases once more. The cost of this technique for m test cases and n functions is ( ).

4.2.3.3 Total FEP Prioritization Total FEP Prioritization [2] [4] [7] technique is similar to FEP in the statement level with replacing statements by functions.

4.2.3.4 Additional Total FEP Prioritization This technique acts as additional total FEP [2] [4] [7] statement level but with an operation in the function level.

4.2.3.5 Total Fault Index (FI) Prioritization Fault index is one of the techniques, which Elbaum et al [4] have mentioned in the function level techniques. Total fault index [4] uses the information of p (version of the program before execution/unmodified version of program), and the information of p′ (version of the program after execution/modified version of program). In total fault index prioritization technique, it has been supposed that each function has a potential talent to have fault in the modified version of that function

Seyed Morteza Hosseini – Master Thesis – 2015

16

called fault proneness. Bearing in mind that P is considered as a program and P′ as a modified version of P, the following steps are required to generate fault indexes for P′: 1- Production of fault index for each function in P. 2- Production of fault index for each function in P′. 3- Comparison function by function indexes for P′ against P. The above steps are consecutively performed to generate the fault index for each function in P′ based on changes brought about by that function. Total fault indexes for each test case would be calculated based on the aggregate number of the fault indexes for that test case across the functions, which have been covered by the target test case. Test cases are classified in descending manner with respect to their total fault index. There is no need to execute test cases, so the cost is lower than that of FEP. The time for executing m test cases and n functions is O(mn).

4.2.3.6 Additional Fault-Index (FI) Prioritization Additional fault index prioritization [4] technique succeeds the following steps: 1- Put all the functions, which have been covered already by test cases in a set of functions. This set of functions is called SOF (if all the functions contain in the SOF, the SOF reset to 0). 2- In order to find the next best test case, we calculate sum of fault indexes for each test case for all functions, which the related test case has covered, except the ones in SOF. 3- The winner test case, which has the next higher priority to execute, is the one with greater fault index value at step 2. 4- This process continues until all test cases are prioritized.

Seyed Morteza Hosseini – Master Thesis – 2015

17

The above process has been depicted in the Figure 1:

SOF

Set of functions that have been covered by previously executed test cases

Do all functions exist in SOF?

YES SOF=0

NO

Calculation the sum of fault indexes for each TC for all functions, which that TC has covered, except the ones in SOF.

Print the TC with greater sum FI

NO

Do all TC have been prioritized?

YES

Seyed Morteza Hosseini – Master Thesis – 2015

18

Finish

Figure 2. Additional Fault-Index Process

4.2.3.7 Total FI with FEP Coverage Prioritization This technique is analogous to total FEP coverage prioritization technique, with the only difference that is on the estimation of FI for each function. Actually, in total FI with FEP coverage technique [4] we have to estimate FI in addition to estimation of FEP for each function, which is executed by a test case. The next step is the calculation of sum of these values (FI+FEP) for each test case toward functions executed by that test case. The test case with greater summation is the next one in descending order of test case prioritization.

F1 F2 Total

FI FEP FI FEP

TC1 1 2 2 4 5

TC2 3 4 1 1 9

TC3 5 2 8 3 18

Table 4. Total FI with FEP [4]

As illustrated in Table 3, the total estimation of FI and FEP to TC3 is greater than other ones, so TC3 is the next one in prioritization.

4.2.3.8 Additional FI with FEP Coverage Prioritization The process using in additional FI with FEP coverage prioritization [4] technique is similar to that of additional FI coverage prioritization. The following steps are administered to classify the test case in ascending order: 1. Estimate the FI and FEP values for each function. 2. Compute sum of FI and FEP for each test case that is not prioritized yet across the functions executed by that test case. 3. Select the test case with the greatest total value. 4. Set the value of FI and FEP to zero for the functions, which are covered by the selected test case. (This step is equal to step 1 in additional FI coverage prioritization that put all the functions which have been covered already by test cases in a set of functions) 5. Repeat step 2 to 4 until all the functions values are zero. 6. If any test case remains, reiterate the step 1 to 5.

Seyed Morteza Hosseini – Master Thesis – 2015

19

4.2.3.9 Total DIFF Prioritization Total DIFF technique [4] uses the information of unmodified version of program that is the version of program before execution (p) and p′ (modified version of program). Given that p′ is the modified version of p, total DIFF prioritization technique relies on the difference between p and p′. The change rate obtained is based on the number of lines which are listed by UNIX diff command as inserted, deleted, and changed lines both in p and p′.

4.2.3.10 Additional DIFF Prioritization Additional DIFF [4] technique is similar to additional FI prioritization but is applied to diff output.

4.2.3.11 Total DIFF with FEP Prioritization Total diff [4] is similar to total FI with FEP prioritization, but is applied to diff command output.

4.2.3.12 Additional DIFF with FEP Prioritization Additional DIFF with FEP prioritization [4] is quite like additional FI with FEP prioritization, except for its application to diff command output.

Seyed Morteza Hosseini – Master Thesis – 2015

20

4.2.4

System Level Techniques

4.2.4.1 Prioritization Using Relevant Slice Prioritization using relevant slice [3] is a system level technique that uses the information of p and p′, unmodified and modified version of the program, respectively. Modified version of the program is the version of program before execution and unmodified version of program is the version of program after execution. It needs prior knowledge of faults. This technique takes into account the number of statements/ branches that affect or have potential to affect the output of the relevant test case as well as total statement/branch coverage. We will illustrate this technique by using the following example. The darker-shaded lines indicate the relevant slice. This example is given from [3]. 1: read(a, b, c); 2: int x=0, y=0, z=0; 3: x := a + 1; 4: y := b + 1; 5: z := c + 1; 6: int w := 0; B1: if x > 3 then B2: if z > 4 then 7: w := w + 1; endif endif B3: if y > 5 then 8: w := w + 1; endif 9: write(w); Figure 3. Relevant Slice Technique Example [3]

When the execution of the above code with given input (a,b,c)=(1,5,4) reaches B1, the value of x becomes equal to 2, so B1 evaluates to false and the value of y becomes 6, so B3 evaluate to true later on. Accordingly, the output value of the program would be one. In this case, the line 3 statement changes to x=b+1 by mistake. By this mistake, the value of x becomes 6 when reaching to B1, so it evaluates to true, subsequently B2 evaluates to true and the value of w in line 7 is equal to 1. Continuing executing B3 evaluates to true (the value of y is 6 at this step) there upon the value of w is 2 as input. This example plays up that line 3 included in the relevant slice has the potential to affect

Seyed Morteza Hosseini – Master Thesis – 2015

21

the output. The order of test case in decreasing order for this technique is based on the test case weight, which determines [3], as would follow: Test case weight = number of requirements in the relevant slice + total number of requirements exercised by the test case.

4.2.5

Coverage-Based Techniques

In coverage-based methods group, we present the methods that use the information covered by each single test case and will be considered in order to prioritize test cases. This group contains two methods.

4.2.5.1 Fault Localization Technique Fault localization [5] is an automatic debug technique, which software developers use in order to decrease the debugging time. This measure is performed through finding the position of the faults in code. By doing this, the information about the position of fault or faults in code is revealed. The main objective of utilizing the fault localization is that parts of the program being covered by a failed test case are more capable of having the fault/faults than the other parts, which have been executed by a passed test case [5].

4.2.5.2 Fault Aware Test Case Prioritization (FATCP) Previous test case prioritization techniques such as statement-level and function-level techniques only used coverage information of the test cases in order to classify test cases. Fault aware test case prioritization technique [5] is a technique, which uses both coverage information and historical information of test cases so as for them to be prioritized. To obtain this information, FATCP employs an automatic tool named APT (APFD Parsing Tool), which estimates APFD values and the order of test cases for execution. Historical information of test case, on the other hand, is calculated through fault localization technique (FL). The reason behind using fault localization technique, as mentioned in [5], is to improve rate of fault detection. To do so, FATCP uses a fault localization technique called TARANTULA. (For more information about TARANULA, refers to [5] Section 2.3 and for complete information refer to [7] in [5]). TARANTULA gives a suspiciousness score to each coverage entity, which indicates the probability of containing faults for each coverage entity. The more suspiciousness score, the more probability of existing fault. Subsequently, higher suspiciousness score means higher rate of fault detection for that test case. The FATCP algorithm has been depicted in Figure 3.

Seyed Morteza Hosseini – Master Thesis – 2015

22

Figure 4. FATCP Algorithm [5]

The process of FATCP is arranged in the following orders: 1. Assigning an order score(OS) to each test case, calculating based on equation 5: Given n test cases in a test suit, i position of related test case in prioritized queue.



(

)=

+1−

Equation 2. Ordering Score [5]

2. Assigning suspiciousness score (SS) to each test case: Prior to executing test suit, test cases with higher suspiciousness score have higher rate of fault detection. However, after executing the test suit, the faults covered by test cases would be removed and, as a result, the rate of

Seyed Morteza Hosseini – Master Thesis – 2015

23

fault detection of the target test case decreases. It means that the higher the suspiciousness priority is the lower the rate of fault detection would be. As such, the new order of the test case should be adapted. To do so, we must replace the suspiciousness score of test cases with the opposite suspiciousness score to test cases. Sejun Kim et al [5] believe that in comparison with coverage-based techniques, FATCP has higher rate of fault detection. The good point about FATCP is that it can be applied to every coverage-based technique as it uses coverage information.

Seyed Morteza Hosseini – Master Thesis – 2015

24

4.2.6

Model-Based Techniques

In model-based methods group, we present the methods that use information of the SUT model in order to prioritize test cases. We will present three methods in this group.

4.2.6.1 Test Prioritization Using System Model Model-based prioritization [8] utilizes the state-based model of system. Actually, model-based prioritization makes use of the information of executing system model due to the fact that administering the system model is faster and less expensive than the system itself. As this technique uses state-based model, executing system model means to execute an Extended Finite State Machine (EFSM). It assumes that the model of the system is executable. By doing so, it takes advantage of executing the model of system as well as the modified model of system in order to obtain the information about changes that are made in the system. The state-based model employed by this technique is Extended Finite State Machine (EFSM). EFSM, an abbreviation used for Extended Finite State Machine, is used to model the state-based systems. The following components are subsumed within the EFSM: state, transition, action, condition, and event. At the time of occurring a condition is validated, if the condition validation is true, then a transition is made and transferred from one state to the other. As a consequence of this process, an action takes place. A sample model is shown in the Figure 4.

Figure 5. EFSM Sample Model [8]

The change, which model-based prioritization considers in modified model of the system, is to gather the information to comprise added and deleted transitions. Any other change that occurs in the system is considered in line with these two kinds of basic changes (adding transition and deleting transition). The process employed in this technique includes the following steps: 1. Identifying differences between two models (primary model and modified model) by recognizing the changes in the modified model of system. 2. Executing modified model for test cases in order to collect information.

Seyed Morteza Hosseini – Master Thesis – 2015

25

3. Prioritizing test cases: different methods used to prioritize test cases based on the information that have been gathered in the previous step. Two kinds of methods that are used in [8] includes:  Selective test prioritization  Model dependence-base test prioritization

4.2.6.2 Selective Test Prioritization The main purpose of Selective [8] method is to assign high priority to the test cases that have covered the modified transition, and to assign low priority to those ones that have not covered any modified transition. Selective method is comprised of two versions: The first version considers only added transitions and, as a result, assigns high priority to test cases, which have given an account of the modified transitions and low priority to the test cases that have not taken into coverage any modified transition. The second version, on the other hand, looks upon both added transitions, deleted transitions and then follows the same scenario for assigning priority to test cases as first version. The precise details of selective method can be found in [8].

4.2.6.3 Model Dependence-Based Test Prioritization The objective of the model dependence-base [8] method is to use model-dependence analysis for achieving information about ways through which added transition and deleted transition interact with other parts of the system and using this information in the test case prioritization process. Two kinds of model dependence analysis have been utilized in [8].

Seyed Morteza Hosseini – Master Thesis – 2015

26

4.2.7

Random Ordering

The first and simplest method in prioritizing test cases is to order them randomly. Random ordering [2] [4] [7] technique can be administered in the experimental manner.

4.2.8

Optimal Ordering

Optimal ordering technique [2] [4] [7] uses the information of program before execution. As a result, it needs prior knowledge of the faults and test cases. Accordingly, it is not a proper technique in case of limitation to prioritize test case before execution. In order to achieve objective of this method that is an optimal ordering of test cases in a test suite, we need to know which fault is exposed by each test case. Considering these facts, the optimal ordering technique cannot be respected as a practical method.

4.2.9

Average Percentage of Fault Detected (APFD)

APFD [1] technique uses the information of the program before and after execution (unmodified and modified version of program). As such, one of the disadvantages of APFD technique is that running all test cases is a requirement and one needs to know execution time of test cases. This technique also uses the test case information pertaining to execution phase, so it needs prior knowledge about faults and test cases. APFD technique puts test cases in an order using average number of faults, which are detected by each test case during its execution time called rate of fault. Rate of fault for each test case is estimated based on the following formula:

Rate of fault (RF) =F /T

Equation 3. APFD Rate of Fault [1]

Where F is the number of faults detected by each test case, and T is test case execution time in minute. The higher fault detection ratio means the higher test case priority. In Table 2 you can observe ten sample test cases, TC1 to TC10 and faults have been detected by each test case.

Seyed Morteza Hosseini – Master Thesis – 2015

27

TC1 F1 * F2 F3 F4 F5 F6 F7 F8 F9 F10 * Total Fault Time

TC2

TC3

TC4

TC5

TC6 *

* *

*

TC7

TC8

TC9

*

*

*

*

TC10

* *

*

* * *

*

*

*

* * *

2

3

1

5

7

11

3

2

3

2

2

2

3

4

10

12

6

15

8

9

Table 5. APFD Rate of Fault [1]

Test case 1 rate of fault detection=2/5=0.4 Test case 2 rate of fault detection =3/7=0.42 Test case 3 rate of fault detection =1/11=0.09 Test case 4 rate of fault detection =3/4=0.75 Test case 5 rate of fault detection =2/10=0.2 Test case 6 rate of fault detection =3/12=0.25 Test case 7 rate of fault detection =2/6=0.33 Test case 8 rate of fault detection =2/15=0.133 Test case 9 rate of fault detection =2/8=0.25 Test case 10 rate of fault detection =2/9=0.22 According to the rate of fault detection mentioned above, TC4 has the highest priority and TC3 has the lowest priority. Accordingly, the order of test cases based on their priority is as follows: (This example is taken from [1]) T4, T2, T1, T7, T6, T9, T10, T5, T8, T3 The advantage of APFD technique is that it would be able to detect higher average number of the faults. This technique would be applicable for evaluating test case orders. We can evaluate the above order by using the following formula suggested by [1]:

=1−



1

+

2+ . . . . . . . . +



+

1 2

Equation 4. APFD Formula

Where m is the number of faults that is 10, n is the number of test cases which is 10, and TFi is the position of the first test case in the above order that is an indication of the fault i. As an example, consider the above order of test case execution. In this example, the first test case TC4 would not be able to find F1; this is the case for the next test case, viz, TC2; finally, TC1, as the third case in this order, would find F1. As seen in the order mentioned above, the position of TC1 in the order

Seyed Morteza Hosseini – Master Thesis – 2015

28

execution queue is 3, so TF1 is 3. The same case occurs for TF2, based on the above test case order, the first test case, which exposes F1 is TC4; and finally, as TC4 is situated in the first position, TF2 in this case would be 1. By putting the values in APFD equation, the following weight of APFD would be revealed: = 1−

3

+ 1 + 2 + 4 + 2 + 1 + 1 + 2 + 5 + 3 1 + = 0.81 10 ∗ 10 20

If we calculate the same values for non-prioritized test case, the APFD value would result as follow:

=1−

1

+4+2+7+2+4+5+3+6+1 1 + = 0.70 10 ∗ 10 20

The APFD values for prioritized and non-prioritized test cases indicate a higher value for prioritized test cases, compared to non-prioritized ones. As can be seen, based on this evaluation suggested by [1], this technique needs prior knowledge of the faults and execution time of the test cases and, as a consequence, it cannot be a proper technique to use in prioritizing test cases before their execution. Another disadvantage of this technique is that in case of having many test cases with long execution time, and also having time constraint, achieving the result takes a long time as running all test cases is required.

4.2.10

Average Percentage of Fault Detected Cost Cognizant/Aware (APFDc)

In order to prioritize test cases, the previous technique has some assumptions to be taken into consideration, which would follow: all faults have equal cost; all test cases have equal cost. The cost can be interpreted in various ways (time, hardware, engineer’s salary, human time, and etcetera). If these requirements are met, then APFD performs as is expected. However, in practice, not all faults have identical cost and there are some test cases with different costs. In such situations, APFD does not perform as is expected before [6]. APFDc [6] unlike APFD considers cost of the faults and test cases. Cost of a test case can include time of execution, setup, validation, and human time [6]. Similarly, cost of a fault, which is called fault severity, can be comprised of the time to alter it or other costs such as losing time to market, damage to assets, and so on [6]. The average percentage of fault detection with cost consideration is calculated using the equation 3. [6]

Equation 5. APFDC Formula [6]

Where T stands for the test suit, n stands for the number of test cases in test suit T. t1, t2, … , tn are test costs, F is fault suit, m is number of faults in fault suit F. f1, f2, …, fm are fault severity(cost). TFi is the first test case in T′ that has revealed the fault i, T′ is prioritized version of T. If all test costs are equal and all faults costs are similarly equal, then the equation is the same as APFD [6].

Seyed Morteza Hosseini – Master Thesis – 2015

29

4.2.11 Case-Based Ranking Test Case Prioritization Case-Based Ranking (CBR) [9] is one of the numerable techniques, which need user knowledge in prioritization process. CBR utilizes user knowledge as well as some other prioritization criteria such as complexity, level of coverage, and historical information with association of machine learning algorithm. Due to space limitation, for more information about learning algorithm refer to [9]. A highlevel view of CBR is shown in Figure 5.

Figure 6. CBR High Level View [9]

Where the TS is test suit, {t1, …, tn} are unordered test cases, f1, …, fm are prioritization indexes, and (ti, tj) are test case pairs. In case of disability for deciding prioritization merely based on prioritization indexes, CBR takes into consideration user knowledge iteratively throughout learning algorithm. In order to produce prioritized test cases, the learning algorithm can discontinue at any time. Some advantages of CBR are listed below:     

Multiple indexes: Unlike previous techniques, CBR uses multiple indexes to prioritize test cases. User centered: unlike the other methods, CBR uses user knowledge and make user involved in prioritization process. Partial information: In case of incompetency of information, CBR can perform as powerful as expected before. Incoherent data: CBR is robust to tackle with adversative information. Wide applicability: CBR can apply in any stage of testing, that is, before execution and during regression testing.

Seyed Morteza Hosseini – Master Thesis – 2015

30

4.2.12 Bayesian Network-based Prioritization The process of Bayesian Network (BN) [11] includes the following three steps: 1) Extracting evidences: Gathering information from different sources such as quality metrics, code changes, and test case coverage. 2) Building BN: to be used in order to estimate the probability of the test cases. (The precise explanation of building BN can be found in [11]). 3) Ordering test cases: The ordering of test cases is arranged in the following three stages: a) Probabilities inference: investigate the succeeding probability of a test case bases on the model. b) Test case selection: selecting some test cases and adding them to the ordered list. c) Updating BN: the test cases, which added to the ordered list in step (b) utilize to modifying the model as the probability of test cases which overlap the same areas decrease. The produced model will transfer to the step (a), and this process will continue as far as all the test cases prioritize. The high-level view of BN-based prioritization is demonstrated in Figure 7.

Figure 7. BN Model Process [11]

Seyed Morteza Hosseini – Master Thesis – 2015

31

4.2.13 Test Case Prioritization Techniques Summary Throughout this thesis work, we have investigated test case prioritization techniques and have grouped them: 1. 2. 3. 4. 5.

Statement level techniques Function level techniques Diff techniques Fault index techniques FEP techniques

Some of these techniques do not need prior knowledge of the faults and test cases before execution. Some others use the information related to the modified version of the program (the version of program after execution) and get feedback. The gathered feedback will be utilized to improve the rate of fault detection. The other kinds of prioritization techniques such as total techniques need knowledge of faults and test cases, prior to their execution. The other important aspect of prioritization techniques is consideration of practical aspect of such techniques. The techniques mentioned at the start part of this summary are practical ones, whereas FEP techniques are not feasible and need more investigation. Sejun Kim, et al [5] believe that both statement-level and function-level techniques are subsumed within the coverage-base techniques group using the rate of fault detection to prioritize test cases. Disadvantages of coverage-based techniques can be enlisted as follow: 

 

Because the faults, which were detected by a test case, will be removed after detection, so the test case, which has executed them already, does not have performance as expected. Actually, the rate of fault detection related to relevant test case will decrease. Coverage information consideration exclusively. Test case rate of fault detection will not change even when the removing faults were detected by the relevant test case; subsequently the priority of that test case will not change and should be adjusted.

We have listed the test case prioritization techniques in Table 6 including the following information about each technique such as technique name, description, time, etc. 

Technique name: technique name presents name of the technique.



Mnemonic: mnemonic illustrates the mnemonic name of the technique that would be easier to be memorized.



Description: The description presents a short description of how the technique performs.

Seyed Morteza Hosseini – Master Thesis – 2015

32



Time: time illustrates how long the technique takes to be executed.



Using the information of P/P′: This column in Table 6 describes that the technique uses either the information of the version of program before execution (p) or the information of the version of program after execution (p′).



Need Prior knowledge of Execution: this column determines if the technique needs the information about test cases prior to execution or not.



Level/Group: level/group presents to which category technique belongs. The “statement”, “function”, and “system” terms mean that the technique belongs to statement level category, function level or system level, respectively.



Use outset info/result info (static or dynamic testing): the outset information means using the information of the beginning of the testing and the result information means using the information of testing result.

Seyed Morteza Hosseini – Master Thesis – 2015

33

Time

Using the information of P/P′

Need Prior knowledge of Execution

Level/Group

Use outset info/result info(static or dynamic testing)

-

-

-

-

-

-

-



-

-

-

-



-

-

O(n log n)

-

-

statement

-

-

-

statement

-

-

-

statement

-

-

-

statement

-

O(mn+mlogm )

-

-

statement

-

-

-

-

-

statement

-

-

-

-

-

-

-

-

fn-total

-

O(mn+mlogm )

-

-

-

-

fn-addtl

M=number of test cases N=number of functions

-

-

-

-

fep-total

-

O(mn+mlogm )

-

-

function

-

fep-addtl

-

-

-

×

function

-



System

both

Technique Name

Mnemonic

Random Prioritization

Random

Optimal

optimal

APFD

-

Total Branch Coverage

branch-total

Additional branch coverage

branch-addtl

-

st-total

Prioritize based on total number of statement coverage

st-addtl

-

FEP-total

-

FEP-addtl

Total Statement Coverage Additional Statement Coverage Total Fault Exposing Potential Additional Total Fault Exposing Potential Prioritization Using Relevant Slice Total Function Coverage Prioritization Additional Function Coverage Prioritization Total FEP Prioritization Additional Total FEP Prioritization Fault Aware Test Case Prioritization (FATCP)

-

Description

Prioritize test cases randomly Order test cases to optimize fault detection rate Prioritize based on branch coverage

O(

)

O(mn+mlogm )

O(

n)

O(

n)

Prioritize based on coverage info p and historical info Table 6. Test Case Prioritization Techniques

Seyed Morteza Hosseini – Master Thesis – 2015

34

In Table 7, you can find the group to which a given technique belongs. We refer to techniques with their mnemonic names in Table 7. Statement Level

Function Level

St-total

Fn-total

St-addtl

Fn-addtl

St-fep-total St-fep-addtl Branch-total Branch-addtl Fep Fep-add FATCP

Fn-fep-total Fn-fep-addtl fi-total fi-addtl fi-fep-total fi-fep-addtl diff-total diff-addtl diff-fep-total diff-fep-addttl Fun-add Fep-total Fep-add Fi-total

System Level Fault-exposingpotential-addtl Fault-exposingpotential-total Relevant-slice

Model Based System model

Table 7. Statement Level & Function Level Prioritization Techniques

In Table 8, it is observed that each technique uses either the outset (the time at which a test case is supposed to begin) information of the program or the result information (the information of executed test case). Mnemonic St-total Fi-total Br-total St-fep-total Fn-total Fn-fep-total Fep-total Fep St-addtl Fep-add Optimal APFD St-fep-addtl Br-addtl Br-add Fn-addtl Fn-fep-addtl Fun-add

Using Outset Information         × × × × × × × × × ×

Using Result Information × × × × × × × ×          

Table 8.Outset and Result Techniques

Seyed Morteza Hosseini – Master Thesis – 2015

35

4.3

Test Execution Effort Estimation Methods

Test effort estimation allows managers, developers, testers, and test managers to assign resources, schedule the project and test activities, performs risk management, and monitor test and project activity. It also enables visibility into the most important parts of the SUT while allowing prioritization of additional test effort in order to achieve high quality software [19][21]. Test effort estimation is increasingly important for the following reasons:   

Effort estimation helps to estimate the budget of the project It enables the manager to assign test team members as well as test tools [16]. Without estimating the test effort, managers are not able to do the mentioned activity by Badri et al and Kushawaha et al in [19] [16] [18] such as allocating test resources, planning and monitoring the test activity, planning to write test cases, drawing out the test result, and determining the critical parts of the software which require more testing effort.

As mentioned before, the methods for estimating the effort can be considered for two parts: test case creation and test case execution. In this thesis work, the focus is on the second part. Test creation effort includes the effort needed for test strategy creation and testing plan [29]. In webpage http://santoqa.blogspot.com, the estimation methods are considered in three groups as the most common methods including percentage, analogy, and expert [35]. Different techniques have been used by different test execution methods. The common techniques are as follow: 



  

Use-case based techniques [13] [14] [17] estimate in the early stage and nullity depending on the code. Using software requirements and different technical and environmental factors are the main features of these kinds of techniques [18]. Code-based techniques [22] [13] [20] that have the following drawbacks: o Code dependency o Inability to predict in early stage of development o Inability to estimate the effort in early stages as we have developed the system [26], the estimation process should be done before creating test cases [18]. Complexity value-based techniques [18]. Requirement-based techniques Model-based techniques [23] that are based on data-flow, control flow, or UML model [19].

Badri et al [19] have utilized use cases in order to estimate test effort in suite size aspect. The authors have considered four metrics to describe the complexity of use cases and three metrics to illustrate different aspects of test suites. They have used K-means clustering method as a clustering technique to categorize use cases in three groups of simple, medium, and complex. Badri et al have done ranking of test cases, which are involved in each use case to simple, medium, and complex. Some Badri et al metrics [19] are in code-level such as the number of lines in test code (TLOC). A

Seyed Morteza Hosseini – Master Thesis – 2015

36

tool to estimate test execution effort is developed by E. Aranha et al in [22] based on the size and execution complexity. Actually, they [22] automate the approach that has been illustrated by Aranha et al [13]. One of the other code-based methods that have been utilized by Singh et al [24] uses artificial neural network (ANN) in order to estimate the test effort. The metric that has been used is the number of lines of codes that were added or changed in the class level. E Aranha et al in [27] have proposed another approach that is the same as the one in [13] using the following risk factors:

  

Team experience Test environment Tested product

The first approach that has been proposed by Aranha et al in [13] is applicable for static test environment. The process of test effort estimation is based on the execution point (“a unit of measure defined for describing the size and execution complexity of test cases” [13]) and conversion factor (“The conversion factor represents the relation between test execution effort and execution points”. “The conversion factor is given in seconds per execution point, indicating the number of seconds required to execute each execution point of a test case” [13].). The second approach they have proposed in [27] is applicable in dynamic test environment and uses a risk factor, which can be team experience, test environment conditions, tested product, etc. Another type of methods test effort estimation that have been proposed in Certified Tester Foundation Level Syllabus [31] is in two following types: 1. Metric-based which estimates the effort by using metrics of former or similar projects. 1. Expert-based that uses the owner or expert’s estimation. 2. The third method divides the system to the smaller parts and estimates the effort of divided parts. Then calculates the total effort of the project by summing the effort of smaller parts. This method is called bottom-up-estimate [30]. Dharmender and Kushwaha [18] have mentioned the commonly used measures and their evolution from 1980 to 2010. Kushawa has presented various methods. Based on their research, the measure, which has been employed in 1980, was test date based estimation. In 1990 model based estimation was the commonly used method. The roadmap indicates that the method’s trend goes to the high-level architecture of the SUT, as the common methods in 2009 and 2010 were use case based estimation and requirements based estimation methods, respectively. Test case effort estimation usually comprises the following factors estimation [12]:    

Size of the product. Effort in person-hours/months. Schedule in calendar month. Cost in currency.

Some metrics for test case effort estimation that have been mentioned by Saadatmand [15] include time, person month, and financial cost. The disadvantages of the methods, which are based on the LOC, can be the lack of any standard on how to count the number of lines as well as no possibility to count LOC prior to the development.

Seyed Morteza Hosseini – Master Thesis – 2015

37

4.3.1

Use Case Points (UCP)

Use case point method [12] makes use of the use cases in order to estimate test case effort. The estimation process is performed in the following steps: 1. Obtaining Unadjusted Actor Weights (UAW) based on the number of actors in the system. Actors are grouped as simple type such as end-user, which interacts with the system through GUI, average type like data stores that interacts via protocol, and complex type like other programs, which interacts with the system through API. Actor weight of simple actor is 1, average actor is 2, and complex actor is 3 [12]. 2. Obtaining Unadjusted Use Case Weights (UUCW) based on the number of use cases in the system. Similar to actors, use cases are also in three types of simple, average, and complex. UUCW assigns to use cases either based on number of transactions or scenarios as have been shown in Table 9: Use Case Type Simple Average Complex

Number of Transaction 7

Factor 1 2 3

Table 9. UUCW [12]

3. Calculating Unadjusted UCP (UUCP) based on the following equation: UUCP=UAW+UUCW Equation 6. UUCP [12]

4. Calculating technical and environmental factors: the technical and environmental factors are listed in Table 10: Factor T1 T2 T3 T4 T5 T6 T7 T8 T9

Description Test tools Documented inputs Development environment Test environment Test-ware reuse Distributed system Performance objectives Security features Complex interfacing

Assigned Value 5 5 2 3 3 4 2 4 5

Table 10. Technical and Environmental Factors [12]

To calculate Total Environment Factors (TEF), we need for each factor to multiply assigned weigh to assigned value.

Seyed Morteza Hosseini – Master Thesis – 2015

38

=

(



∗



ℎ)

Equation 7. TEF Formula

5. Adjusted UCP (AUCP) computation: AUCP=UUCP*[0.65+ (0.01*TEF)] Equation 8. AUCP [12]

Two values 0.65 and 0.01 are constant in the above formula. 6. Calculating the final effort that is obtained by multiplying AUCP to conversion factor. Conversion factor is person-hours effort. Estimation in high level is a disadvantage of this method as UCP-based method because it works in testing activities level including test planning, test design, and test reporting as a whole. It might not be able to estimate each single test unit level such as test case or test suit [14].

Some other methods that according to [12] are subsumed within the category of conventional methods, which would follow in the next sections.

4.3.2 Conventional Methods

4.3.2.1 Ad-hoc Method In this method test effort estimation is not based on an accurate timeframe. The estimation of test cases is accomplished based on either pre-decided timeline by managers or budget constraint. The adhoc method [12] is usually utilized by inexperienced organizations. 4.3.2.2 Percentage of Development Time As the name suggests, percentage of development time [12] method estimates the test efforts based on the development time, which are obtained through line of code (LOC) or function points (FP) techniques. Although this method does not meet the scientific or technical principles, it serves in most cases.

4.3.2.3 Function Points Estimation The efforts in Function Points Method [12] are basically measured on the basis of person-hours and transaction factors that are mostly extracted from previous projects. This method needs the required details and is not practical in that it does not document the use cases. As Xiaochun, et al [14]

Seyed Morteza Hosseini – Master Thesis – 2015

39

put it this method suffers from both constructive and adaptive problems. Another demerit of the method is that it requires a high cost to be implemented for the projects that are conducted in a big scale.

4.3.3

Estimation Model for Test Execution Effort based on the Test Specifications

Estimation model for test execution effort based on the test specifications method [13] uses test specifications such as size and execution complexity, which can be achieved through using control natural language (CNL). Actually, CNL is in a way the same as natural language but the only difference stems from some restriction in terms of the grammar and lexicon (these constraint can vary depending on the target domain) as the sentences written in CNL follow a given standard format. As a result, there is no possibility to describe an event and/or an object in different ways. The rule that should be obeyed in this technique is that each sentence contains one verb and zero or more argument in a sense in which verb indicates the action and argument implies the additional information about action. As an example, consider the following sentence: start the message center. In this example, the verb start is an action of starting an application and the argument the message center demonstrates the application that should be carried out. A wide view of how this method works is shown in Figure 8.

Figure 8. The Es ma ng Process of Execu on Effort [13]

The entire process would be described as follow: 1. Analyzing and assigning an execution point (size and execution complexity) to each test case. 1. Summing up all the execution points obtained through conducting step 1. 2. Estimating effort for executing all test cases in person-hours. Generally, the size of a test case intimates the steps needed to execute it and execution complexity, on the other hand, implies the relation between tester and the product that is to be tested. In this regard, the TC1>TC2 means TC1 has bigger size and execution complexity than TC2. In addition, the similar relation (≈) means TC1 has smaller size and execution complexity than TC2.

Seyed Morteza Hosseini – Master Thesis – 2015

40

Assumption: TC1>TC2 only if TC1 is not similar to TC2 The equation 9 demonstrates the relationships between a and b that are execution points of test case t1 and t2; the first relation reveals the condition in which a is bigger than b, and the second relation illustrates the condition in which a is similar to b.

Equation 9. Estimation Relation [13]

The following is the whole process by which the size and execution complexity of a test case is computed: The first step is to analyze each test step based on the system characteristics (C1, C2, …,Cn), which are functional and non-functional requirements. In the next step, with respect to the impact each characteristic has on the size and execution complexity of a test, we will assign an impact rate to them based on three scales of low, average, and high. Then, this impact rate would be considered as the criterion by which we can assign an execution point to each characteristic. Following this, the execution point of a test statement would be obtained by summing up the execution points of characteristics. Finally, summation of execution points of test statements is done to obtain test case execution point. Some disadvantages of this method are listed below [14]: 1. Test cases should be ready prior to estimation. 2. As this method works on test step level, in case of having many test cases, the cost of estimation is too high. 3. The relation between execution points and execution effort needs to be proven. 4. In order to use historical data, we need to execute the test cases prior to estimation.

4.3.4

Early Test Effort Estimation by Using Use Case

The main objective of this technique [14] is to use Use Cases in order to obtain the number of test cases, and then estimating effort based on three parameters; namely, test case number, execution complexity, and tester rank. Tester ranking will be assigned based on testing experience and knowledge in product to be tested. Test execution complexity is measured through equation 10:





=

[

( )∗

Seyed Morteza Hosseini – Master Thesis – 2015

41

ℎ ( )]

Equation 10. Test Execution Complexity [14]

Where scale(fi) and weight(fi) mean the ordinal scale value and the weight value of factor fi respectively. A high-level view of the whole process is shown in Figure 9.

Figure 9. Early Test Effort Es ma on Approach [14]

4.3.5

Cognitive Information Complexity Measure

Cognitive Information Complexity Measure (CICM) Method [16] uses the cognitive aspects of the practitioners. This approach is classified in the code-based category as it uses LOC and the number of operands and operators in each line.

Seyed Morteza Hosseini – Master Thesis – 2015

42

4.3.6

New, Search, Modify/ Management Information System Functionality

NSM/MIS method [17] uses the information of use cases in order to estimate test case effort. Actually, the basis of this method is the Use Case Point Method (UCPM), which is used in [12]. The use case information that is used in this method are actors, number of classes/transactions, technical factors, and environmental factors. This information delineates the values and if multiplied by a conversion factor, the estimated effort in time would be achieved. The difference between NSM/MIS and UCPM is that the two types of use cases, i.e. simple and complex, are considered in this method. As this method utilizes use case, we can put it in the group of model-based approaches. Disadvantage of this method is lack of the use case information.

4.3.7

Simplified Method based on the NSM/MIS method

Actually, this method is a simplified version of NSM/MIS [17] approach which considers just two types of actors that are human, and interface actors.

4.3.8

Software Requirement Specification (SRS)

SRS method [23] makes use of the software requirement specification. As this method uses required documents, it can estimates effort in the early stage of development phase. It uses improved requirement based complexity (IRBC), the input complexity (IC), output complexity (OC), and storage complexity (SC). It has grouped data storage, input, and output to different types and has assigned weight to them [23]. Then the calculation of IC, OC, and SC has been done. The input output complexity is obtained by combination of IC, OC, and SC.

4.3.9

Accumulated Efficiency Method

The main purpose of the Accumulated Efficiency Method [20] is to estimate test case execution effort by focusing on team efficiency. It puts into use the LOC and use case. Accumulated Efficiency Method is applied to manual execution and does not consider automatic execution since small test teams usually utilize manual testing instead of automatic test execution due to the cost restriction. Some advantages of this method would follow:  Objective judgment has no effect on it.  It does not need any kind of controlled language.  There is no need for data information. The following variables have been used by this method to obtain accumulated efficiency metric: test case execution time and test case steps. The accumulated efficiency is calculated based on equation 11:

Seyed Morteza Hosseini – Master Thesis – 2015

43

Equation 11. Accumulated Efficiency [20]

Where di is the ith day, which is the sum of test steps, executed by day i, divided by total execution time. A disadvantage of this method is that it cannot estimate the effort prior to execution, whereas this is possible in our method (will be presented in Section 5) in the early stage. The other disadvantage is that it estimates the team effort, not each test case individually.

4.3.10

Case-Based Reasoning (CBR) Method

The CBR [21] approach uses data mining method to reduce the cost of testing and clustering technique to classify the data. It uses case base reasoning (CBR) which is a technique used for figuring out new problems by using previous situation and reusing that information. As this method uses the information from previous similar situations, one of its disadvantages is that, in case of no existence of these situations, we do not have any previous information to reuse in new problem. In addition, even in case of having similar situation it is not clear that it can adapt the information for new problem. In comparison with our approach (will be presented in Section 5) that is applicable for any size of project, this method is appropriate for small projects.

4.3.11

Cuckoo Search Method

Cuckoo search method [28] is similar to Nageswaran approach in [12]. The total number of parameters used by Nageswaran in [12] is 16, as well as 2 other parameters; namely, expertise of development team and expertise of test team. The main difference between Nageswaran approach [12] and cuckoo search method is that Nageswaran approach employs a fixed value weight to assign to the parameters while this method makes use of a range for each parameter. This range can be fixed or dynamic based on the target domain. With respect to the two new parameters (expertise of development team and expertise of test team), it divides them to three experienced groups including mixture of experienced and non-experienced as well as non-experienced, then assigns them a weight value but unlike its claim, it has used a fixed value for these two new parameters rather than a range. Another drawback of this method is that it needs at least a project as a historical data for which actual effort is known. Accordingly, in comparison to Nageswaran method [12], its advantage is that the weights assigned to parameters are not fixed and can be dynamic in a given range.

Seyed Morteza Hosseini – Master Thesis – 2015

44

5. Our Effort Estimation Model In this Section, we present our new approach for estimating test case execution effort. Most of the existing methods that have been studied in the Related Work Section either are not in the system level or have not utilized architecture of the SUT to estimate in the early stage of the SDLC. Aranha and Borba have employed system characteristics exercised by the test step in each test case, which can estimate the execution effort in the early stages [13], but in case of having many test cases their model would be time consuming. Verifying system characteristics in each single test step for many numbers of test cases can be a rough task. We adopted our model for using system modules exercised by each test case instead of using test step. Our model works based on the test case. It means that the input of our model is test case. We have considered a significant degree/factor based on the importance of subsystems (modules, components, parts) in the system that are exercised by each test case. Consequently, as it is presented in Figure 10, test case serves as an input to the model. In the first phase, we determine the modules (components, parts) of the system that each test case would cover, demonstrated by module1, module2… modulen in step In the next step we analyze each module to find out the metrics that have been covered by that module. Then, an importance level (low, average, or high) will be assigned to those metrics in step Following this, in step the importance degree’s qualitative value will be mapped to a quantitative value. Finally, the importance degree of the test case that is the summation of all importance degrees related to all modules covered by the test case would be calculated. The guideline for importance degree of each part is illustrated in Table 11. The metrics value mapping process for qualitative values of low, average, and high to quantitative values of 3, 5, and 8 is based on Aranha and Borba method [13]. “In general, values 3, 5 and 8 are used to weight the levels Low, Average, and High, respectively” [13].

Seyed Morteza Hosseini – Master Thesis – 2015

45

System Modules Exercised by the Test Case ...

Module1

Metric1

L

3

A

H

...

...

Metric n

...

L

A

Module n

...

H

...

5

...

Importance Degree of Test Case

50

Metric1

L

A

H

5

...

...

...

Metric n

L

A

H

8

Figure 10. Our Approach View

Importance degree would be determined by the factors that have been illustrated in the metric description section (the guideline for the importance level of modules can be different depending on the target domain). The way in which you can use Table 11 is that for each metric there are three qualitative values (low, average, and high). For data dependency metric, this means that if the number of data dependency is greater than 0 and less or equal to three, then its qualitative value will be low. If the number of data dependency is greater than 3 and less or equal to 5, then its qualitative value will be average. And finally, if the number of data dependency is greater than 5, its qualitative value will be high. Then for mapping the qualitative values to quantitative values, you can use the values in the second row of each metric that will map the qualitative values of low, average, and high to quantitative values 3, 5, and 8, respectively. The same scenario will go for security dependency metric, number of commuting, finish-to-start dependency, finish-to-finish dependency, external dependency, logical dependency, and the number of involved modules. For customer delivery priority, development time, and complexity metrics, the lower degree means the lower value of relevant metric. For instance, the complexity value of 5 has more complexity than complexity degree of 4. The degrees are ranked on a range of 1 to 5 for lower to higher degrees, respectively. That is, 1 means the lower degree and 5 means the higher degree.

Seyed Morteza Hosseini – Master Thesis – 2015

46

ID

1

2

3

4

5

Metric

Low

Hardware Consumption

12

13

14 15 16

3

5

8

0