Test Automation at ABB Automation Technologies AB - Implementation and Evaluation

Master’s Thesis Elin Alwall Ardina Tkalcevic Supervisors Thomas Thelin, LTH Stefan Calling, ABB

Department of Communication Systems

CODEN:LUTEDX(TETS-5511)/1-62/(2004) & local 13

Abstract Software testing is an essential part of software development. Defects are more expensive to find and correct later in the development process, it is therefore important to have a good testing procedure to minimise the costs. Automation of manual tests can be an excellent way to save time and money, make the test process more effective and relieve employees from boring, time-consuming and monotonous tasks. Whether automation becomes successful or not depends primarily on how much maintenance the automated test requires. The amount of maintenance depends on the stability of the product tested, how much the product changes between test sessions, how easy the test is to update and much more, including factors that are unique for every case of automation. Hence, for every test automation proposed there is a need to evaluate if the test is appropriate for automation. The objective of this master thesis is to automate a daily regression test at ABB Automation Technologies AB, as well as presenting the test result in an understandable and useful manner. It also includes an investigation regarding if time and money are lost or gained by performing the test automatically. Hence, this thesis will be the foundation of ABB's decision whether to continue performing the test automatically or not. A program, which was going to execute the test automatically, was implemented, mostly by using the automation tool WinRunner. The test result is presented in two Excel-sheets, one for a deeper understanding and one for a more comprehensive approach. To acquire deeper knowledge of which factors to consider in determining the costs of using the automated test, a questionnaire was handed out to employees at ABB. By using the answers of this questionnaire, together with a literature study and the experience achieved during the implementation, a cost estimation was performed. Many factors that affect the costs are hard to estimate since they vary depending on, amongst others, the pace of the development cycle and the stability and development of the product tested. The cost estimation is therefore based on a couple of different scenarios. For the automation to be profitable, it is calculated that maintenance must take less than 2.5 hours per day. The scenarios show that this is most often not the case; the maintenance often takes more than 2.5 hours per day. The reasons for the large amount of maintenance are many but the instability of the product is the most significant. The conclusion is that in most cases the automation is not profitable for ABB.

Test Automation at ABB Automation Technologies AB

2

- Implementation and Evaluation

Acknowledgements We would like to thank our supervisors, Thomas Thelin at the Department of Communication Systems at Lund Institute of Technology and Stefan Calling at ABB Automation Technologies AB, for their help and support throughout the work with this master thesis. We have been given a lot of support and help from several employees at ABB, and would like to give special thanks to Olle Persson, Dag Ehnbom, Jörgen Jönsson, Magnus Björkman, Gunnar Hillbom, Sven-Åke Nord and Karolina Gajski. Thanks also to Clara Alwall who read this report in proof. Elin Alwall and Ardina Tkalcevic

3

Test Automation at ABB Automation Technologies AB

4

- Implementation and Evaluation

Table of Contents 1

INTRODUCTION.......................................................................................................7 1.1 1.2 1.3 1.4 1.5 1.6

2

BACKGROUND ............................................................................................................................ 7 OBJECTIVE .................................................................................................................................. 8 OUTLINE ..................................................................................................................................... 8 WORK PROCESS OF THE MASTER THESIS............................................................................... 9 DELIMITATION ........................................................................................................................... 9 TERMINOLOGY ......................................................................................................................... 10

SOFTWARE THEORY.............................................................................................. 13 2.1 THE WATERFALL MODEL ....................................................................................................... 13 2.2 THE V-MODEL .......................................................................................................................... 14 2.3 DEVELOPMENT PROCESS AT ABB ........................................................................................ 15 2.4 SOFTWARE MAINTENANCE .................................................................................................... 17 2.4.1 Maintenance Cost .................................................................................................................... 17 2.4.2 Measuring Maintainability ...................................................................................................... 18

3

SOFTWARE TESTING THEORY ........................................................................... 21 3.1 TESTING DEFINITION ............................................................................................................. 21 3.2 VERIFICATION AND VALIDATION ......................................................................................... 22 3.3 GOOD TEST PRACTICES .......................................................................................................... 22 3.4 TEST LEVELS ............................................................................................................................ 23 3.4.1 Component Testing .................................................................................................................. 23 3.4.2 Integration Testing ................................................................................................................... 23 3.4.3 User Testing ............................................................................................................................ 24 3.4.4 Regression Testing.................................................................................................................... 24 3.5 TEST TECHNIQUES .................................................................................................................. 25 3.5.1 Dynamic and Static ................................................................................................................. 25 3.5.2 White-box and Black-box ....................................................................................................... 26 3.5.3 Formal and Informal ............................................................................................................... 26 3.5.4 Program Interface Test ............................................................................................................. 26 3.6 TEST PLANNING....................................................................................................................... 26 3.7 TEST DESIGN............................................................................................................................ 27 3.8 TEST MAINTENANCE............................................................................................................... 27 3.9 TEST ROLES .............................................................................................................................. 28 3.9.1 Tester ...................................................................................................................................... 29 3.9.2 Team Leader ........................................................................................................................... 29 3.9.3 Test Manager .......................................................................................................................... 29 3.9.4 Test Automation Roles ............................................................................................................ 29

4

AUTOMATIC TESTING .......................................................................................... 31 4.1 4.2 4.3 4.4 4.5 4.6 4.7

5 5

ADVANTAGES AND DISADVANTAGES .................................................................................. 31 WHEN TO AUTOMATE............................................................................................................. 32 WHAT TO AUTOMATE ............................................................................................................. 33 GUI-TESTS ................................................................................................................................ 33 RECORD AND PLAYBACK ........................................................................................................ 34 REQUIREMENTS FOR AUTOMATED GUI TESTS .................................................................. 35 COSTS FOR MANUAL AND AUTOMATIC TESTS .................................................................... 35

EVALUATION METHODS...................................................................................... 37

Test Automation at ABB Automation Technologies AB

5.1 5.2 6

QUALITATIVE AND QUANTITATIVE METHODS .................................................................. 37 INTERVIEWS .............................................................................................................................. 37

AUTOMATION OF THE DAILY TEST ................................................................. 39 6.1 THE DAILY TEST...................................................................................................................... 39 6.2 IMPLEMENTATION ................................................................................................................... 40 6.2.1 Solution................................................................................................................................... 40 6.2.2 The Ghost ............................................................................................................................... 40 6.2.3 The WinRunner Scripts........................................................................................................... 41 6.2.4 Cold and Warm Restart .......................................................................................................... 42 6.2.5 Conclusion ............................................................................................................................... 42

7

EVALUATION........................................................................................................... 45 7.1 ADVANTAGES AND DISADVANTAGES .................................................................................. 45 7.2 WHEN TO AUTOMATE............................................................................................................. 46 7.3 INTERVIEW AND QUESTIONNAIRE ....................................................................................... 48 7.3.1 Summary of the Questionnaire ................................................................................................. 48 7.3.2 Conclusion ............................................................................................................................... 49 7.4 TOTAL COST ............................................................................................................................. 51 7.4.1 Cost Estimation for Manual Performance ................................................................................ 52 7.4.2 Cost Estimation for Automatic Performance ............................................................................ 52 7.4.3 Conclusion ............................................................................................................................... 55

8

CONCLUSIONS ........................................................................................................ 59 8.1 8.2 8.3

9

SUMMARY .................................................................................................................................. 59 CONCLUSION ............................................................................................................................ 59 FUTURE WORK .......................................................................................................................... 60

REFERENCES .......................................................................................................... 61

APPENDIX A – HARDWARE AND SOFTWARE AT ABB ........................................... 63 CONTROLLER ......................................................................................................................................... 63 CONTROL BUILDER ............................................................................................................................... 63 PPA ......................................................................................................................................................... 63 OPENIF................................................................................................................................................... 64 BATCH BUILDER .................................................................................................................................... 64 VISUAL STUDIO ...................................................................................................................................... 64 OPC-SERVER AND OPC-CLIENT ......................................................................................................... 64 APPENDIX B – TEST TOOL EVALUATION................................................................ 65 APPENDIX C - WINRUNNER........................................................................................ 67

CREATING TESTS ............................................................................................................................... 67 RUNNING TESTS .................................................................................................................................... 68 MAINTAINING TESTS ............................................................................................................................ 68 APPENDIX D – THE RESULT SOLUTION.................................................................. 71 APPENDIX E – QUESTIONS AND ANSWERS ............................................................ 73 QUESTIONS ............................................................................................................................................. 73 ANSWERS................................................................................................................................................. 75

6

- Implementation and Evaluation

1 Introduction

Testing is a very important part of software development, but it can also be an expensive and non-stimulating task if it is performed manually. Test automation is therefore very attractive since it can minimise both the costs and the boring tasks as the tests can be executed automatically at night. It was with this in mind that ABB Automation Technologies AB decided to make the automation of a daily regression test into a master thesis.

1.1

Background

ABB Automation Technologies AB in Malmö delivers solutions for control, motion, protection, and plant integration in the full range of process and utility industries. Controllers (CPUs), used in industries to control different processes, are built and controller software is developed. The software is the Control Builder with which controllers can be programmed and controlled, and firmware, which is the operating system for the controllers. Protocols are also made when e.g. new input and output elements, I/Os, are built. Other important hardware and software tools are built in Västerås. This master thesis is a part of a project called Titan. Titan is a test and integration project and its task is to perform program interface tests (PIT). Different developers implement different parts of the Control Builder. These parts are integrated and tested to make sure they work together. Most of the project members have roles as testers. They write test specifications for the controller software and perform the testing. Their tasks also include validation, i.e. a check that the defects, which have been corrected by the developers, really are corrected. If defects are detected they are sent back to the developers for correction. Some members of Titan are part of the build team. They build and compile the product, by using the Batch Builder, and decide when to release a new version to be tested by the testers. For a new product version to be released for testing it is required that a certain part of the product is correct, among other things the compilation has to succeed. A new product version is released approximately once a day. Accordingly, the developers continuously develop software, and the Titan project is responsible for integrating their work to check if the new features work with the rest of the product. The build team does the integration, as well as the release of a test version of the product. The testers then test the product. The Control Builder is released to customers once or twice per year. One of the tests performed in the Titan project is the Daily Test. The Daily Test is performed manually and tests the basic functionality in the Control Builder and the PPA, which are two software programs used at ABB. This is a time consuming and non-stimulating task done during daytime by one employee. The test is performed approximately once every day, which is considered as often. Overall the test looks the same; changes are made occasionally. Hence, it seems there are great conditions for automation. This master thesis includes an automation of the Daily Test, and an evaluation that investigates if the automation is profitable for ABB.

7

Test Automation at ABB Automation Technologies AB

More information about the Batch Builder, Control Builder and the PPA can be found in Appendix A. The Daily Test is described in section 6.1.

1.2

Objective

The objective with this master thesis is to perform the Daily Test automatically at night and to have the test results ready when the developers come to work in the morning. Since automated testing should be faster, the test could be performed more often and during daytime too. The purpose is to make ABB reach higher efficiency and quality, and to decrease the costs since no one has to be paid to perform the test. If automation is profitable, it might be possible for ABB to extend the test to reach a higher coverage in the future. The task is to automate the Daily Test, to present the results in an understandable manner, and to make an evaluation to decide if ABB profits from performing the tests automatically.

1.3

Outline

The Introduction describes the tasks of the Titan project and the objective of this thesis. The work process is described, and delimitation as well as terminology is presented. The chapter Software Theory includes information about important software development theories frequently used in software companies. Software maintenance is discussed since it is an important part of software development. A thorough description of the different areas of software testing, relevant for this thesis, is given in the third chapter called Software Testing Theory. Automatic Testing describes among other things the purpose of automatic testing, when it should be used as well as the advantages and disadvantages. The costs for manual testing, respectively automatic testing are compared. The chapter Evaluation Methods includes some of the different methods that are used in the master thesis to collect information. In Automation of the Daily Test the entire work process of this thesis is described. In the chapter Evaluation a discussion is held concerning whether the automation of the Daily Test should be used or if the manual testing is the preferable way to perform the test. Based on the evaluation, conclusions are drawn. The last chapter Conclusions contains some comprehensive conclusions and some suggestions of what has to be supplemented to the work performed during this master thesis if the automation will be used in the future.

8

- Implementation and Evaluation

1.4

Work Process of the Master Thesis

The initial study included reading about software theory, understanding the Daily Test and investigating what different software tools that are available at ABB, how they work and how they are used. At first, the objective was to investigate a couple of different automation tools and then decide which one was best suited for the automation of the Daily Test, and which tool and method that was most profitable for ABB. The investigation was soon discarded because of the realisation of how much time it would take. Evaluations had already been made, both at ABB and in another master thesis [ADELSWÄRD 98], comparing the different automation tools in the market establishing that WinRunner was one of the best tools. Therefore the decision was made to use WinRunner, which was also already in use at ABB. The implementation with the chosen tool included not only automation of the Daily Test, but also failure tracing and presentation of the test result. For the result to be significant, it has to encircle and describe the failure when the test fails, and state that everything is correct when the test passes. Ultimately, the result should be presented in a way that is comprehensible for the developers as well as for the managers. Another important part was the evaluation. The main question to be answered was: “is it profitable for ABB to automate the Daily Test?”. The cost when performing the Daily Test manually had to be compared to the cost when the test is automated. To estimate the cost for the automated test it was necessary to inquire into which elements, specific for ABB, which affect the cost. To gather information about that a questionnaire was handed out to developers, testers and project managers. The questionnaire also provided knowledge of the general attitude at ABB of automatic testing. The main evaluation is also based on the experience acquired during the implementation. The stages mentioned in the sections above were not performed separately, but rather worked with in an iterative manner. The term cost used includes money, time and maintenance. The last stage made was suggestions on future improvements and changes.

1.5

Delimitation

One of the objectives from the beginning of this master thesis was to test a couple of different automation tools. Considering the amount of time this would take, it was realised that this was not possible. There had already been made evaluations, both at ABB and in another master thesis [ADELSWÄRD 98], comparing the different automation tools in the market establishing that WinRunner was one of the best tools. It was therefore decided that WinRunner, which also already was available at ABB, should be used. In the automatic approach, the installation of the PPA is still done manually, once every time a new PPA version is available. One reason for that is that it is unnecessary to perform the installation every time the test is executed, since the same version of the PPA is used for at least two weeks. Another reason is that the installation usually comes with different roll-ups and patches, which vary in such extent they cannot be predicted and automated successfully. The scope of the Daily Test is successively increased with new areas to test. This master thesis only comprises the Daily Test, as it was when the master thesis started. Since then a couple of 9

Test Automation at ABB Automation Technologies AB

things have been changed and added. These things have not been taken into account because they are not included in the scope of this master thesis. A new version of the Control Builder is released for testing approximately once every day, and a new PPA version is released approximately every second week. For this master thesis to be feasible it had to make use of an appropriately working version of the Control Builder and the PPA. Adapting the test to every new version would have made this master thesis an everlasting process, since every version differs from the previous one and this often results in necessary and time-demanding adaptations of the test. Different versions of the Control Builder and the PPA were used in the beginning of the work process. Two months before the end of the master thesis the thought of adapting to every new version was discarded, and a released version of the Control Builder and the corresponding PPA version were chosen and worked with for the rest of the time. In a part of the Daily Test an I/O is deleted in the Control Builder and then it is checked that the contact between the corresponding I/O on the controller and the test computer is lost. The Control Builder is used to program and control the controller. Manually, observing that a diode is turned off on the I/O corresponding to the deleted one, makes the check. Performing the check automatically would have required a rebuild of the controller. Lack of time resulted in this part being excluded in this master thesis.

1.6

Terminology

AUT: Application Under Test; the application being tested. Batch Builder: a software tool, which can start compilation and execution of different jobs. In this master thesis it is used to start the automated Daily Test automatically at night. Control Builder, CB: a software tool developed at ABB Automation Technologies AB. It is programmed to control controllers in industries. The tool is used together with the PPA. Data-driven test: an automated test that uses an input test data file separated from the test script. The data file can contain e.g. parameters that can be used as input to the test script. Ghost: a Ghost is a copy of the contents on a hard drive. GUI: Graphical User Interface. An interface that provides the user a way of interacting with the computer. GUI-map: the GUI-object container in WinRunner. GUI object: an object in an application identified by WinRunner during recording and added to the GUI-map. Examples: window, list, and button. OpenIF: the interface between the Control Builder and the PPA. Program Interface Test, PIT: a test where programs or modules are integrated to test the interface between them. Used in the Titan project.

10

- Implementation and Evaluation

Process Portal A, PPA: a software tool developed at ABB in Västerås. It is used to monitor processes in industries. Record/playback: a function in some software automation tools. The tools can be used to record the actions of a user. The recorded actions can be replayed automatically. Regression test: a test that tests all functional components in the tested application to ensure that they still work together. Test Case: the smallest unit of a test, which tests a single aspect or function of the application. Test Programme: the smallest executable test entity that contains one or more test cases. Test Suite: a collection of test programs. It tests an entire application or one of its subsystems. Titan: the test and integration project this Master Thesis was a part of. The Daily Test: the regression test, which is automated in this master thesis. It tests basic functionality in the Control Builder and the PPA. TSL: Test Scripting Language; the language generated in record mode by WinRunner. Validation: a process of determining whether the product satisfies the customer's needs. Verification: a process of confirming that the product meets the requirement specification. Voyager: a development program at ABB, which Titan is a part of. WinRunner: a record/playback tool. The activity of the user can be recorded and then replayed automatically.

11

Test Automation at ABB Automation Technologies AB

12

- Implementation and Evaluation

2 Software Theory

Software development can be performed in different ways. Some of them are described in theoretical development models. Following one of those models makes the development more structured and well arranged. One of the models is the Waterfall model.

2.1

The Waterfall Model

The Waterfall-model is the first explicit model for the software development process; it was derived from other engineering processes. It was named Waterfall model since its phases cascade from one to another. There are numerous of variations of this process model, but the principal stages, which are shown in figure 2.1, are [SOMMERVILLE 95]: • • • • •

13

Requirements definition. The service of the system, which constraints and goals are established by consultation with system users. The requirements are defined in a manner, which is understandable for all people involved. System and software design. The system design process partitions the requirements to overall system architecture. Software design involves representing the functions of the software system in a form that may be transformed into executable programs. Implementing and unit testing. The software design is realised as a set of programs or program units. Unit testing involves verifying that each unit fulfils its specification. Integration and system testing. The program units are integrated and tested as a complete system to ensure that the software requirements have been satisfied. Finally, the software system is delivered to the customer. Operation and maintenance. This is usually the longest phase. The system is installed and put into practical use. Maintenance involves correcting defects that were not discovered in earlier stages, improving the implementation of system units and enhancing the service of the system as new requirements are discovered.

Test Automation at ABB Automation Technologies AB

Requirements definition

System and software design

Implementation and unit testing

Integration and system testing

Operation and maintenance Figure 2.1. The Waterfall model

In practice the phases overlap and feed information to each other. The process is not a simple linear model; it involves a sequence of iterations of the development activities. During the final phase, program and design defects emerge and the need for new functionality is identified. This means that it might be necessary to repeat some or all of the previous phases. Because of the frequent iterations it becomes difficult to identify definite management checkpoints for planning and reporting. Therefore it is normal to freeze parts of the development after a small number of iterations and then continue with the later development stages. Problems are instead left for later resolution, ignored or programmed around. This might lead to the system not meeting the customer's requirements, since there might be problems still lingering in the system at the point of release. Even though the waterfall model is rather inflexible, partitioning the project into distinct stages, it reflects engineering practice. For large software system development, it is therefore likely that the description of the steps in the software process will be based on this approach even though another model is used [SOMMERVILLE 95].

2.2

The V-model

The V-model is based on the Waterfall-model. In system development, it is a graphical representation used to relate test levels to the lifecycle model. On the left side of the model, seen in figure 2.2, are the phases in which the system is being developed - from wish to solution, and on the right side the appropriate test activities. The dotted line indicates the division of responsibilities between consumer, user and system manager (above the line), and software developer, supplier and programmer (below the line). In practice, however, software developers will be involved in the requirement analysis, in preparing the specifications and giving support 14

- Implementation and Evaluation

during the acceptance test, and users and system managers will be called upon during development activities [POL 02].

Operation and control

Wish

Functional design

Expectations

Acceptance test

Technical design

System realisation

System test

Unit and Integration test

Figure 2.2. TheV-model

Only the phases of performing the various tests appear in the V-model. This represents no more than a small part of the total effort of that test level. Other activities that are not shown in the picture are planning, preparation and specification. The arrows indicate the path of the initial documentation to the execution of the test, i.e. the planning and preparation activities. The left side of the V-model should force testers to think through the specifications, design and code, and thus lead to the detection of defects and potential problems at an early stage where they cost less to resolve [POL 02].

2.3

Development Process at ABB

Voyager is a development process at ABB. It includes preliminary study, development, test, education, documentation, pricing etc. and is partitioned into five development projects, two test projects and several market activity projects. Titan is one of the test projects. The software development model used is the V-model, which is an iterative version of the Waterfall model. These models are described in the previous sections of this chapter. This section describes how the V-model is used in the Voyager process. This V-model can be seen in figure 2.3 below. •

15

Requirements Definition. The requirements are written. The Project Managment, PM, develops the System Requirement Specification, SRS, after discussions with the customer. The SRS is the basic specification for the complete system that will be developed, i.e. for all products/projects in that system. The SRS is treated and a Product Requirement Specification, PRS, is developed for every product/project with more detailed specifications. Requirements are formulated from the PRS and an implementation proposal, IP, is made. The IP is used as a base for the estimations. The estimation issues the terms time,

Test Automation at ABB Automation Technologies AB

• • • • • • • •

resources and scope, and results in a plan. The steps requirements, IP, estimation and plan are iterated until there is a balance between time, resources and scope. When the requirements are agreed upon a requirement specification is written. Requirements Analysis. The requirement definition is used to formulate a Function Description. These descriptions contain UML-diagrams. Analysis & Design and Detailed Design. From the function description a Design Description is developed. The design description describes how the functions should be implemented technically. Implementation. With the design description as a basis, the functions are implemented. Besides the implementation a review is made on the code with a software tool to verify that the code conventions are followed. Module Test (MT). Testers perform tests to check that the design is correct. The modules, which are dependent code components put together, are tested individually. Function Test (FT). Each use-case in the function description is tested individually. Product Integration Test (PIT). The interface between the modules is tested to check that the modules work together. The use-cases are also tested together. The PIT-stage is performed by the Titan project. Product Type Test (PTT). The PRS is used to check that the product requirements are fulfilled in the design and implementation. System Type Test (STT). The SRS is used to check that the system requirements are fulfilled in the design and implementation. STT

Requirements Definition

Product Type Test (PTT)

Requirement Specification

Requirements Analysis

Product Integration Test (PIT)

Function Description

Function Test (FT)

Analysis & Design and Detailed Design

Module Test (MT)

Design Description

Implementation

Figure 2.3. The V-model used at ABB

16

- Implementation and Evaluation

The dotted arrows depict which documents, developed in the phases on the left side, which are used in the stages on the right side of the V-model. An example is the requirement specification, which is used as an input document in the phases STT and PTT.

2.4

Software Maintenance

After development, software requires more effort. Some defects in software products are not discovered until after their release. The correction of these defects is called maintenance. Failures do not always occur because of a defect directly in the product. They can also occur when the environment, in which the product is used, changes. The environment consists of hardware products and other software products. A change in the environment can sometimes require changes, i.e. maintenance, in the tested software product. It is preferable to decrease the need for maintenance during the development of the product since maintenance entails additional costs. Defects found late in the development process bring much heavier expenses than defects found early.

2.4.1

Maintenance Cost

Maintenance costs of software systems vary widely from one application domain to another. Sometimes they are roughly comparable with system development costs. Occasionally, however, maintenance time and costs may be up to four times as high as the development costs. It is all depending on what kind of system that is being dealt with [SOMMERVILLE 95], [MOSLEY 02]. It is usually cost-effective to invest effort to reduce maintenance costs when designing and implementing a system. Adding functionality after delivery is very expensive and should be avoided as much as possible, since it involves understanding and an analysis activity. A high development cost may decrease the maintenance needed, and by those means the total cost of development and maintenance. Maintenance costs are related to a number of factors concerning product, process and organisation. The principal technical and non-technical factors, which affect maintenance of a system, are [SOMMERVILLE 95]: • • • • • •

17

Module independence. It should be possible to modify one component of a system without affecting other parts of the system. Programming language. Programs written in a high-level programming language are usually easier to understand and therefore also easier to maintain than programs written in a lowlevel language. Program style. The way a program is written contributes to it being understandable and hence to the ease with which it can be modified. Program validation and testing. The more time and effort spent on design validation and testing of the program, the fewer defects. Thereby maintenance costs are minimised. The quality of program documentation. With a good documentation, the task of understanding the program will be much easier and time for maintenance reduced. The configuration management techniques used. Keeping track of all system documents and ensuring that these are kept consistent can be expensive. Effective configuration management can help controlling this cost.

Test Automation at ABB Automation Technologies AB

• • • • •

The application domain. If the application domain is clearly defined, well understood and the initial requirements are not modified frequently, less maintenance may be needed. Staff stability. Maintenance costs are reduced if system developers are responsible for maintaining their own programs. There is no need for other engineers to spend time to understand the system. The age of the program. As a program is maintained, its structure is degrading. The older the program, the more maintenance it requires and the more expensive the maintenance becomes. The dependence of the program on its external environment. If a program is dependent on its external environment it must be modified as the environment changes. Hardware stability. Programs must often be modified to use new hardware, which replaces obsolete equipment.

Taking these factors into consideration, much time and money can be saved in maintenance.

2.4.2

Measuring Maintainability

It is hard to anticipate how much maintenance a product is going to need. To make a rough estimation, maintainability metrics can be used. These do not measure the cost of making a particular change to a system, nor do they predict how much maintenance a particular component will need. They are rather based on the assumption that the maintainability of a program is related to its complexity. High complexity is correlated with difficulties in maintaining the system component. There are many theories on how to measure a program’s complexity, for example graph theoretic techniques, considering the number of unique operators and operands, a spectrum of metrics etc. All these techniques may have some validity but they also have disadvantages; they do not take everything into consideration, something that is almost impossible to do [SOMMERVILLE 95]. Therefore perhaps more than one technique should be taken into account when trying to measure maintenance. After a system has been put into service, data about the process can be collected. Trends in the data can give indications about the program’s maintainability. There are always several different factors influencing the maintenance of the program. Here are some examples, which may be useful for assessing maintainability [SOMMERVILLE 95]: • • •

•

Number of requests for corrective maintenance. If the number of failure reports is increasing, this may indicate that more defects are being introduced into the program than are being repaired during the maintenance process. This may indicate a decline in maintainability. Average time required for impact analysis. This reflects the number of program components that are affected by the change request. If the average time increases, it implies that more and more components are affected and maintainability is decreasing. Average time taken to implement a change request. This is not the same as the time for impact analysis although it may correlate with it. The activities involved are changing the system and its documentation rather than assessing what components are affected. If the time to implement a change increases, this may indicate a decline in maintainability. Number of separate change requests. If this number increases with time, it may imply a decline in maintainability.

Any measurements must be analysed with care. It does not necessarily mean that there are clear relationships between these metrics and system maintainability but it might give a clue of how 18

- Implementation and Evaluation

much maintenance there is going to be in the future. It is important just to know whether the maintenance is decreasing or increasing. Prediction of maintainability is indeed worth investigating, but whatever technique is used, its limits should be taken into account.

19

Test Automation at ABB Automation Technologies AB

20

- Implementation and Evaluation

3 Software Testing Theory

Testing is performed on software products to evaluate if they work as intended and to detect failures and thereby defects. It is often performed during the development of the product to find defects and also prior to the release of a product to confirm that it is free from defects and ready to be released. Furthermore, testing is also performed on products that have already been released, to discover potential defects that were not discovered prior to the release. It is important to make a clear distinction between the two terms failure and defect used throughout this thesis. A failure has occurred when something in the test execution goes wrong, and often prevents the test from being able to continue executing. A failure is caused by a defect in the tested application or in the software used, and the developers introduce the defect.

3.1

Testing Definition

The term testing is very broad. In IEEE Standard Glossary it is defined as [SCHULMEYER 98, p.16]: “Testing is the process of operating a system or system component under specified conditions, observing or recording the results, and making an evaluation of some aspect of the system or component.” Another, more narrow, definition is [SCHULMEYER 98, p.16]: “Software quality testing is the systematic series of evaluation activities performed to validate that the software fulfils technical and performance requirements.” Finally, a third and even easier definition is as follows [RAKITIN 01, p.131]: “Testing is the process of executing programs with the intention of finding errors.” With testing, defects are found earlier in the development process than without. This reduces the costs, since it is more expensive the later a defect is found. A defect found late in the development takes more time, and therefore more money to correct, than it would if it was found early on [LÖFQUIST 04]. Software companies often strive to produce software of the highest possible quality. Testing along the development of a software product is a requirement of accomplishing good quality. There are many different definitions of quality. A product has to meet certain requirements to be considered of good quality. An important purpose is the satisfaction of the customer. The definition of quality by IEEE is [IEEE]:

21

Test Automation at ABB Automation Technologies AB

“The degree to which a system, component, or process meets customer or user needs or expectations.” Testing increases the quality by revealing defects. The defects are then corrected and the software product has taken another step towards fulfilling the requirements and thus the needs of the customers. Testing has its limitations, which is expressed in the following statement [RAKITIN 01, p.131]: “Testing can show the presence of bugs but never their absence.” This implies that even though a test does not find any defects it is not guaranteed that there are none, due to the human factor. Even experienced testers cannot come up with much more than half of all the test cases needed to prove that there are no defects [RAKITIN 01].

Verification and Validation

3.2

Verification and validation are two different ways of performing software testing and are often used interchangeably. Verification is the process of confirming that the product meets the requirement specification. It is performed parallel with the software development and answers the question [SOMMERVILLE 95, p.446], [RAKITIN 01, p.57]: “Are we building the product right?” Validation is the process of determining whether the product satisfies the customer's needs and is performed during or after the software is developed. It can be expressed as [SOMMERVILLE 95, p.446], [RAKITIN 01, p.57], [SCHULMEYER 98]: “Are we building the right product?”

3.3

Good Test Practices

There are some good practices to follow when writing tests. They can be summarised as follows [RAKITIN 01, p.32]: • • • • • • •

A good test has a high probability of finding undiscovered defects. The person who wrote the application to be tested should not perform the test. Each test case should have a description of the expected result. A test should be written for conditions with invalid input as well. There must be some objectives with the testing. Skilled and experienced people should perform the testing. The results of the test should be thoroughly inspected.

22

- Implementation and Evaluation

3.4

Test Levels

Testing needs to follow the process of software development in order to be efficient. Software development starts with implementation of units, and then succeeds with implementation of modules, sub-systems and entire systems. Therefore, testing is often composed into the corresponding levels, which can be seen in figure 3.1 below [SOMMERVILLE 95]. These levels can be compared to the right side of the V-model, which was described in section 2.2.

Unit testing

Module testing

Sub-unit testing System testing

Component testing

Integration testing

Acceptance testing

User testing

Figure 3.1. Test levels

3.4.1

Component Testing

Component testing includes unit testing and module testing. During unit testing small components of code are tested individually. Module testing includes testing of dependent components, which are put together, forming a module. The modules are tested separately, and the objective is to find defects in individual modules [SOMMERVILLE 95]. Unit and module testing are used for checking if algorithms and logic have been correctly implemented, if data structures are used correctly, if the different modules are compatible etc. In many organisations, the developer who wrote the code performs the testing of the units and modules. Yet the testing is prone to find more of the defects if it is performed by a partner [RAKITIN 01].

3.4.2

Integration Testing

With integration testing, groups of modules are tested together to discover any defects in the interfaces between the units – sub-unit testing. Integration testing also includes system testing or validation [RAKITIN 01]. All the subunits are put together and tested as a complete system. The 23

Test Automation at ABB Automation Technologies AB

objective is to find failures related to defects in the interfaces of modules and to determine if all specified requirements are met [SOMMERVILLE 95]. Integration testing is necessary even though the units are tested separately at first, as subfunctions are combined they do not always produce the key function expected. Additionally, defects in interfaces are not always detected with only unit testing. Unfortunately, many software companies do not pay enough attention to integration testing. Instead, the testing starts when all of the modules in the product already have been integrated together. This results in difficulties in finding which modules that cause the errors. Integration testing covers a broad range of activities, from testing a few modules to a complete system. There are a couple of different approaches that can be used to perform integration testing and the interested reader is referred to Rakitin [RAKITIN 01]. Integration testing encompasses the entire software system to ensure that all the components work together as they are intended to. Examples of integration testing include navigation, stress and performance testing, data verification and data integrity across screens and platforms. Integration testing should provide a large number of possible scenarios, including deliberately entering wrong information. The goal is to try to break the system and find all its hidden weaknesses. Stress and performance testing is especially important for high volume applications, to ensure that the system will be able to handle demands from the expected number of peakperiod users [HAGEN 98]. At ABB Automation Technologies AB in Malmö, integration testing is performed by the Titan project. The integration tests include testing the entire Control Builder and thereby the PPA and the controller. 3.4.3

User Testing

User testing includes testing where the customers are involved to see if the product meets their requirements. During acceptance testing the customers are directly involved, which is the difference from validation testing mentioned in the previous section. It can include the same tests as validation testing, or it can use tests developed entirely by customers [RAKITIN 01]. Alpha testing is another term for acceptance testing. Beta testing can be performed on a product that will be released [SOMMERVILLE 95]. Subsets of the customers test the pre-released version of the product. The chosen customers execute commonly specific test scenarios, about which the company wants feedback. The customers are more motivated if the company promises to repair the defects that are detected during the beta testing [RAKITIN 01].

3.4.4

Regression Testing

Regression testing is often performed during the phase of integration testing. Regression testing differs from integration testing, since it tests all existing functional components to ensure that they still work together, regardless of whether they were involved in the project or not. Defects can unwittingly be introduced in the unchanged parts of a system; this is a phenomenon called regression. That is why unchanged components still have to be tested [POL 02]. This is especially important for systems with numerous interfaces to other internal or external systems. The 24

- Implementation and Evaluation

objective of regression testing is not to try to break the system, but rather to make sure that everything still works, and that the corrections of a defect not have introduced new defects. Regression testing involves selectively repeating existing integration tests, not developing new tests [HAGEN 98]. Testing unchanged parts of a system takes less effort than testing changed or new parts. Notwithstanding, the total effort needed for regression testing is much greater than the effort needed for detailed testing of the changes. The reason for this is maintenance since the number of changed functions is limited. Regression testing is performed frequently, at least once for every release. The changes between the releases are usually negligible, which means that the regression test cases are very reusable and require only minor adaptations for every new release. A test to validate a specific change, on the other hand, is very different since it usually is executed for one specific release only. The frequent use and the minor changes between releases speak in favour of high reusability and thereby regression tests. Therefore it is important to have, and maintain, a structured and documented set of tests. The set of tests consists of existing test cases, especially those that control integration, and is composed during the development of the system. The regression test always has to be up to date, which is performed by adapting and enlarging the set of tests, based on changes in the system. Because of the nature of regression testing described, the test is a primary candidate for automation [POL 02]. Automated tests are best used for finding defects during regression tests, and manual tests should be used to find defects in the tests preceding the regression tests [MOSLEY 02]. The Daily Test at ABB is a regression test. It is performed frequently, in the same manner every time and it tests the entire product, regardless of which parts that has been changed since the previous version. The Daily Test is a part of the Titan project, which otherwise mainly performs integration tests.

3.5

Test Techniques

Different techniques can be used to test software. Some of them are described in this section. Several of these techniques can be used during the development of a software product.

3.5.1

Dynamic and Static

When dynamic testing is performed the tested program is being executed. The result of the test is compared with the expected result. There are, however, parts that cannot be tested dynamically. Testing performed without executing the code is called static testing and tests and evaluates among others the hardware, the documentation and security procedures [POL 02] with reviews, walkthroughs, inspections etc. [HEISER 97].

25

Test Automation at ABB Automation Technologies AB

3.5.2

White-box and Black-box

White-box testing, also called glass-box or structural testing, is a test-specification technique based on the program code, program descriptions and technical design. It requires knowledge about the internal structure of the system. Black-box testing, also called behavioural or functional testing, is based on the functional specifications and the quality requirements. Knowledge about the system structure is not necessary, the implementation is treated as a black box [POL 02], [HEISER 97].

3.5.3

Formal and Informal

With informal test-specification techniques, the tester is allowed to use his/her creativity when writing test cases. This makes the quality of the test more dependent of the tester's knowledge. Informal test-specification techniques are not as dependent of the quality of the basis of the test as formal test-specification techniques are. The disadvantage is that the degree of coverage in relation to the test basis is less clear [POL 02].

3.5.4

Program Interface Test

The program interface test, PIT, called project integration test in the Titan project, is a test-specification technique used to test interfaces between programs and/or modules. Before the programs can be integrated, they have to be tested individually using simulated data flows. When the parts work properly independently of each other, a PIT can be started to verify if the programs work correctly even after integration. Real data flows are used instead of simulated ones [POL 02].

3.6

Test Planning

The best way to plan how the resources should be distributed is to write a test plan. Rakitin advocates the following three documents for the planning [RAKITIN 01, p.149], [MOSLEY 02, p.235]: • • •

The test plan. Defines what must be done and who will do it. Contains information about the purpose and goals of the testing. All of the testing activities planned should be described. The test procedure. A document that contains all the test scripts that will be used. Detailed instructions for set-up and execution. It should contain a not executed, clean copy of every test to make it easy to reuse the tests. The test report. The result of the executed test scripts including e.g. found and corrected defects.

26

- Implementation and Evaluation

3.7

Test Design

Planning and design are important parts of the test process and should be done first, before the implementation. This ensures a better quality than if planning and implementation are made simultaneously. A good test case design includes the following elements [FEWSTER 99]: • • • •

Test name or test ID - unique identifier for the test Test purpose - a brief description of what the test is supposed to accomplish Test method - a description of the steps, which the test will be done according to. The steps should be clear enough so that a person could perform the test manually given the information provided. Pass/fail criteria - a few lines describing how to assess the results of the test.

These elements help to give a good survey over what the test does and which part of the application that is tested. The test is then well documented, and thereby easier to maintain, as mentioned in section 4.6. By including these elements in the design, many defects are discovered during the development of the design. Additionally, the elements are useful when deciding what parts of an application that are suitable for test automation. A test case tests a small part of the software. Every test case should be independent of the other test cases, and should also have its own set-up, verification and clean-up. This makes it possible to run different test cases in any order. The set-up prepares for the actual test, the verification is the test, and the clean-up returns the application to a base state, ready for the next test. The clean-up should be made even when the entire test cannot be executed because of a defect detected. If that is done, all test cases start from the same point in the application. An initial or base state is when the application is open with all menus enabled, but no child window or dialogue boxes open [MOSLEY 02]. The term home-based test cases is used by Mosley [MOSLEY 02] for test cases that are built with respect to a base state. The independence described is very important, since a defect in one test case does not affect the other test cases and they can be run anyway. Furthermore, the chance of test execution failure is minimised when each test case begins at a known point and cleans up after itself. The interested reader is referred to [FEWSTER 02] and [MOSLEY 02]. To make a test case as short and simple as possible, every test case should test one thing only. This makes the test case easier to understand and maintain. Defects are easier to detect and trace when only a small part of the application is tested.

3.8

Test Maintenance

Tests have to be maintained when the application that is being tested changes, or when the test needs to expand to test new functionality. Reducing the need of maintenance is just as important for tests as for the application, which was discussed in section 2.4. Building maintainable tests very much resemble building maintainable software; building tests is in fact software development. Therefore many of the rules applied to software development also apply to developing tests. Maintenance is sufficient but expensive here as well. Building a test that 27

Test Automation at ABB Automation Technologies AB

minimises the need of maintenance is important to minimise the costs. A test that is easy to maintain and does not need much maintenance has to fulfil certain standards. These standards are presented in the following list [FEWSTER 02]: •

• •

•

•

•

Many tests in a test suite. A test suite contains several test cases and tests subsystems of the application, or the entire application. A test suite should contain as many tests as possible, since the test cases are easy to add and the cost, as well as the time to execute each of them, is negligible. Too many test cases on the other hand, can result in a complex, expensive and time-consuming test suite. Furthermore, the risk for duplication and redundancy increases. The conclusion is that it is essential to have a fair amount of test cases in the test suite, and thereby a test suite that is easy to survey. To decide what is a fair amount is not an easy task. Not too much test data. Much test data in a tests makes many combinations of data input and a thoroughly test possible. The negative effects are that it makes the debugging efforts and test failure analyses more difficult. Small test cases. The test cases should be as small and focused as possible. A long test case, instead of many small ones, makes the set-up and clean-up easier since it only has to be done once. The problem is that it becomes harder to debug, and defects that are detected early in the test case make it hard to discover defects later in the test case since a test is often unable to continue after a failure. Independent test cases. The output of one test case should not be the input of another. This makes the test cases independent of each other, and consequently more defects can be discovered when a failure in one test case does not affect the other test cases. A single long test case can be complex to understand, whereas many and shorter test cases are more comprehensible. Comprehensive results. An effort that can be made to make debugging as easy as possible, is to present the test results in an as comprehensive way as possible. The more information that is given about what went wrong, the less time has to be spent finding the problem. Instead the defect can be corrected immediately. This saves a lot of time, effort and money. Name conventions. Maintaining the tests can also be made easier by having name conventions. This is recommended when more than two persons are involved in test work, to avoid non-consistency, chaos, duplicates and trouble in finding script and files. The tests should also always be well documented, as mentioned in the section 3.7 above.

The standards specified above may be difficult and time-consuming to implement, but tests that fulfil these have the advantage of much lower maintenance costs. Consequently, time spent on making the test easy to maintain will pay off later, since it will decrease the maintenance costs considerably.

3.9

Test Roles

It is important to have a good understanding of the different responsibilities and tasks involved in testing, to be able to decide what kind of knowledge and skills that are required. The various responsibilities and tasks in testing gives rise to roles such as, among others: tester, team leader and test management [POL 02].

28

- Implementation and Evaluation

3.9.1

Tester

The role tester includes the primary testing tasks, which are the most creative but also the most labour intensive. A tester’s tasks can embrace reviewing of the test specifications, specifications of test cases, execution of test cases, registration of defects, preserve test-ware etc. [POL 02].

3.9.2

Team Leader

A team leader is responsible of providing guidance to a team, which usually consists of two to four testers. Different tasks a team leader can have are creating detailed plans of the activities, spreading workload over team members, monitoring the progress, keeping record of the progress and allocating the time budget, guidance and support for testers etc. [POL 02].

3.9.3

Test Manager

The test manager has responsibilities as planning, management and execution of the test process within different frameworks, such as schedules and budgets and according to the quality requirements [POL 02].

3.9.4

Test Automation Roles

A test team that wants to introduce automated testing has to add new test roles. There has to be a test automation architect, who designs the test suite and a test-automation engineer, who is a tool specialist and who can create the test suite according to the design [POL 02].

29

Test Automation at ABB Automation Technologies AB

30

- Implementation and Evaluation

4 Automatic Testing

Software testing can be performed automatically. This requires some kind of automation tool, which is used to build and perform the automatic test. Automatic testing can be performed unattended and is often faster than manual testing. Despite these advantages not all tests are recommended for automation. The main purposes of automated tests are [MOSLEY 02]: • • •

Test that the application does what it is intended to do. Test that the application does not do anything that it is not supposed to do. Test that the application is robust.

This agrees with the intentions for manual tests, the difference being the automated tests are more often used to verify that applications, which have already been tested, still work. The Application Under Test, AUT, has to work correctly for automation to be feasible. Automated tests detect defects that have been introduced accidentally after the feature that is being automatically tested has been delivered, and defects that were not discovered prior to the delivery [MOSLEY 02]. The usage of modern tools for automated testing, when performing software testing, can help speeding up product development, as well as improving quality and reliability at the same time [GASCOYNE 00]. When the automated tests are built, the same steps as when executing the tests manually are performed. These steps are performed once for the automatic tests but every time the test is executed for manual tests. Hence more defects are discovered during the development of the automated tests than during the execution of the finished test scripts [MOSLEY 02]. Automation begins when the automated tests are built, i.e. when all the test cases are defined and after a build of the AUT is delivered. A general rule of thumb is that it takes three to four times more time to build the automated test, than to perform the manual test [MOSLEY 02, p.30].

4.1

Advantages and Disadvantages

There are many advantages with test automation [POL 02], [MOSLEY 02]: • • • • 31

A large number of tests can be performed unattended and at night. The reliability is greater, since the tests are performed in the same way each time. Different people carry out manual test differently and the results of the tests can therefore vary. Many tests can be performed during a short period of time. Automatic tests are cheaper to execute than manual tests.

Test Automation at ABB Automation Technologies AB

• • •

It possible to extend the test to a higher coverage with a lower price. The quality increases. The test is performed more often and takes less time to perform than manual tests, which makes it possible to deliver the product earlier. The human interaction factor is removed.

There are, however, also disadvantages [POL 02]: • • •

The building takes much time, since each test must be recorded manually. High maintenance costs. Changes in the tests must be done when the functions in the product to be tested changes. It is not as dynamic, if a failure occurs, as a manual test would be. With manual testing, it is easier to work around a failure and be able to proceed with other parts of the test.

By considering the advantages and disadvantages with test automation before trying to automate a test, a lot of time and money can be saved. In the following sections, what to reflect on before automation is more thoroughly described.

4.2

When to Automate

There are several aspects to consider when deciding whether automation is a good alternative or not. The set-up of automated tests always brings much more work in the beginning than the setup of manual tests. The automated tests have to be built, and the same steps as when executing manual tests have to be performed. Automated tests can be used only if the application features to be tested work correctly. Otherwise, it is not possible to create valid and completed automated tests against a requirement or application feature. The primary aim with an automated test is to verify that there are no defects and that a requirement is met in all successive modifications of the AUT. Since automated testing reduces the time to execute regression testing, this is where its benefits are realised the most. Automation costs more up in front, but pays off rapidly every time regression testing is performed. Hence automation in projects with very tight schedules can be difficult to achieve, since the set-up consumes a lot of time. A few questions to consider when determining if automation should or should not be used are as follow: [MOSLEY 02]: • • • •

Is the feature set in the AUT simple or complex in nature? Is the feature set in the AUT subject to significant change during the development process? Is the feature set in the AUT currently working as stated in the requirements? Is the test tool being utilised for automated testing capable of interacting with all the necessary attributes of the feature for test purposes?

The questions above are not easy to answer. There are a couple of guidelines on when to not use automation: • •

If the AUT is not complex and not very large If there will be less than three builds 32

- Implementation and Evaluation

• • •

If the feature does not work at 100%. It is all right to plan for it but the automated test scripts should not be created until the AUT is finished and working correctly. If there is not enough time between deliveries. If the feature cannot be tested with 100% accuracy via automated testing, unless it saves a considerable amount of manual test time. The results should always be predictable and be able to produce a pass or fail condition.

The advantages and disadvantages should also be considered when determining whether or not to use automation. These can be found in section 4.1.

4.3

What to Automate

The goal should be to automate testing of the critical parts of the AUT. Primary it includes automating the functions that will be performed by the end-user, since these parts often are the most central in the beginning of the development. Then, in course of time, parts that are not that critical can be automated. It is important to have a general view during the entire automation process, of which portions of the test that have been automated. According to Mosley, practical experience has shown that, on an average 60% of a project can be automated [MOSLEY 02]. Testing on parts as login, user preferences and other options should not be automated early in the automation process. Nor should testing of status bars, help screens etc., since these parts are not important in the development until later on in the process. Still, the planning of the automation strategy for these parts should be done in the development phase. The implementation of the strategy should, however, wait until the development effort focuses on them. The advice above can help in deciding what to automate, but the decision-makers must have in mind that what is appropriate to automate varies from project to project.

4.4

GUI-tests

A graphical user interface, GUI, is an interface that provides the user with a way of interacting with the computer. It allows keyboard and mouse input and organises the information in a graphical way. The interface of the operating system Windows from the company Microsoft is an example of a GUI. A GUI-test performs a test by using the GUI of the application tested. The Daily Test tests the basic functionality in the Control Builder, and consequently in the PPA, by performing actions in the GUIs of the two programs. The Daily Test is therefore a GUI-test. GUI-tests differ from other automatic test techniques. There are often many ways of executing an operation when using a GUI. In new versions it is also common that the GUI changes, but not the underlying functionality. The GUI-tests have to be rewritten every time the GUI changes. Different GUI-tools are available for testing GUIs automatically. They are usually complex, mainly because of the many common GUI-controls, such as list boxes, pushbuttons and menus, which they have to be able to manipulate. The speed of the script execution and the application speed have to be synchronised. They also use an own scripting language, which makes it more difficult for the scriptwriter [FEWSTER 02]. 33

Test Automation at ABB Automation Technologies AB

GUI-tests can be automated by using record and playback. This is described in the next section.

4.5

Record and Playback

Record and playback, or capture and playback, can be used in some testing tools, which have a recording mode. In this mode, the tool captures the actions performed by a user in the GUI of an application, and records them in a script. The script can be replayed, and thereby the application is manipulated automatically. A record/playback-tool is a GUI-tool, which was mentioned in the previous section. The positive aspects of record and playback are, firstly, that it can generate lots of test code at a low cost. Secondly, it is also easy to use for people with no coding experience. Lastly, it minimises the cost of building the automatic tests. The negative aspects are that it maximises the maintenance costs, manual changes in the script are necessary for the test to work properly, and it does not fulfil any of the requirements for automated GUI tests [FEWSTER 02]. The requirements are described in section 4.6 below. The scripts, which are generated with recording, contain hard-coded values which demands manual changes if anything changes in the application. Maintaining these scripts is tremendously expensive. The scripts are not stable; they can fail on replay even though neither the script, nor the application, has changed since the last successful replay. Possible reasons can be the occurrence of e.g. pop-up messages and windows that did not occur when the script was recorded. If a tester makes a mistake, or if the application changes, the test must be re-recorded. Programming the scripts manually makes the automated tests more reliable and can make the test more powerful and complex as well. A recorded script can be made more stable if it is combined with manual programming. The script must be recorded, or manually programmed, to take care of all kinds of events that can occur during execution, to prevent the execution from failing. This results in a lot of code being generated and thereby in a great deal of script maintenance later on. Every time anything changes in the GUI of the AUT, the test script has to be partly changed, either by recording or by programming, or the test will fail. Automation can sometimes be questionable, as Mosley establishes, when the features being tested already work, the defects are found in the manual part of the automated test, that is in the recording phase, and a script cannot be recorded until the defects are corrected. Considering this, Mosley wonders what is actually being tested [MOSLEY 02]. According to Mosley the only way to effectively test application functionality with an automated test tool, is to use data-driven testing. He defines data-driven testing as follows [MOSLEY 02, p.106]: “Data-driven testing is testing where data contained in an input test data file control the flow and action performed by the automated test script.” The test script program reads test data input from an external file, separated from test script program. A data-driven test script can include parameters that can be used as input to the 34

- Implementation and Evaluation

program, sequences of test data through which the program is driven etc. Data-driven testing holds several advantages: • • • •

Highly maintainable. The test script does not need to be modified when new test data is added. The new data is added to the text file. Simple input text files. These files are easy to understand and make modifications in. Easy for nonprogrammers to use. It is easier to make modifications to the text file than to the test script. Multiple input data files. Multiple data files can be created and used when required.

For more information about data-driven testing, the reader is referred to [MOSLEY 02].

4.6

Requirements for Automated GUI Tests

There are requirements an automated GUI-test has to fulfil to be feasible. If the requirements are not followed, the automation can bring more work and higher costs than a manual test. Manual tests should also take the requirements into account, which is why several of the requirements are mentioned in section 2.4.1. The requirements are [POL 02]: • • • • •

4.7

Maintainability. The application to be tested changes between different releases. This requires changes in the test script as well and therefore it is important that these can be accomplished without too much effort. Modularity. Changes in one part of the application should not affect other parts of the application. Dependent software components are merged, forming modules. A module can be tested without interacting with other modules of the system. Robustness. Changes in the application should not require major changes in the test script. Well documented. The same engineer, who wrote the test script, does usually not make the changes and therefore the test script has to be well documented. Reusable components. The components in the test script should be reusable, for avoiding having to build the same component again, when developing the script between releases.

Costs for Manual and Automatic Tests

Initially automated tests are more expensive compared to manual tests with respect to the cost to build them, but they are cheaper to execute. Hence, the aim with automatic testing is that the total cost for a longer period of time will be lower than the corresponding costs for manual tests. The total cost for manual respectively automatic testing is: Total costman = cost of one session * number of sessions per release * number of releases Total costauto = cost to build + (cost to run one session * number of sessions per release + maintenance cost * number of releases) This is based on the assumption that the cost for one session between the different sessions does not change, and that the number of sessions is constant between the releases [FEWSTER 99]. The factor cost of one session in the manual test equation has been divided into cost to build, cost to run 35

Test Automation at ABB Automation Technologies AB

one session and maintenance cost in the automatic test equation. The maintenance cost is the factor that has the greatest impact on the total cost, whereas the cost to run one session only bears a little part of the cost. A session is one test execution and cost to build is the cost to create and implement the automatic test. A release can be made to the customers or to the test team within the company. Either way a new version of the product, with defects corrected or perhaps some new functionality, is at disposal. The maintenance cost is the cost to update the test by correcting defects and adding functionality. Building a test suite is a one-time cost while the maintenance cost is frequent because it increases every time the software is updated. This makes the impact of maintenance costs larger than the one of building cost. Decreasing the costs for maintenance is substantial, even if it means increasing the costs for building the test. Measuring how much maintenance that is needed is very difficult though, as explained in section 2.4.2. Some people underestimate the cost of maintenance as can be seen in the utterance below [MOSLEY 02, p.27]: “Test Automation is an investment. The initial investment may be substantial, but the return on the investment is also substantial. After the number of automation test runs exceed 15, the testing from that point forward is essentially free.” This can only be true if the test that is performed automatically fulfils the guidelines on when to automate, which were described in section 4.2 [FEWSTER 99]. When calculating the automation cost it is important to search for unique factors that affect the automation cost of the particular test. While doing this it is essential to follow some strategy to receive a reliable result. More about the different strategies can be read in the following chapter.

36

- Implementation and Evaluation

5 Evaluation Methods

A research can be performed in different ways depending on the purpose with it. It is important to follow some strategy to receive scientific results.

5.1

Qualitative and Quantitative Methods

Qualitative methods, also called flexible, are performed to understand how a target group thinks about a certain issue. This kind of research can be suitable when preparing quantitative methods, for example what to ask about in a questionnaire. Qualitative methods can also be used to follow up quantitative methods and get a deeper understanding of the answers. Qualitative methods are action research, case study research and ethnography. Qualitative data sources include documents and texts, group discussions, interviews, observations and the researcher’s impressions and reactions. Quantitative, or fixed, methods are usually used to prove relations and get key numbers. It is often performed by the usage of questionnaires, laboratory experiments, formal methods and numerical methods such as mathematical modelling [RADVIS 02], [MYERS 97], [LÖFQUIST 04].

5.2

Interviews

Interviews can be structured in many different ways. An open interview means that the interviewer asks a broad and open question on which the interviewee can answer quite freely. When openended interviews are used, the interviewee is given an open question with subjects and deepens in what the interviewer finds meaningful. The semi-structured interview gives the interviewee certain subjects to discuss in a special order and the answers are both open and fixed. When questions have answer alternatives they are said to have fixed answers. In structured interviews the interviewer has prepared questions and subjects to discuss, the answers are fixed and the questions are asked in a special order. In open and open-ended interviews the interviewee describes his/her apprehension of the subjects discussed. The interviewee determines the context in contrast to the structured interviews where the interviewer is more in charge. The most structured way of asking questions is the questionnaire with questions and fixed answers [LANTZ 93]. Some of the strategies described were used when searching for unique factors that affect the cost of the automation of the Daily Test. More about this can be read in section 7.3.

37

Test Automation at ABB Automation Technologies AB

38

- Implementation and Evaluation

6 Automation of the Daily Test

This chapter starts with a description of the Daily Test and continues with a description of the work process during this master thesis.

6.1

The Daily Test

The Daily Test tests the basic functionality in the Control Builder to check if the integrated parts implemented by different developers work together. Consequently the basic functionality in the PPA is also tested. The Control Builder is used to program and control controllers, and is started from the PPA, which is used to supervise processes in industries. Both programs have graphical user interfaces, GUIs. The Daily Test is a GUI-test since it is performed by clicking in the corresponding GUIs. The contents of the Daily Test are listed below, a more detailed explanation of the software can be found in appendix A. Installations • • • • • •

Load the ghost to the test computer Install the PPA Install the Control Builder Install the OPC-server Install the firmware System extension

Actions performed in the PPA • • • • • • •

Create a control network Change from function structure to control structure Create a project Open the project (the Control Builder opens containing the project) Backup the system Delete the system Restore the system

Actions performed in the Control Builder • • • • • • 39

Replace CPU-unit with the correct one Set ip-address to the correct one Download firmware to the controller Add IO:s, corresponding to the IO:s connected to the controller Download the empty project to controller Check mirroring of hardware between the Control Builder and the PPA

Test Automation at ABB Automation Technologies AB

• • • • • • • •

Write a program, connect some of the variables to I/Os on the controller Download the project, now containing the program, to the controller Check if the right variables are kept in the controller Make a change in the program, and download the project once again to the controller Check if the right variables are kept in the controller Perform warm and cold restart, check if the right variables are kept in the controller Upgrade with an old project to test compatibility with an older version of the Control Builder Import a library

The three steps Backup the system, Delete the system and Restore the system are performed last, after Import a library. The term ghost is explained in section 6.2.2. The content of the Daily Test was changed during this master thesis. This was not taken into consideration in the implementation.

6.2 6.2.1

Implementation Solution

The record and playback tool WinRunner was chosen for the automation of the Daily Test for a couple of reasons. Firstly, the project Mjölner at ABB in Västerås had already compared Rational Robot and WinRunner and found them equal. More information about the evaluation can be found in Appendix B. Secondly, WinRunner was already in use at ABB in Malmö and therefore people there had knowledge about it already, thus no extra investment was required. Additionally, according to the master thesis Automated Testing of Software with Graphical User Interfaces the tool has extensive functionality and intuitive error reporting and was considered the best of the three tools investigated in the thesis [ADELSWÄRD 98]. The solution in short consists of: • •

A Ghost with the PPA, WinRunner and some other essential programs installed. A WinRunner script containing the test.

The WinRunner script was developed partly with help from functions that already existed and had been written by one of the testers in the Titan project. The authors of this thesis wrote the rest of the functions used. All the functions used by the test script are gathered in a separate script.

6.2.2

The Ghost

When a disc on a computer is ghosted, an image or a copy of everything on the ghosted disc is created. Loading a Ghost means that the affected disc is restored to exactly the same state, as it was in when the Ghost was created. 40

- Implementation and Evaluation

It is important that the test computer is clean and looks exactly the same every time the Daily Test is run. Therefore a Ghost is made with only the PPA, WinRunner and some other essential programs, such as Excel, installed. This Ghost is then reloaded on the computer at the beginning of each test execution. By doing this, the computer environment looks the same every time the test is executed, without having to reinstall the necessary programs. Just deleting all traces of the previous test is not enough since there might be parts that cannot be found, lingering in the system. The Daily Test always has to use the latest version of the PPA. Therefore a new ghost has to be made approximately every second week, with a new PPA installed. The installation of the PPA takes about two hours and hence the creation of a new Ghost takes about 2.5 hours. As mentioned in section 1.5, the installation has to be performed manually. The testing process is started from a computer in the network every night. To start executing the test at night the tool Batch Builder is used. It restarts and loads the ghost to the test computer, which then performs the test automatically. Appendix A contains more information about the Batch Builder.

6.2.3

The WinRunner Scripts

WinRunner is a record and playback tool, as mentioned above. There are three main stages in testing with WinRunner: creating tests, running tests and analysing test results. These stages are explained in Appendix C. To make the test as easy as possible to maintain, the decision was made to build the test script based on functions. Functions are reusable and the same code does not have to be recorded or programmed more than once. This decreases the size of the test that has to be written and, saves a lot of time. Each function is made as small as possible, comprising only a small part of the entire test. This results in functions that are easy to take in and understand, and makes it easier to find and encircle defects. The functions used in the Daily Test script are partly written by the authors of this thesis, partly by an employee in the Titan project, and they are all gathered in one script. Some functions have parameters, which should be easy to change. Examples of parameters are the project name and the path to the installation files of the Control Builder. To make these changes easier, the parameters are placed in an Excel-file, from which the WinRunner functions collect them. Consequently, the automatic Daily Test is a data-driven test, a term discussed in section 4.5. It is much easier to change and overview the parameters in an Excel-file, than it would have been to change them in the test script. As described in the section 6.2.2 above, the test computer, which executes the Daily Test, is ghosted automatically at night. The test computer then automatically logs on. WinRunner is placed in the start-up and thereby starts automatically, with the script containing the Daily Test. An important part after the execution of the Daily Test is the presentation of the test results. To be able to present judicious results the information about exactly where the test goes wrong is needed. This encircles the problem and makes it easy to decide what is wrong. To fulfil these demands, the functions have not only to be small but also a result from every function has to be 41

Test Automation at ABB Automation Technologies AB

achieved. The solution is to let every function write its result or status in an Excel file. Appendix D contains more information about this solution. As described in the theory sections, it is essential to return to a base state between the execution of test cases. Many of the functions used in the test script restart the Control Builder and the PPA, to be sure that the tools are in their base state before performing their actual tasks. In that manner the function can start executing even if the previous function did not complete correctly. In this case the functions are dependent of each other, and the gain is therefore not as significant as it would have been if the functions were independent, and could proceed no matter what happened in the previous function. Instead, a function can start, but because of the dependency, it cannot perform all the tasks correctly if a previous function has failed. To reduce the problem of dependency, the test is divided into three parts, which are independent of each other. If a function in one part fails, the test automatically starts to execute the next part. Thereby at least parts of the test can be executed even though a failure has occurred. More about the dependency can be found in Appendix D.

6.2.4

Cold and Warm Restart

The Daily Test includes parts called cold restart and warm restart of the controller. Pushing a button on the controller manually performs cold restart, and breaking the power to the controller does warm restart. By a cold restart only the values of the program variables that have been declared as cold retain are kept. The others receive their initial values. By a warm restart, both the values of the cold retain as well as the retain variables are kept. To be able to perform the cold and warm restart automatically, the controller on the test computer was rebuilt. The controller was connected to two COM ports on the test computer, one for each type of restart. A script and a graphical interface made in Visual Basic perform cold and warm restart, by communicating through the COM ports mentioned. In the GUI it is possible to choose which COM port the script should communicate through, and thereby what kind of restart will be performed. The duration in seconds for the power breakage or button pushing can also be chosen. The procedures for performing cold respectively warm restart were recorded with WinRunner and made into functions.

6.2.5

Conclusion

The part of the test where the ghost is loaded to the test computer works well, and can most probably be executed at night without much maintenance. The only time a problem has been encountered is when a job in the Batch Builder fails, and the rest of the queue in the Batch Builder, including the ghost job for the Daily Test, is stuck. This results in the Batch Builder failing to execute the ghost job for the Daily Test at night. By queuing the ghost job for the Daily Test earlier, no other job can fail before the ghosting is performed. The problem is thereby solved. The successful execution also applies to the first part of the test script where the Control Builder, OPC-server and firmware are installed. Few difficulties have appeared in this part of the test. The rate of failure for the rest of the test, except for backup and restore, is considerably higher; therefore it will also require a lot of maintenance. 42

- Implementation and Evaluation

Failures can occur in the Daily Test for several reasons. They can be caused by defects in any of the products used: Windows, WinRunner, the Control Builder, the PPA, Batch Builder and Visual Studio. Changes made in the GUIs of the Control Builder or the PPA, as well as pop-up windows, can also cause failures. If changes in the PPA or the Control Builder caused the failure, the test scripts have to be modified for the automatic test to work properly. If the failure is caused by something else, the test can either be executed again or the script can be rewritten, to try to handle the problem encountered. By using functions and documenting the code properly, the Daily Test script is quite easy to overview and to comprehend. Despite all effort to fulfil the requirements for automated GUItests and make the test maintainable, however, quite a lot of maintenance is still necessary. WinRunner seems to be rather sensitive to changes in the GUI of the AUT and almost imperceptible changes can cause failures. Reading the parameters from an Excel-file works well, presuming the content of the Excel-file is correct. Writing the results in an Excel-file is a little unstable though; the writing operation sometimes fails without any known reason. When it works as intended, it provides a comprehensible overview of the results. The results give a rather exact view of where the test failed, but not the reason why. Since there are a couple of different programs involved in the test and none of them are 100% stable, any one of these programs could have caused the failure. The only way to know exactly why the test failed is to try to repeat the test at the point of failure. Some failures occur only occasionally. Their cause can be rather difficult to find, since they cannot be replicated. An example of this kind of failure is when Excel fails to close properly. The division of the test into three parts is made to manage to execute as large parts of the test as possible. The jump to another part when a failure occurs works very well. If the installations fail, however, none of the other parts can be executed since the installations are needed. Both warm and cold restart works very well. The automatic test is quite sensitive to changes, both in the Control Builder and in the PPA. This limitation is partly the result of the test being made relatively stable for one version only. It is quite unlikely for the test to function with a new version of the Control Builder, even though the Control Builder itself is free from defects. Changes in the test therefore have to be made for almost every new version. Thereby, the part of the test after the installations is close to impossible to perform successfully at night with new versions. Since only parts of the test can be performed automatically if a failure occurs, the rest has to be performed manually, and perhaps the scripts have to be modified, for a complete evaluation of the product. In chapter 7 it is discussed and determined whether it is profitable to use the automated Daily Test or not.

43

Test Automation at ABB Automation Technologies AB

44

- Implementation and Evaluation

7 Evaluation

Apart from automating the Daily Test, the task in this master thesis is to conclude if an automation of The Daily Test is profitable. This is done by discussing different parts mentioned in the theory chapters, considering the conclusions in the previous chapter and estimating the total cost.

7.1

Advantages and Disadvantages

In order to determine if it is profitable for ABB to use an automation of the Daily Test, the guidelines in section 4.1 are commented from ABB's point of view. The term lead-time is mentioned in the answers. Lead-time is the time it takes to execute the test, i.e. the time the result has to be waited for. If the test is performed at night the lead-time is zero. Advantages: •

A large number of tests can be performed unattended and at night. The Daily Test, essentially no matter how great, can be performed unattended at night. • The reliability is greater since the tests are performed equally each time. Different people carry out manual test differently and the results of the tests can therefore vary. Generally, the same person carries out the manual Daily Test at ABB all the time, but the result can still vary. • Many tests can be performed during a short period of time. The Daily Test takes as long time to perform automatically as it takes to perform manually. However, the test is executed unattended at night, and therefore requires no time from the employees and takes no lead-time. • Automatic tests are cheaper to execute than manual tests. The automatic execution of the Daily Test costs nothing as long as it succeeds at night – neither money nor the time of employees. • It is possible to extend the test to a higher coverage with a lower price. It is possible to extend the test, and still have a lower cost, since the test is performed unattended. This requires, however, that the maintenance cost is low. • The quality increases. The test is performed more often and takes less time to perform than manual tests which makes it possible to deliver the product earlier. Whether or not the quality increases depends on the cause of the failures. If the failures at night depend on defects in the Control Builder, time is saved. If they, on the other hand, depend on WinRunner, time may be lost. The test then has to be performed again at daytime, to discover possible defects in the Control Builder. Hence, the test is performed more often only if the test is executed several times due to defects. The performance of the test takes less time, i.e. no lead-time, with regard to the nightly executions. The maintenance at daytime, however, increases the total time, which implies that it is not sure that the automatic performance of the test takes less time than 45

Test Automation at ABB Automation Technologies AB

•

the manual. Thereby it is not certain that the quality increases and that the product can be delivered earlier. The human interaction factor is removed. The test is performed automatically, thereby it is performed in the same way each time.

Disadvantages: •

The building takes much time since each test must be recorded manually. The building of an automated Daily Test takes much time and the cost is thereby high. • High maintenance costs. Changes in the tests must be done when the functions in the product to be tested changes. The maintenance costs are considerable. The Control Builder changes between the releases of test versions and the test scripts have to be updated, which takes a lot of time. • It is not as dynamic if a failure occurs. With manual testing it is easier to work around a failure and proceed with other parts of the test. An automated Daily Test is not very dynamic as such. The test script is however divided into three parts to make it more dynamic. There is an agreement at all points but the sixth one in the comparison between the general advantages and disadvantages with test automation, and the corresponding ones for ABB. The advantages have to carry a greater weight than the disadvantages for automation to be profitable. Whether it does so or not is analysed in section 7.4 below.

7.2

When to Automate

Another part of answering the question whether automation is profitable or not, is to consider the points and guidelines described in section 4.2. The answers of the questions are as follows: • • • •

Is the feature set in the AUT simple or complex in nature? The feature set in the Control Builder is complex in nature. Is the feature set in the AUT subject to significant change during the development process? Changes significant for the test are made in the Control Builder during the development process. Is the feature set in the AUT currently working as stated in the requirements? The Control Builder is mostly not working completely correct. Is the test tool being utilised for automated testing capable of interacting with all the necessary attributes of the feature for test purposes? The test tool WinRunner is, with moderate modifications, capable of interacting with all of the necessary attributes of the Control Builder and the PPA.

The AUT is the Control Builder, and consequently the PPA. The feature set is the part of the AUT that is tested. The first and the last of the answers above argue in favour of automatic testing. The others, on the other hand, imply that automation is not preferable. Comments on the guidelines on when not to use automation are shown below: 46

- Implementation and Evaluation

•

If the AUT is not complex and not very large. The AUT is complex and large. • If there will be less than three builds. There will be more than three builds. • If there is not enough time between deliveries. There is enough time; the time between the releases of the Control Builder is one year. • If the feature does not work at 100%. It is all right to plan for it but the automated test scripts should not be created until the AUT is finished and working correctly. The feature does not work at 100%. • If the feature cannot be tested with 100% accuracy via automated testing, unless it saves a considerable amount of manual test time. The results should always be predictable and be able to produce a pass or fail condition. The features cannot be tested with 100% accuracy. When considering the guidelines specifying when not to use automation, it is stated that the last two of the five guidelines are fulfilled. Those two guidelines are discussed further: •

If the feature does not work at 100%. It is all right to plan for it but the automated test scripts should not be created until the AUT is finished and working correctly. The Daily Test is meant to verify that the integrated parts of the Control Builder still work together correctly. There are supposed to be no or just a few defects. Most of the executions should be possible to perform flawlessly. In practise, the fact of the matter is the opposite: most of the executions reveal defects in the Control Builder, as well as in the PPA, and cannot be executed correctly. In some cases it is possible to work around the error and continue with other parts of the test but in other cases the testing cannot proceed. This fulfills the guideline and contradicts that the Daily Test should be automated.

•

If the feature cannot be tested with 100% accuracy via automated testing, unless it saves a considerable amount of manual test time. The results should always be predictable and be able to produce a pass or fail condition. The feature cannot be tested with 100% accuracy, since the result is far from predictable. The test succeeds sometimes, but mostly not. The reasons why the test fails vary. It could be changes or defects in the AUT, failures or actions in Windows and failures in WinRunner. The unpredictable sources of failure make it difficult to make the test produce an accurate pass or fail condition. The results show approximately where the test failed, but not exactly what was the reason of failure. Additionally, automated testing of the Daily Test does not save a lot of manual test time, since the test needs a great amount of maintenance. This fulfils the guideline and contradicts that the Daily Test should be automated.

The fact that the feature does not work at 100% introduces a lot of extra work. It is necessary to first state what went wrong and then perform the parts that failed manually, at daytime. Alternatively, the test scripts have to be modified, and then performed automatically or manually, if the failure is due to a defect in the scripts or a change in the AUT. The fact that the feature cannot be tested with 100% accuracy also involves more work, since the results are not detailed and reliable enough. All the extra work makes the time to execute the Daily Test automatically highly comparable with the time it takes to perform the test manually. 47

Test Automation at ABB Automation Technologies AB

With respect to the discussion in this section, the conclusion taken is that a profitable automation is not feasible. The theory in chapter 4 states, that automatic testing can be used on features, which have already been released, to confirm that they work and to detect defects that were introduced accidentally after the release. After a release of the Control Builder, that version is tested occasionally. Remaining defects are corrected and service packs are released. Testing only the released versions automatically would agree with the theory.

7.3

Interview and Questionnaire

For the evaluation, information is also needed about which factors that affect the cost when the Daily Test is performed automatically. The cost for manual testing is already known at ABB and is presented in section 7.4.1. To gather the information, it was decided that a questionnaire should be handed out to employees at the company. To prepare this quantitative research, a qualitative research was made by open interviews with ten employees. The purpose with the qualitative research was to reduce the number of questions of interest to ask in the questionnaire. The interviews gave an apprehension of the thoughts about test automation and the cost. It gave a better basis for asking the right kind of questions in the questionnaire, and for formulating them so that there would be no risk for misunderstandings. The questions were chosen to give information about the costs, opinions and knowledge about automation. Employees with different work experiences were interviewed and given questionnaires to cover as many aspects as possible. 22 questionnaires were handed out and 18 of them were answered. The questionnaire and all the answers can be found in Appendix E.

7.3.1

Summary of the Questionnaire

The questionnaire was handed out to 12 testers (11 answered), 6 software developers (3 answered), 2 quality engineers (2 answered) a team leader (answered), and a project manager (answered). Two of the questions asked for opinions about the positive things with automation in general and with an automation of the Daily Test. The answers given were that test automation: • • • • • • •

assists employees with monotony jobs can execute at night and have the result ready in the morning saves time is always performed the same way is repeatable increases the product quality finds defects earlier.

When it comes to the negative part about automation the opinions were that: •

less defects are found because the test is always performed the same way 48

- Implementation and Evaluation

• • • • • • • •

it gives less variation some parts are hard to automate work-arounds are not possible the cause of a problem can be difficult to understand it requires maintenance there will be additional sources of failures if the test fails, it will have difficulties continuing it will be difficult to find a good tool for automation.

In spite of the negative aspects mentioned above, most of the asked employees saw no or very few disadvantages with test automation. Most people had heard about WinRunner or seen it in action, very few had actually worked with it. Questions were also asked about which factors would influence the cost when building the Daily Test and performing it automatically. The following factors were mentioned in the answers: • • • • • • • • • •

time; building the tests time; maintaining the tests because of changes in the product time; looking for what caused the failure when the test was executed time; delay when parts of the test has to be performed manually licence costs development of the test procedures well-constructed tests bring down the costs since more defects are found. With fewer defects the product is more stable computers for testing and necessary hardware educating personnel to handle the test tool complexity of the test.

Those of the employees who have insight in the development of the Control Builder GUI stated that very little is changed between the different test versions of the Control Builder. Everyone asked but three were positive about an automation of the Daily Test, however some of them thought that only parts of it should be automated. They also thought that it would be profitable for ABB to automate the entire, or parts of, the Daily Test. Of the three who did not have sympathetic attitudes towards automation, two were unsure and one person thought that the product is too unstable to be tested automatically, and therefore too much time will be spent modifying the test.

7.3.2

Conclusion

The main reason of the questionnaire was, as mentioned earlier, to acquire better understanding of which factors would play a role in the automation cost of the Daily Test. The conclusion after reading the answers is that there is not much knowledge of which factors, special for ABB only, that affect the cost and how much maintenance of the Daily Test that will be required. The

49

Test Automation at ABB Automation Technologies AB

answers did not give any new information and the information about how much the GUI of the Control Builder changes was very vague. This indicates that the automation cost will be difficult to calculate. Calculating the cost when the test is performed manually is easy. Nevertheless, estimating how much maintenance the automated test will need, and how difficult it will be to find the defects in the test scripts and the product, is very difficult. To be able to make a correct cost estimation of the differences between manual testing and automatic testing, the following factors have to be considered: •

• • • •

How often the test will fail because of a problem, which is not related to a defect in the product, and how much time it will take to find the problem and to correct it. The problem can be a change in the GUI of the Control Builder, a failure in WinRunner, a failure in Windows etc. How often the test will go wrong because of a failure in the Control Builder and how much time it will take to find it. How much of the test will not be able to execute automatically, because of the reasons stated above. How much time it will take to localise the problem, correct the scripts if necessary, and to execute the rest of the test manually, or automatically if possible. If the test will be executed more than one time per version. In that case, it is interesting to know how many new failures that will appear if the test is executed a second or third time on the same version of the Control Builder.

Most of these factors were mentioned in the answers of the questionnaire, but no one could estimate their magnitude. What is even more difficult to calculate is: • • •

If defects in the product will be discovered earlier than if the test was executed manually, and how much the cost will decrease by this potential earlier knowledge. If more or less defects will be found when the test is automated, and the human factor eliminated, and what gain or cost this will generate. If the time and money saved when performing the test unattended will compensate and exceed the extra costs involved. These include maintaining the test, tracing the defects and performing some part manually when needed.

The knowledge about automation in general and its disadvantages were little. Almost everyone greatly underestimated the maintenance factor. Many mentioned that time would be saved if the test is performed automatically at night. Solely one person mentioned that test automation only could be performed successfully if the product is stable. Several simply thought that automation in general is a good thing because it makes work easier and saves time. They did not think of automation as a kind of software development and all work that includes.

50

- Implementation and Evaluation

7.4

Total Cost

According to the section 4.7, the total cost can be estimated by the following formulas: Total costman = cost of one session * number of sessions per release * number of releases Total costauto = cost to build + (cost to run one session * number of sessions per release + maintenance cost * number of releases) A new Control Builder is released to the customers approximately once a year. The calculations of total cost made in this section are therefore based on the time period of one year, thus on one release to the customers. With that taken into account when using the formulas above, the factor number of releases is 1 and the factor number of sessions per release corresponds to the number of sessions per year. The term maintenance cost includes locating the failure reason, correcting the scripts when the test fails due to changes in the GUIs of the Control Builder or the PPA, and finishing the test. It also includes creating a new Ghost when a new PPA version is available, and extending the scripts when the contents of the Daily Test changes. The total cost is, as mentioned above, based on one year. The term ghost is explained in section 6.2.2. The cost can be split into three parts: money, lead-time and effort. Lead-time is the time it takes to execute the test, i.e. the time the result has to be waited for, and effort is the time an employee spends to execute the test. If the entire test is performed automatically the time it takes is called lead-time, since no tester spends time on it. However, if the entire test is performed automatically at night the lead-time is zero. If parts of the test are executed manually, the time spent by employees is called effort. Hence, the effort is often lower than the lead-time. If the entire test is performed manually the lead-time is equal to the effort. In the calculations in this section the cost is expressed in money and lead-time. The only cost in money for running the test automatically is the electricity bill and the cost for the used hardware since the test is performed unattended. These costs exist irrespective of if the test is performed automatically or manually. Therefore they do not have to be included in the calculations of the total cost at all. The corrections and executions needed when the test fails is counted as maintenance. The factor cost to run one session is then equal to zero and the formula can be rewritten as follows: Total costauto, money = cost to build + maintenance cost For the cost in terms of lead-time the factor cost to run one session is the cost measured in lead-time, and can be called time to run one session. The factor maintenance cost can be called maintenance time and cost to build is changed to time to build. With those changes the formula is: Total costauto, time = time to build + time to run one session * number of sessions per release + maintenance time The lead-time and thereby the term time to run one session in the formula above is equal to zero if the automatic test is performed successfully at night. Time for checking the result could be included here, but it is instead included in maintenance time since it is difficult to separate

51

Test Automation at ABB Automation Technologies AB

checking the result from looking for what caused the failure and maybe correcting it, which is maintenance. To perform the total cost calculations some estimated values had to be used. The following estimated values are used in the calculations made throughout the Total Cost section: •

• • •

The time to execute the test automatically is equal to the time that manual execution takes, i.e. approximately four hours. Extensions of the Daily Test will be made, and have been made during this master thesis, and will increase the execution time. This is not taken into account in this thesis; the execution time used in the estimations is four hours. It takes about 15 minutes to ghost the test computer, and the installations and the system extension take 1.5 hours. Backup and restore takes about 1.5 hours, which leaves 45 minutes for the rest of the test; the middle part. The cost for an employee is 700 SEK/hour, including among other things payment and office space. A new version of the PPA is released about once every second week, which requires a new ghost to be made. That will take approximately 2.5 hours and is done manually. The Daily Test is executed every workday, 50 weeks per year and failures occur in about 80% of the test executions.

Furthermore it is assumed that the automatic Daily Test is performed once every night, except for the executions at daytime as the scripts are corrected.

7.4.1

Cost Estimation for Manual Performance

The total cost for manual testing is relatively easy to estimate since it does not vary. A new version of the Control Builder is available for testing generally every day. Every version takes about four hours to test. One employee, who costs ABB approximately 700 SEK per hour, performs the test. The cost per day, i.e. per session, is on an average: Lead-time: 4 hours/session Money: 700 SEK/hour * 4 hours/session = 2800 SEK/session One new version of the Control Builder is, as mentioned, available for testing, and tested, every day. The cost per year, and hence per release, for manual performance is on an average: Lead-time: 1 session/day * 4 hours/session * 5 days/week * 50 weeks/year = 1000 hours/year Money: 1 session/day * 2800 SEK/session * 5 days/week * 50 weeks/year = 700 000 SEK/year

7.4.2 7.4.2.1

Cost Estimation for Automatic Performance Cost to Build

The cost to build the automatic test includes the amount of money we were paid for this master thesis, the employment tax and the time spent building it, both by us and by employees. We worked approximately 40 hours each per week during 20 weeks, and were paid 50 000 SEK each. The employment tax is 32,7 % of the gross amount [SKATTEV 04]. The employees assisted with 52

- Implementation and Evaluation

approximately 200 hours of help. The employees cost approximately 700 SEK per hour, in which employment tax is included. The cost for building the automatic test is: Lead-time: 2 * 20 weeks * 40 hours/week + 200 hours = 1800 hours Money: Employment tax: 0.327 * 2 * 50 000 SEK = 32 700 SEK Master thesis: 2 * 50 000 SEK + 32 700 SEK = 132 700 SEK Help from employees: 200 hours * 700 SEK/hour = 140 000 SEK Total: 132 000 SEK + 140 000 SEK = 272 000 SEK

7.4.2.2

Maintenance Cost

The maintenance cost includes the time it takes to locate where in the test the execution failed, to exactly state what is wrong, perhaps to correct defects in the scripts, the time to finish the test, the time to extend the test and the time to make new ghosts. In this master thesis all these actions, and all executions in connection with the correction, are counted as maintenance. How much maintenance will cost is the most difficult part of the total cost to estimate. According to section 2.4.2, most factors, such as time for impact analysis or the average time to implement a change request, just indicate increase or decrease in maintainability. Whether or not maintenance is increasing or decreasing over a period of time is hard to decide, since the Daily Test only has been made stable for one test version of the AUT. In the beginning of creating the test scripts, a couple of different test versions were investigated and it was established that several parts of the Daily Test had to be updated, to be able to work with the new versions of the AUT. It is possible that maintenance start out on a high level and then decreases somewhat, when the usual problems are known and perhaps corrected. This includes e.g. handling of the usual pop-up windows, such as virus scan windows. Most failures, especially the failures that the new functionality added to the AUT introduces, cannot be predicted. After a while, maintenance will probably increase again though, since the maintenance need increases with the age of the test scripts as mentioned in section 2.4.1, and since the Daily Test is extended with regular intervals. Depending on what goes wrong, different courses of action can be applied. If the failure is due to a failure in Windows, or a Microsoft product used, it is possible that the test can simply be restarted and executed successfully. If a failure occurs because of a defect in the Control Builder or in the PPA, the rest of the test can partly be executed manually. Since the Daily Test script is divided into three parts, it is possible that parts of the test after a failure first occurred can be executed automatically. Hence, the rest of the test includes the parts that have not been executed because of the failure. The failure can also depend on changes made in the Control Builder or in the PPA. Then the rest of the test has to either be performed manually, or the test script modified and the test restarted. It is also possible that the failure is of such nature, that none of the actions mentioned above is taken, and that the test is not executed until the next version is available the next day. Another alternative is that the test is both finished manually, the test script corrected and executed again automatically. Depending on the reason for the failure, the maintenance takes different amounts of time. The statements in section 7.3.2 present in great detail what needs to be considered to make a cost estimation. Since they are difficult to answer, and since it cannot be predicted neither which 53

Test Automation at ABB Automation Technologies AB

courses of action that will be taken nor what will cause the test failures, different scenarios are calculated. One of the courses of action mentioned above is to perform the rest of the test manually. Performing the rest of the test manually takes different amounts of time depending on what goes wrong and where. The worst case is that the execution fails in the beginning and that the entire test has to be performed manually. The best case is when the failure occurs at the end of the test, thus only the restoring has to be performed manually. The average case is somewhere in between. The later the failure occurs, the lesser has to be performed manually and the smaller is the probability of finding more defects in the rest of the test. Another solution is to execute the rest of the test automatically. The time to find the defect, to correct the scripts and to execute the rest of the test automatically is also dependent of what goes wrong and where. The cause of the failure affects the time it takes to correct the scripts substantially, but where in the test scripts the failure occurs has also great impact. If the failure occurs early in the execution, the probability of finding more defects when the test is re-executed is much greater. Every time a failure is due to changes that require modifications in the scripts, the scripts need to be corrected, and the test executed partly or entirely. This is very time demanding since it is possible that the test has to be corrected and re-executed several times. The time to correct the script is very difficult to estimate. It can take 10 minutes, but, unfortunately, it can also take days. The occasions when nothing is done after a failure bring no extra time or costs. Which course of action that will be taken depends on the cause of the failure, as discussed above. The scenarios are calculated based on correction time, i.e. how long time it takes to trace and correct a defect. The correction time varies a lot depending on the defect. Calculations of the cost are made in four different scenarios. The times chosen are based on the experience of how much time it took to correct defects during this master thesis. They are one hour, one workday, three workdays and seven workdays. Generally, a correction time between one and three workdays is the most likely one, since one failure often reveals more than one defect. Scenario 1a Lead-time: 1 hour Money: 1 hour * 700 SEK/hour = 700 SEK Scenario 2a Lead-time: 1 workday = 8 hours Money: 8 hours * 700 SEK/hour = 5 600 SEK Scenario 3a Lead-time: 3 workdays = 24 hours Money: 24 hours * 700 SEK/hour = 16 800 SEK Scenario 4a Lead-time: 7 workdays = 56 hours Money: 56 hours * 700 SEK/hour = 39 200 SEK 54

- Implementation and Evaluation

In spite of the correction of defects, the test is still executed every night, and will fail at the same point until the defect is completely corrected. That is if no new defects are introduced that make the test fail earlier. The times and costs in the scenarios 1a-4a include tracing and correcting the defects in the WinRunner scripts. Additional to correcting the defects, it is possible that the rest of the test is executed either manually or, if it is possible, automatically. If that is the case, more time and cost has to be added to the calculations in the scenarios above. Performing the rest of the test takes different amounts of time depending on how much of the test that is included, and how much of the rest that can be executed. It can take up to four hours. The time spent on execution of the rest of the test is also considered as maintenance. If two hours are added every day the time and cost in the scenarios above will be: Scenario 1b Lead-time: 3 hours Money: 3 hours * 700 SEK/hour = 2 100 SEK Scenario 2b Lead-time: 10 hours Money: 10 hours * 700 SEK/hour = 7 000 SEK Scenario 3b Lead-time: 26 hours Money: 26 hours * 700 SEK/hour = 18 200 SEK Scenario 4b Lead-time: 58 hours Money: 58 hours * 700 SEK/hour = 40 600 SEK The calculations in the scenarios above are based on the correction time of only one defect and the additional execution time. It is possible that more defects are revealed in that version of the Control Builder after correcting the first one. When this is the case, the maintenance time increases further. The scenarios give an apprehension about how much maintenance that can be needed when a failure is detected, and to what extent maintenance increases the total cost. A factor that has not been taken into account, but which affects the total cost, is the cost to educate personnel to be able to handle the test tool.

7.4.3

Conclusion

The outcome of the Daily Test execution is strongly connected to the development stage of the Control Builder. The test fails at a high rate during the development phase and has a high probability to success during the finishing test phase. Both phases have duration of about six 55

Test Automation at ABB Automation Technologies AB

months. Still, in average something in the test fails in 80% of the manual executions according to Stefan Calling, the team leader in the Titan project. The failures depend on defects in the Control Builder or in the PPA, when the test is performed manually. This failure rate alone implies that automation of the Daily Test cannot be successful. When the test is performed automatically in this case more factors, such as WinRunner and Excel, can cause failures and the risk for failures increases. These factors cause failures almost every time the Daily Test is performed automatically. This means that the risk that something goes wrong is reaching 100%. The consequence is that the test will fail, and that maintenance will be needed, almost every day. How much maintenance can be spent on the Daily Test for it still to be profitable? Performing the test automatically, i.e. unattended at night, costs nothing in terms of neither money nor leadtime, as long as the test succeeds at night. Manual execution takes four hours of lead-time and costs 2 800 SEK. That leads to the conclusion that as long as the test scripts take less than four hours per day to maintain, it is profitable, in terms of lead-time, to execute the Daily Test automatically. The time to execute the test at night is excluded, since this does not increase the lead-time. In a year and money perspective, automation is profitable if the total cost does not reach 700 000 SEK, which is the cost per year for the manual execution. The cost to build the test is, according to the calculations, 272 000 SEK. That means that the maintenance cost has to stay below 428 000 SEK (700 000 SEK - 272 000 SEK). Thus, it must take less than 612 hours of lead-time (428 000 SEK / 700 SEK/hour). Maintenance includes, apart from tracing and correcting defects, executing the test as the defects are corrected and extending the scripts. The latter is necessary when the Daily Test is extended to test more functionality. Moreover, maintenance also involves making a new ghost, approximately every second week. Since the calculations are based on a period of only one year, the impact of the cost to build the test is greater than if the calculations where made for a period of several years. The approach in this master thesis is however justifiable since the changes to the Daily Test, made at regular intervals, increase the maintenance time and cost substantially. It is therefore likely that the total cost would be roughly the same per year even when calculated over a period of several years. It is stated above that the maintenance has to stay below 612 hours per year. For the time to correct the defects to stay below 612 hours in one year, the maintenance time per day has to stay below 2.5 hours (612 hours/year / 250 workdays/year). As can be seen in the scenarios above, the maintenance often takes more than 2.5 hours per day. The time and cost to educate personnel also affects the total cost. This results in that the time allowed for maintenance decreases further below 2.5 hours per day. Taken into account that the extension of the Daily Test, as well as making new ghosts, must also be included in the maintenance time, the maintenance will not stay below 2.5 hours per day. Thus, the conclusion is taken that an automation of the Daily Test is not profitable when considering the money part of the cost. When it comes to lead-time, it is stated above it that the test scripts have to take less than four hours per day to for the automation to be profitable. Since the maintenance usually takes longer than that, there is no profit in lead-time.

56

- Implementation and Evaluation

The conclusion agrees with the conclusion drawn in section 7.2. Considering the section 7.1 with the Daily Test in mind, the disadvantages with high building costs and maintenance costs have a greater impact than the positive aspects of automation. The conclusion might have been different if the automation would not have increased the risk for failures, i.e. if the failures only were due to failures in the Control Builder and the PPA, as they are with manual testing. The automatic execution would still fail in 80% of the executions at night, which would require manual executions of the rest of the test at daytime. Nevertheless, the parts performed at night would save lead-time, effort and money. However, the assumption about an automation tool, which does not cause failures and does not need maintenance, is not realistic. Since WinRunner has been considered the best automation tool in two investigations, as mentioned in chapter 6.2.1, choosing another automation tool cannot be recommended. The most stable part when the test is performed automatically is creating the ghost, installations and the system extension. An alternative could be to only perform this part automatically. That would require little maintenance, and could save almost two hours every day. This would thereby decrease the total cost per day with 1 400 SEK. Another possible time and money saving alternative is to perform the Daily Test automatically only on the released versions of the Control Builder as discussed in section 7.2. It could also be an alternative to perform the entire test automatically during the test phase, when the test succeeds at a much higher rate. The test phase duration is about six months. The calculations below are based on one year, where the Daily Test is performed manually half of the year and automatically the other half. Cost without maintenance: 700 000 / 2 SEK/year + 272 000 SEK/year = 622 000 SEK/year Maximum cost for maintenance: 700 000 SEK/year - 622 000 SEK/year = 78 000 SEK/year Workdays per 6 months: 250 workdays per year / 2 = 125 days/year Maximum maintenance cost per day: 78 000 SEK/year / 125 days/year = 624 SEK/day Maximum maintenance time per day: 624 SEK/day / 700 SEK/hour ≈ 1 hour/day Applying this alternative would, according to the calculations, be profitable as long as the maintenance time stays under one hour per day during the test phase. In summary, the experiences with this master thesis lead to the conclusions that: • Automation of the entire Daily Test, is not profitable since the product to be tested is too unstable, i.e. it changes too much between the different test versions. Automation of the Daily Test saves neither money nor large lead-time. The need for maintenance is too massive. • WinRunner and Excel are very unstable and cause a lot of failures. However, choosing another automation tool would probably not stabilise the automatic test. The experiences also lead to the conclusions that: • Performing only the ghost, installations and system extension automatically could save lead-time and money. • Performing the test automatically only with released versions of the Control Builder could also be profitable. • Combining manual and automatic execution depending on the development phase, as was also suggested, would be profitable if the maintenance takes less than 1 hour per day. 57

Test Automation at ABB Automation Technologies AB

No definite conclusion is made on these three alternatives, since investigating them is outside the scope of this master thesis.

58

- Implementation and Evaluation

8 Conclusions 8.1

Summary

One objective of this master thesis is to automate a daily regression test, called the Daily Test, at ABB Automation Technologies AB. Additionally the test result must be presented in an understandable and useful way. Another objective is to perform an evaluation concerning whether time and money is lost or gained by performing the test automatically. Test scripts designed to perform the test automatically were implemented mostly by using WinRunner and by rewriting the scripts manually to make them function properly. The test scripts cover the original test completely. Considering the circumstances with changes in the program to be tested, and that the other programs involved in the test can be described as rather unstable, the automated test works as good as can be expected. The result of the test is accurately documented in two Excel sheets, one for detailed results and one for a more comprehensive overview. Making the automation to always work flawlessly is considered impossible and is outside the scope of this thesis. The objective to automate the Daily Test and to present the test results is considered accomplished. The matter that it is next to impossible to know how much maintenance the automated test is going to need, complicated the evaluation. Since the maintenance time is a factor of much variation several possible scenarios were evaluated. The conclusion taken subsequently was that performing the complete Daily Test automatically is not profitable. Performing only parts of the test automatically could be profitable, as well as testing solely certain versions of the product automatically. Chapter 7 includes thorough discussions on whether to use the automation or not, and gives ABB the knowledge to make a profitable decision concerning the future of automation for this test. Consequently, the evaluation part of the objective is also considered accomplished.

8.2

Conclusion

Automation of the entire Daily Test, is not profitable because: • •

The product to be tested is too unstable; it contains too many defects and its features are changed too often. Therefore the need for maintenance is too massive and neither money nor large lead-time is saved. WinRunner and Excel are very unstable and cause a lot of failures. However, choosing another automation tool would probably not stabilise the automatic test since WinRunner is rated as one of the best tools in the market in several independent evaluations.

The automation of the Daily Test could be profitable if: • •

59

Only the ghost, installations and system extension are performed automatically. The Daily Test is only performed automatically with released versions of the Control Builder.

Test Automation at ABB Automation Technologies AB

•

The Daily Test is performed automatically during the test phase, when the test succeeds at a much higher rate, and manually the other half.

No definite conclusion is made on these three alternatives, since investigating them is outside the scope of this master thesis.

8.3

Future work

In chapter 7, it is stated that performing the complete Daily Test automatically is not profitable. Care should therefore be taken to avoid any larger expenses put on extending the automation of the Daily Test until the decision of whether to perform the Daily Test automatically, partly or entirely, or not is definitely taken. When taking the decision of whether to continue with the automation of the Daily Test or not, the calculations and discussions in section 7.4 should be taken into careful consideration. It was outside the scope of this thesis to adapt the test script to every new version of the Control Builder; thus it is not certain exactly how much maintenance the test is going to need. If the decision is to continue the automation, it is suggested to perform the test automatically for a couple of weeks without extending it, only to make sure the right decision was made. If the decision then turns out to be wrong and the automation is abandoned, the only additional time spent on the automation is for regular maintenance when adapting the test. If the decision is right and turn out to be profitable, the following actions can be taken without any losses: •

As mentioned in section 1.5, the Daily Test has been extended and changed but the automatic Daily Test has not. If ABB decides to perform the entire Daily Test automatically, updating the test script with these changes has to be done.

•

Concerning the installation of the PPA, ABB is currently working on a way to install it automatically. If they succeed and their implementation is stable, their work can be added to the Daily Test and the maintenance costs will thereby decrease.

•

It is important to verify whether or not the connection between the test computer and the I/O is lost when deleting the corresponding I/O in the Control Builder. To accomplish this, the controller has to be rebuilt, as mentioned in section 1.5.

60

- Implementation and Evaluation

9 References [ADELSWÄRD 98]

Kristina Adelswärd Bruck, Patrik Rosenquist, Automated Testing of Software with Graphical User Interfaces, Master Thesis, Department of Communication Systems, Lund Institute of Technology, 1998

[BELZ 01]

Frank-Martin Belz, Theory and practise of case study research, http://www.oikosstiftung.unisg.ch/academy2001/Presentation_Belz1.pd f , August 2001, 28 April 2004

[FEWSTER 99]

Mark Fewster, Dorothy Graham, Software Test Automation, Effective use of test execution tools, AddisonWesley, 1999

[GASCOYNE 00]

Stuart Gascoyne, “Productivity improvements in software testing”, Electronic Engineering, vol. 72, no. 885, pp. 65-67, 2000

[HAGEN 98]

Gail Hagen, “System Testing”, Mortgage Banking, vol. 58, no. 11, pp. 97-99, 1998

[HEISER 97]

John E. Heiser, “An Overview of Software Testing”, JEH Consulting Inc., 1997

[IEEE]

IEEE, IEEE Standard Glossary of Software Engineering Terminilogy, std 610.12-1990, IEEE, 1990

[JONSSON 02]

Clarens Jonsson, Erik Danielsson, AIP Automated Testing Concept, Experiences and Realization, Project Mjölner ABB, 2002

[LANTZ 93]

Annika Lantz, Intervjumetodik, Studentlitteratur, 1993

[LÖFQUIST 04]

Anette Löfquist, Anna Wickholm, Improving Software Testing in Small Projects, Master Thesis, Department of Communication Systems, Lund Institute of Technology, 2004

[MOSLEY 02]

Daniel J. Mosley, Bruce A. Posley, Just Enough Software Test Automation, Prentice Hall, 2002

[MYERS 97]

Michael D. Myers, Qualitative Research in Information Systems, http://www.qual.auckland.ac.nz/ , 1997, 28 April 2004

61

Test Automation at ABB Automation Technologies AB

[POL 02]

Martin Pol, Ruud Teunissen, Erik van Veenendaal, Software Testing, A guide to the Tmap Approach, Addison-Wesley, 2002

[RADVIS 02]

Undersökningsmetoder, Rådvis Tjänstekvalitet, 14 March 2002, 28 April 2004, http://www.radvis.nu/undersokningsmetoder.htm

[RAKITIN 01]

Steven R. Rakitin, Software Verification and Validation for Practioners and Managers, Artech House, 2001

[SCHULMEYER 98]

G. Gordon Schulmeyer, James I. McManus, Handbook Of Software Quality Assurance, 3rd edition, Prentice Hall, 1998

[SKATTEV 04]

Skatteverket, Arbetsgivaravgift – Inkomstår 2004, 23 August 2004, 25 August 2004, http://skatteverket.se/index.html

[SOMMERVILLE 95]

Ian Sommerville, Software Engineering, Addison Wesley, 1995

[WINRUN 00]

Mercury Interactive, WinRunner, 2000

62

- Implementation and Evaluation

Appendix A – Hardware and Software at ABB Controller ABB Automation Technologies AB builds controllers and develops software for them. The controllers are used in industries to control different processes, e.g. production processes. The controller can execute when firmware and software has been downloaded to it. This is done from the Control Builder. The firmware is basic software and serves as the controller's operating system. The software is a program written in the Control Builder. A controller can be connected to I/Os, which can be connected to variables in the program. Thereby the statuses of the variables are indicated by the I/Os.

Control Builder The Control Builder, is an engineering-tool. It is used to program and control controllers, CPUs. The programmed controllers can control processes by controlling I/Os. The firmware and software to the controller is downloaded from the Control Builder. The Control Builder is sold to different companies, and is therefore programmed in different ways depending on the customer's needs. Consequently, the software downloaded to the controller differs between customers. The software downloaded is included in a project, and therefore downloading software is actually downloading a project. The Control Builder communicates with the controllers over a network. It is started from the PPA.

PPA The PPA , Process Portal A, is used to supervise different industry processes. It is usually used in control rooms. Many different devices, e.g. tanks, can be connected to and supervised in the PPA. In the Daily Test, the PPA is mainly used to create control structures and projects, and to open projects in the Control Builder. A control structure can contain multiple projects but only one project at a time can be opened in the Control Builder. The Control Builder is started from the PPA when a project is created or opened. The programs that are written in the Control Builder are all part of projects. The GUIs of the PPA and Control Builder both contain one tree structure each. This tree structure looks exactly the same in the Control Builder as in the PPA. The structure is created when the project is created. In the Daily Test, the mirroring between the PPA and the Control Builder is checked, i.e. that the tree structures look the same.

63

Test Automation at ABB Automation Technologies AB

OpenIF OpenIF is the interface between the PPA and the Control Builder. It can be used to control the Control Builder via commands instead of clicking in the GUI of the Control Builder. OpenIF is not used in the automation since it was considered to lack some functionality and since many functions, made with WinRunner, already existed.

Batch Builder The Batch Builder uses source code from a database, supervises the compilation of it and then packs the compiled files to a product. The generated products can be the Control Builder, the PPA etc. The jobs, which are run by the Batch Builder are scripts performing various things. Developers can follow the actions in the Batch Builder using a web interface. Except watching what happens in the Batch Builder, the web interface can be used to add new jobs, remove jobs, execute jobs, choose if the jobs should be executed at night, view and edit the scripts of the existing jobs etc. A job that automatically ghosts the test computer, which performs the Daily Test at night, has been added to the Batch Builder.

Visual Studio Visual Studio is the program that compiles the source code to the products that are to be built by the Batch Builder. The Batch Builder orders the compilation. Visual Studio is also used to write GUIs. The programming language used is called Visual Basic. The GUI-files are called Visual Basic scripts.

OPC-server and OPC-client An OPC-server is used to read the values of the variables in the controller during the test. An OPC-client fetches the values from the OPC-server and presents them. In this case the PPA acts as an OPC-client and presents the fetched values in a GUI written in Visual Basic. The GUI is situated in the PPA. The WinRunner script uses these values to check if they are correct compared to the values sent to the controller by the Control Builder. The values are also used to check if the cold restart and warm restart work properly, i.e. if the values of the cold retain and retain variables are kept or not. The variable control is made after the following three parts of the Daily Test: Write a program, download project to controller; Change software, download project to controller; Warm and cold restart. The complete content of the Daily Test can be seen in section 6.1.

64

- Implementation and Evaluation

Appendix B – Test Tool Evaluation ABB Automation Technologies AB in Västerås has evaluated the test tools on the market. The selected vendors were Mercury Interactive, Rational, Compuware and Segue. After the evaluation, the result was, at least from a theoretical point of view, that all tools support the same functionality. The differences were more related to how, and to what extent, this functionality was supported. Segue have its main focus on web application, and therefore it was soon counted out. Compuware was soon also discharged since its approach to support several different platforms had been done on the expense of functionality provided. Left were Rational and Mercury, which are the two largest vendors on the market. They have been used for an extended hands-on trial and evaluation. The evaluation was limited to test management tools and GUI test tools, i.e. Rational Test Manager, Rational Robot, Mercury Interactive TestDirector and Mercury Interactive WinRunner. The result is presented in the matrix below [JONSSON 02]. The conclusion was that the two tools basically provide the same functionality, but in different ways. Rational Robot was more intuitive to use, while WinRunner provided a large number of wizards. The major difference between the tools distilled down to the price tag. The difference in cost originates from their provided solutions for distributed execution. Tests developed in WinRunner must also be executed in WinRunner, as WinRunner must interpret the scripts during execution. This implies that each test client in the distributed network needs to install WinRunner, which is not a problem since floating licence is available. However, in order to perform parallel executions, one needs to access as many WinRunner licenses as the number of parallel executions. In Rational all developed test scripts are compiled. Therefore one only needs to install the agent that the test manager controls on each test client. Only one Rational Robot is needed and as many virtual user licenses as test cases running in parallel. Automation feature Test script language

Environment for development of test scripts Interface for execution of test executables Organisation of tests in test plans, test suits and test cases Local and remote test execution

65

Comments with respect to each vendor Mercury Rational Mercury Interactive proprietary SQABasic a Rational proprietary Test Script Language, TSL. Basic-like scripting language. VB Script, Java Script. MS VBScript, MS JavaScript, Support for command line PearlScript and PythonScript execution. Support for through the VAPI-XP testing tool. implementation of any script interpreter interface Supported Supported Test Executables can be launched within WinRunner. Supported Distributed tests are executed (as interpreted code) in a local installation of WinRunner at each node.

Test executables can be launched within test scripts or through the command line interface. Supported Distributed tests are executed (as binary code) through a test agent installed at each node.

Test Automation at ABB Automation Technologies AB

Central test repository Reporting functionality

Supported Reporting functionality supported in TSL. Mercury also provides open interfaces for interaction with TestDirector.

Version control of tests

Integration with version control systems supported. Not a part of the functionality provided by testing tools. Supported

Automated test execution Object recognition

Maintainability of test environment Long-term cost Market share Ease of use

Supported for VB, ActiveX and web testing through specific Addins. Support for object identification tool. Depends on how the tools are used. Larger than Rational. Largest market vendor for test tools. Yes.

Supported Reporting functionality built-in in SQABasic, e.g. verification points. Rational also provides open interfaces for interaction with the Test manager. Integration with version control systems supported. Not a part of the functionality provided by testing tools. Supported Support for ActiveX through manual addition of ActiveX test container. Support for object identification tool. Depends on how the tools are used. Less than Mercury. Largest market vendor for tools over the development and test life cycles. Yes.

Table B.1. Feature matrix.

66

- Implementation and Evaluation

Appendix C - WinRunner

There are three main stages in testing with WinRunner: creating tests, running tests and analysing test results. The three stages are explained in the following sections [WINRUN 00].

Creating tests Tests can be created by recording or programming. The two techniques can be combined. With recording, WinRunner records all operations that are made on GUI-objects in different applications. It does so by generating Test Script Language, TSL, in a corresponding script. TSL is very similar to C, which is a more known programming language. To create a more powerful test with logic, as statements and loops, it is necessary to use programming, partly or entirely. During recording mode WinRunner also identifies every GUI-object that is selected and adds them to the GUI-map. Examples of GUI objects are: windows, dialog-boxes, lists, buttons and edit-fields. Examples on the TSL, which WinRunner generates, can be seen in the three examples below. Example 1 The menu “View” and “Expand all” is selected in the Control Builder.

# Control Builder win_activate ("Control Builder"); set_window ("Control Builder", 4); menu_select_item ("View;Expand All");

Example 2 The button “Install Software” and “Control Builder M Professional” are clicked upon with the left mouse button. # Control IT setup set_window ("Control IT setup", 3); obj_mouse_click ("Install Software", 62, 14, LEFT); obj_mouse_click ("Control Builder M Professional", 99, 10, LEFT);

Example 3 The button “Start” is pressed and then “Shut Down” is chosen from the start menu. Finally “Restart” is chosen and “OK” is pressed. # Shell_TrayWnd set_window ("Shell_TrayWnd", 2); button_press ("Start"); # Start Menu set_window ("Start Menu", 1); toolbar_button_press ("ToolbarWindow32", "Shut Down"); # Button Number 1;

67

Test Automation at ABB Automation Technologies AB # Shut Down Windows set_window ("Shut Down Windows", 2); list_select_item ("What do you want the computer to do?", "Restart"); # Item Number 2; button_press ("OK"); texit("");

The last line, texit(“”) , has been added manually, i.e. programmed. It stops the execution of the script. Checkpoints can be inserted to compare the current behaviour of the application with the expected behaviour. There are four different checkpoint types: GUI checkpoints, database checkpoints, text checkpoints and bitmap checkpoints. With a GUI-checkpoint it is possible to check the status of an object in the GUI, e.g. checking if a button is enabled. Database checkpoints are used to examine contents in a database. To check the text in GUI objects, text checkpoints are used. Eventually, bitmap checkpoints can compare two snapshots of a window. Data-driven tests can be used to check how the application responds when different sets of data are used. This can be done to assure that the same operations work correctly independently of the input. The different input values can be stored in an external data file, instead of changing the fixed values in the test application. WinRunner then reads different sets of data in the file every time the test is executed. When the recorded test is executed, WinRunner performs it automatically.

Running Tests WinRunner executes a test by interpreting the lines in the test script and performing the corresponding actions. It looks as if a human is using the mouse and pressing the buttons. WinRunner can run in three different modes: verify mode that checks the application, debug mode that debugs the test and update mode, which updates the expected results. More detailed information is found in [MERCURY 00]. Test results are saved and give detailed information of the events in the test execution, it is colour coded and marked as passed or failed.

Maintaining Tests The tested application, the AUT, changes during the development. For the test to be reusable on the different versions of the application, the test has to be maintained. WinRunner offers some tools useful to help maintaining the test. The GUI Map Editor makes it possible to quickly update the GUI-objects in the GUI-map. The objects are used to recognise objects in the tested application. As mentioned in section Creating Tests above, WinRunner creates a GUI-map during recording. The map contains all the objects that WinRunner has learned and maps them to logical names. The user can modify the logical names. The objects that appear in the same window in the application are grouped together, as leaves to a root item where the window is the root item [ADELSWÄRD 98]. Every GUI-object in the GUI-map is given a minimum set of properties or physical descriptions that uniquely identify the object. WinRunner uses a default set of properties to describe objects. The properties are based on which class, e.g. button, list or window, an object belongs to. If these 68

- Implementation and Evaluation

properties are not enough to identify an object uniquely WinRunner adds additional properties to the object. When the application changes, the objects in the GUI-map can be modified by using the GUI Map Editor. When a window name changes in the application the user has to change the logical name of the corresponding window object. It is wise to give it a general name that is not believed to change in the future, to prevent further changes on the same object. A general name can be given by choosing only a part of the old name. However, sometimes changes in the GUI-map involve manual changes in the test script as well. The Function Generator makes it is easier to add TSL-functions and user-defined functions to the test. This helps the user to quickly generate his or her favourite functions, and to insert them into the test script [MERCURY 00]. The tool can be used either by pointing to a GUI object, or by choosing a function from a list. When argument values to a function are specified the function can be executed from the Function Generator or from inside the test. There are also debugging tools that help to identify problems in the test. If WinRunner detects a problem as a syntax error or a defect in logic, the test is stopped. Step commands, breakpoints and the Watch List are tools that can help the user to isolate the problem. With step commands the user is able to run a single line at a time in his or her own pace. Breakpoints are used to pause a test at pre-determined points. The value of different variables, expressions and array elements can be viewed during a test run by using the Watch List [WINRUN 00]. Applications developed with e.g. Visual Basic, Java and Oracle can be tested in WinRunner.

69

Test Automation at ABB Automation Technologies AB

70

- Implementation and Evaluation

Appendix D – The Result Solution Every function used in the WinRunner scripts writes its result or status in an Excel file just prior to the end of the function. The result is written in a column named just as the function. Some functions are called more than once, and there are as many columns for the same function as the times the function is called from the test script. These columns are separated from each other by adding the order number in every column as can be seen in the figure below, for example CB_AddIOunit(1), CB_AddIOunit(2) etc. The figure shows solely three of the approximately 60 columns in the result file. Some of the functions write their result in another Excel-file as well. The difference between the files is that the first one is more detailed and shows the results from all the functions used. CB_AddIOunit(1) CB_AddIOunit(2) CB_ProjDownload(1) Correct Correct Correct Correct Correct Figure D.1. Results from three functions in one of the Excel-files.

The order of the columns is the same as the order in which the functions are called from the WinRunner script. The user can choose if WinRunner should stop the execution and give a failure message when a failure occurs, or if the execution should continue, ignoring the failure. To prevent WinRunner from stopping the test before the result is written in the Excel file, WinRunner's defect detection is disabled. This way, the result is written in the file even after a failure has occurred since WinRunner continues to execute. The file can then be presented as a web page; thereby the results are immediately available and presented in an understandable manner. In the Daily Test, the functions depend on each other. Generally, if a failure occurs in one function the next function cannot perform its task properly. The reason is that a function performs its work based on what should have been done in the functions before. For example a function cannot connect program variables to the I/Os if the function that should have written the program containing the variables, did not manage to do so. The variables do not exist and the function cannot continue because of the failure in another function. Another example is the function that creates a project in the PPA. If the function that creates a control structure fails to do so, a project cannot be created by the corresponding function. A few functions are not depending on the previous ones, and can proceed even though a failure has occurred. If the function which controls the mirroring between the PPA and the Control Builder discovers that it is not correct, the execution of the following function can still proceed since nothing depends on the mirroring being correct. Scrutinising the test script shows that it can be divided into three relatively independent parts. The first part includes the installations, system extension, creation of a project, adding I/Os, writing a program etc. The second part includes among other things the project upgrade and importing a library. Finally, the third part includes backup and restore. If one part fails, the test automatically starts to execute the next. Thereby at least parts of the test can be executed even though another part failed. Consequently the best alternative in most cases, is to stop the execution after the function that went wrong, and to proceed with the next part of the test. This is only possible with the defect 71

Test Automation at ABB Automation Technologies AB

detection in WinRunner disabled. Otherwise the execution of the test stops if a failure occurs, and does not proceed with the next part of the test. The functions that are not executed produce no result, and the corresponding columns in the result files are left blank. This can be seen in the figure below, where the function CB_ProjDownload failed and the test continued with the third part of the test. The third part begins with the function PPA_BackupSystem. CB_ProjDownload(5) CB_GoOffline(5) PPA_InsertLibrary PPA_BackupSystem PPA_DeleteSystem Not correct Correct Correct Figure D.2. Results from functions in one of the Excel-files. As can be seen CB_ProjDownload failed and the test continued with the third part, which begins with PPA_BackupSystem.

72

- Implementation and Evaluation

Appendix E – Questions and Answers This appendix contains the questionnaire handed out to employees at ABB and the answers received. The answers were analysed and then the questions and answers were translated into English.

Questions Questionnaire Automation of the Daily Test – evaluation The purpose of this questionnaire is to gather information for our master thesis. Our task is to automate the Daily Test and to evaluate if it is profitable for ABB to perform the test automatically. The automation is implemented by using a record and playback tool called WinRunner. We are grateful if you want to help us with the evaluation by answering the following questions. Please answer as detailed as possible. On the last page you can see what is included in the Daily Test, and also some general information about how we have planned the automation. If you have any questions, we will be glad to hear from you.

1. What is your post and what kind of assignments do you have? 2. Which benefits does automation give in general? 3. Which advantages are there with an automation of the Daily Test? 4. Which disadvantages does automation give in general? 5. Which disadvantages are there with an automation of the Daily Test? 6. How much experience do you have with WinRunner?

□ None □ I have heard about it □ I have seen it in action □ I have worked with it a little □ I have worked with it a lot 7. Which factors influence the cost when the Daily Test is automated? In what way and how much? Cost can be time, money etc. 8. Which factors influence the cost when the Daily Test is performed manually? In what way and how much? Cost can be time, money etc. 9. How much does the GUI/user interface change between the daily builds of the Control Builder? 73

Test Automation at ABB Automation Technologies AB

10. Do you think the Daily Test should be automated? Explain! 11. Do you think ABB will gain or lose by automating the Daily Test? What is gained and what is lost? 12. Do you have any other comments? Here the same information about the Daily Test as in section 6.1 was inserted along with the following information: General information about the work with the automation • • • •

The PPA is included in the ghost The computer is ghosted automatically The Batch Builder starts the test automatically at night. The rest of the Daily Test is made with WinRunner

74

- Implementation and Evaluation

Question

Answers Answers

1.

Product Integration Tester, Quality engineer.

Software product developer

I write tests and support those who perform tests; sometimes I perform tests myself.

Pit-tester, which involves performing tests, reading, writing or reviewing test specifications. I am a consult who usually writes programs for control system.

2.

Provides help with monotony jobs. Saving resources. The test procedure is exactly the same every time; which makes it easier to repeat the failures.

Great timesaving. Many tests can be performed repeatedly in a short period of time. The quality of the product increases.

All tests are executed the same way. No differences in the test results depending on the human factor. Test can be executed at nights and weekends, not depending on personnel.

3.

Providing help with monotony jobs like Ghosting the computer, product installations of PPA, Control Builder and other ABB family products and simple tests like creating simple projects and download them to the controllers. Resources will be saved if the installation can be performed without any human supervision after working hours. Every individual is manipulating the product in his/her way. Therefore more defects will be found in tests where humans are involved.

Fast feedback. We do not use the Daily Test. We perform automatic tests several times a day on the entire system.

We know that all the steps in the test specification are performed, and that they are performed in a correct manner. It can be cost-effective, we can also perform tests that take much time and that would not be possible to perform in another way, for example download, warm restart and hardware I/O tests. The advantage would be that the person performing the Daily Test today would be able to do something else. We would also save time if we had some apprehension of the nightly build already in the morning.

None.

Variation, the system can be used in many different ways. Automatic testing can not be performed in all areas. Further, the automatic test does not have knowledge whether the customers will accept a function and so on. It will be hard to give the developers feedback.

5.

Same as above.

None.

I think the quality will decrease, some parts are hard to make automatic.

6.

I have seen it in action.

I have heard about it.

I have seen it in action.

If there are failures in the application the test may stop. If the test would be performed manually, the failure may be worked around and the tester would able to continue the test. The testing can get stereotype, the product is tested the same way time after time. If there are different persons testing the product on “lose conditions” the probability to find more failures is greater than for automatic tests. Failures outside the automatic test slips through and may be undiscovered for a long time. Several different persons testing the product in different ways have a greater possibility of finding more defects than automatic tests. An automatic test has no imagination and performs the test in the same way every time. I have seen it in action.

7.

Installation of PPA and Control Builder and then creating system and adding AC800-extension takes about 2 hours.

Our automatic tests are so extensive that they cannot be performed by manual execution. If they were, we would not have time writing code.

Time decreases: + Quality decreases: Cost decreases: +

8.

Same as above.

See answer number 7.

I do not have insight in the Daily Test this way.

9.

Normally there are no changes. But we know today that there will be lots of changes in the product installation procedure. Parts of the Daily Test, like product installations and simple use of the products can be done. But I do not think auto test can replace a person when it comes to more advanced tests like visual interaction with the SW and HW products.

We use/develop PLCConnect, not the Control Builder.

Very little I hope.

Yes. We have automatic tests for PLCConnect and it has resulted in time-gain and quality improvement.

Yes, if it is possible. It is a very dull, monotonous regression test that cost a couple of thousand Swedish crowns a day.

ABB will save lot of recourses by performing monotony jobs and simple tasks automatically. But do not forget this depended on the product WINRUNNER’s requirement. Good luck!

See answer number 10.

See answer number 7.

If you would like to know more about how we have introduced and use automatic tests, please contact me.

Make a good master thesis report, because it can haunt you many years when you apply for a new job.

4.

10.

11.

12.

75

The product’s main functions are tested the same way every time. No failures depending on the human factor. The result of the product has a high quality.

It can be a problem verifying that the automatic test has executed with an approved result. Data must be logged and saved. One must be able to see if the test passed at a later time. This makes the test procedures complicated and expensive to prepare. Is the test good enough to be a quality document? It is obviously more expensive to manually test a product. The advantage is that the person fills in a test protocol, which later can be verified and approved. This can later be a quality document. I normally do not perform the Daily Test, but my understanding is that the user interface does not change much. Sometimes there are some larger changes though. I think one has to have a mix of automatic and manual tests. Automatic tests discover the failures that happen to be within the test procedure. If someone manually controls the function now and then, faults that lies outside of the test procedures could be found.

I think ABB will profit by having a mixture of automatic and manual tests. To only trust in automatic tests will decrease the quality of the product. To only trust in manual tests will also decrease the quality of the product. –

Test Automation at ABB Automation Technologies AB

Question

Answers

1.

Writing test specifications; Team Leader for integration tests CB – PPA autumn 2003.

I work as a test engineer, writing test specifications and performing tests. My title is quality engineer.

Quality issues. Quality engineer for the Program, Atlas Safety and Titan.

Test leader. Write test specifications.

2.

- Saves time when the test is performed. - Tests are performed the same way every time.

- repeatable - perfect test method to verify already developed functions - perfect test method for regression tests - can be executed around the clock

The tests are executed several times without test personnel and it takes less time to execute the tests. Tests can be started when the product is built. Tests can be performed at night.

- One knows exactly how the test is performed. - One can execute the tests at night. - We do not have to “waste” expensive engineers by having them doing time demanding manual work.

3.

The test is executed daily, much time is saved.

Can in a quick and effective way test the basic installation/configuration/down load/warm + cold restart functions

The test can be executed at night. If everything is automated, it can start directly after a build is finished and give very fast feedback.

4.

- The test cases require maintenance (WinRunner code) - The interpretation of what went wrong in the test may take time. - Continuing testing after a failure may be hard. - One has to explicitly check everything humans handle with common sense. If, for example, one sees text mixed in English and Swedish one reacts and thinks it is weird. An automatic test checks all text to see if it is genuine. It may be a lot to test if one wants to check all texts and every common sense situation.

Automation programs usually: - Demands more maintenance than one think. - Small changes in the product can cause much work maintaining the existing automation program. - Are hard to apply to newly developed functions.

The test can be executed when the nightly build is made. Everybody gets the result in the morning when they come to work. We have more products that are built; this takes more time if we do not test automatically. They must be maintained. It can be hard to understand the result of the test.

5.

Same as number 4.

Same as answer number 4.

Maintaining the test. It can be hard to understand the result of the test.

See above.

6.

I have seen it in action.

I have seen it in action.

I have seen it in action.

I have seen it in action.

7.

- “One-time-cost” to build a test environment. - Maintenance of test scripts etc. - Time for interpreting the test result. - Some parts of the test was not executed but it would have been possible to continue. This causes a delay, maybe some part has to be performed manually. - License costs for WinRunner etc. Time to perform the tests.

Factors: - Changes in the product in shape of newly developed functions. Costs: - Changes can cause much work with maintenance of the automated test. (Costs can be both time and money)

How much maintenance that is needed, more maintenance gives higher cost. How many daily tests that are executed, more daily tests give lower cost per test.

- The development of the test procedures. - Running-in period problem (It will probably not work 100% the first time..) - Maintenance of the test program. - WinRunner licence (?)

Same as answer number 7.

The cost for the person who performs the test.

9.

Small changes I think, but I do not know.

I do not know.

How many daily tests that can be executed, more daily tests give higher cost. I do not know.

10.

Yes. It is executed daily. The gain of tests automation is largest with tests that are performed often.

11.

I think we gain on automation. I think time/cost according to number 7 is lower than number 8.

Yes! 1) Most test cases in the Daily Test are rather stable. 2) The Daily Test only contains already developed functionality so the automation fits perfectly. ABB gains by an automation of the Daily Test. Above all, the test engineer does not have to be there to execute the test! He or she can execute other tests at the same time.

12.

–

–

8.

Yes. It can be executed as often as it is profitable.

Gains. Both because it takes less time to perform them than what it costs to maintain them, and because the tests can be executed at night. Then everybody knows the status of the nightly build when they come to work in the morning and do not get the build if it is poor. Time is then gained for both developers and testers. –

If a failure occurs the test stops. One looses the variation in the procedure. A certain operation can be executed in different ways, in sequences, different names etc. and it usually gives a nice result doing things in different ways.

Nothing, or very little. The GUI is not changed very often, possibly if a new function is added, and then the test has to be changed anyway. We have a person who can perform other tests. We have the test result already the following morning or earlier.

We gain a test resource, and probably the test will be executed faster.

I think it will be much fuss in the beginning, before everything is in its place and the problems have been worked around. It would be charming if one also could automate the publication of the test result on the web.

76

- Implementation and Evaluation

Question

Answers

1.

Integration tests between the PPA and the controller, test engineer.

Tester, Component Delivery.

Software development in the CB; Open IF, source code handling, Search & Navigation. Developer.

I am a tester who works in the large lab and executes most release tests.

2.

Timesaving, boring and monotonous tests, which risk that parts of the test are “forgotten”, can be automated (if it is possible). The tests are always performed the same way, returning failures are discovered early.

- The test is always performed the same way. - It can be executed at night. - Gives a quick answer to how the product is working. - Takes away boring and monotonous work.

It takes little or none workforce to execute the tests. The tests will be performed the same way every time. One will be able to perform the tests more often.

3.

With a few exceptions, all parts can be automated; the condition for the test will be the same over time. One does not have to spend so much time maintaining the test code. It can be hard to decide what the cause of the problem is and how the problem arose. In many cases functionality is added or changed, which causes the maintenance of the test code to take more time than the tests themselves.

Gives a quick answer on how the product’s basic functions work.

Hopefully raises the quality of the product.

- It can be performed in the day/at night, which gives more time for the tester to do other tests. - It is a monotonic test; it is always executed the same way. - It can be executed faster. The computer does not take coffee breaks. Same as answer number 2.

- If a failure occurs the test stops. - It takes time to implement. - Same test procedure as last test. No variation. - More sources of failures. WinRunner is a new source of failure. - Loses focus on the tests, concentrates more on getting the automatic test to work and completing the test. - Creates a large amount of versions of automatic tests, which is modified to comprehend specific failures.

Takes time to develop automatic tests. They must be maintained when the product changes. Hard to do work-around if something goes wrong. Hard to vary.

- It can be a disadvantage to have the tests executed the same way. Different testers perform in different ways, which makes the test wider. - Automatic tests do not discover changes that a human tester finds; thereby a test can pass even though there are failures. - Automatic tests must be updated every time a program/version changes.

5.

If a failure occurs it can take longer time to trace it than if a person sits by the computer and sees what is going on directly. Besides that I do not see any disadvantages.

Failure tracing can be harder when no one was there when the failure took place.

Same as answer number 4.

6.

I have worked with it a lot.

The Daily Test is first receiving a new build. The risk is that the build has a lot of defects in the basic functions. This probably interrupts the automatic test early and that a person is still needed to supervise, report failures and modify the automatic test so it can finish. I have heard about it.

I have heard about it.

I have seen it in action.

7.

The objective is to save time and thereby money, but the activity has to be organised and seen long-term. There is always a “bump” of maintenance in the beginning, which involves a large effort. One has to weigh this effort against possible gain of time in the future. With wellconstructed tests one gets a more stable product, which brings down the costs. The man-hours, which must be put on testing. The tests can only be executed during office hours. My understanding is that it changes relatively rarely.

- Service/maintenance of WinRunner and test programs. - Educating personnel to handle Winrunner - License for WinRunner. - Computer for testing and necessary hardware.

Constructing the automatic test – more time. Maintenance of the test – more time. Executing tests – less time. Test report – less time.

–

- Testers - Computer for testing and necessary hardware.

The larger the test is, the more working time is needed.

–

- New alternatives in the menu. - New dialogs for failures. - New library versions.

The basic functions change a little or never. If they are changed, it is about one time per month or less.

10.

Yes. The Daily Test can be expanded to include more, for example installation of different OS. Then a person can start several tests at once.

Yes! Se below.

11.

I think ABB can gain on it longterm; especially the experience of automatic tests received. ABB can maybe adapt the product to get more effective tests.

12.

Yes!

No. To make automatic tests to work as smooth as possible they should only be performed on stable builds. The test should then be executed to check that existing functions not have been affected by a correction. It should only be used on release versions of alpha, beta and RC. Otherwise the risk is high that too much time is spent on modifying the automatic test when the builds are bad. Loses. Too much time will be spent on modifying the test. There will be no quick response to the developers, who must make corrections in the build. Demands one person to supervise the test. What can be gained on the test is to have all installed and ready when one comes to work in the morning. This assumes that the build is built at night and not in the morning, which it is today. Automatic tests should only be performed on products that are stable. Only as a complement to the other tests before delivery to a customer.

The basic functions (open/download etc.) are probably very rarely changed. But new functions of course change menus and windows. Yes, it would save time if the test was performed at night and the result was ready in the morning.

4.

8. 9.

77

I think ABB will profit by automating the tests. They can be executed more often, and if a quick verification of a build is needed, the test will do it fast.

I think ABB will gain time if there is an effective way to make small changes in the automatic test, like changing version of what to be installed. The loss is the flexibility of a person performing the test. But the gain of not having several testers installing system if the build is not great.

It would have been nice to see a demonstration of what you have accomplished so far.

–

Test Automation at ABB Automation Technologies AB

Question

Answers

1.

I work with products, delivery, quality of builds and build-environmentissues.

- Testing early integration of the Safety controller. - Test leader

Developing the user interface. Developer (project leader, technical co-ordinator, CCB etc.)

Right now I work with the Daily Test. My position is project assistant (material administrator, purchaser, and tester, a little of everything.)

Project programming. Testing 800xA.

2.

The primary advantage would be to eliminate the human factor and to some extent release resources from monotonous work. Same as above + the test can be performed at night, directly after the products are built. - The understanding of the product possibly decreases. - One special person gets locked maintaining the test.

- It is possible to execute the basic tests in a quick and effective way. - The tests can be executed at night without personnel.

Lesser resources to execute the test.

Time saving, cost effective, repeatable.

That the test is the same every time, one knows what has been tested. It does not have to be supervised. (Saves time).

- Get a fast result of the test. - The test can be performed the same way every time the test is executed.

Finding defects earlier.

See no. 2.

See above.

Do not know. I have not worked with the Daily Test so I do not know what it means.

- Hard to continue if the test fails. - It can be hard to find out what went wrong. - The tests can be performed the same way every time/people work in different ways and the finished product is used by people. The product must be able to handle different ways of usage.

It is always the same test case executed.

After a while no one really knows what has been tested. (One runs the test, which says ok to what?)

It depends on what tool is used, for example WinRunner, then you must almost know from the beginning if the test works or not.

5.

It can require a great amount of maintenance of scripts and ghostimages. For example a new PPA version is released rather often.

- Hard to continue if the test fails. - Hard to find out what went wrong.

–

Failures outside of the test. Changes in WinRunner, when functions in the CB are changed. Someone has to check the logs from the automatic execution of the automatic test for all suspected defects. I guess will be time demanding. See no. 4.

If it goes wrong, it can take longer time before one knows what went wrong. Maybe one has to start all over to see what went wrong and where, and time is lost.

I do not know!

6.

I have heard about it.

I have seen it in action.

I have heard about it.

I have heard about it.

I have seen it in action.

7.

- Hardware (PC and other equipment). - Development of scripts - Maintenance of scripts and hardware.

- Time for making the test automated. Do not know time.

I have worked with it a little. –

That the test can be executed at night and be done in the morning. Saves time. It is performed the same way every time. (One knows what has been tested.)

Maintenance of WinRunner script could be a cost.

8.

- Hardware - Time to perform test - Maintenance of test specification and hardware.

- Time to perform the test, about 8 hours/test.

–

I, a daily tester, can do my main work tasks or other kinds of tests. (750kr/hour). Maybe two computers will not be needed (about 800 kr/month). Someone has to check the logs and divide the defects between the developers. See no. 7.

Small defects are expensive to find through manual work.

9.

I do not have a clue.

Not very much when a new version is tested but between different versions.

The menus change (new choices are added, old are taken away) Dialog boxes change. Sometimes the look of the editors is changed.

Not very often.

It takes more time. Manually one does not do exactly the same every time. (This can be good sometimes not to do the same, to find things no one has thought of could be done to the system.) Different. Sometimes it probably changes pretty much and then looks the same for a while.

10.

Definitely, more effective test and fewer monotonous tasks for testers.

- Yes. It is a basic test, which can be executed quickly when a new build arrives. Then the decision if further test shall be executed is based on the result.

Yes. The more the tests are executed, the sooner we will get rid of the defects.

I want to see it action before I can tell.

Saves time. Faster to find the problem. Same test every time.

Yes, because the test will be safer and easier to repeat.

11.

I guess the cost would be very high developing the automation. Possibly we can even the costs after a long time using automatic tests. No, go for it and good luck!

I think the cost will be +/-0 - Less time testing. + More time maintaining the automatic test. The advantage is quicker test results and less time testing bad builds.

I think we will lose short- term because it takes time and resources to make the Daily Test work properly.

I doubt an automatic daily test can be made, which does not decrease the quality.

Everything, which can be automated, is good, so time can be spent on more important tasks.

I think the cost of maintenance will be the same as the cost for the manual work.

If the test can be executed automatically directly after the daily build is built, I think one will get the greatest usage of the test.

–

–

-

-

3.

4.

12.

I work with support issues in our internal environment and some client issues. I install all Batch Builders and take care of our old environment. Automatic tests perform test repeatedly and are easier to compare. One can be surer of the tests being executed the same way every time.

I do not know.

78

Question

- Implementation and Evaluation

Answers

1.

Omega, team leader/developer/defect tracer

Project Manager in the Integration Project Titan

–

Developing engineer, team member.

Team leader for configuration and product build

2.

- Find defects, I think that it will be easier to find defects that were not looked for. - It is easier to make the tests cover more and be more systematically.

Fast feedback Low cost

2.1 - Some can be executed after working hours without personnel. 2.2 - Time critical tests, which cannot be performed manually. 2.3 – Sometimes it can be an advantage if a test is always performed in the same way (identical tests).

The test cases are repeatable. Verification of a correction can be done automatically when executing the test. The test can be executed at night.

Fast feedback and a secure environment that gives the benefit of making last minute changes without “fear”

3.

- Finding defects quickly.

Fast feedback Low cost

A quick way of confirming that the basic functions work, without personnel having to perform these monotonous tasks.

Cost and test coverage

4.

Someone has to work all the time making the test work. I think there is no one who can manage this and his or her regular assignments.

Hard to get it up and running Hard to find a good tool

- Sometimes it can be an advantage if a test is not performed the same way every time (and a certain defect can be missed every time). - It takes more time to change and modify a test (implementing the test, making sure the test works before it is put into use). A person is easily told how the test should be performed. That is it takes time to maintain the test.

Boring to manually test the same thing every day. The tests are performed exactly the same way every time. The test will be executed at night. Not flexible; one failure and the test cannot continue. Cannot handle work-around.

5.

It demands further resources; otherwise I can not see any disadvantages.

Hard to get it up and running Hard to find a good tool

I have heard about it.

None

Cannot handle workarounds. Messages, texts etc are hard to handle. Demands some extent of maintenance. I have heard about it.

Flexibility

6.

- If anything goes wrong, it is not possible to make a small workaround and then continue the test. - Maybe it is hard to report what and when the test went wrong, just knowing it went wrong. I have heard about it.

7.

The quality has a great value. In the beginning of a development project, I think there might be many “known” failures and that it then takes much energy finding out what caused them. Someone has to analyse the failures though.

Number of test cases Complexity Product quality Person skill

Positive: Test can be executed during nonworking hours and without personnel. Negative: Maintenance of the tests. If one part of the test goes wrong, maybe the rest of the test is interrupted.

?

Test coverage Complexity of test.

8.

The quality of the testing probably depends on the tester to a large extent. More testers are probably needed in the manual case (at least if the automatic test is working correctly)

Number of test cases Product quality Person skill

?

Man-hours Time of test. Are people waiting for a test result because the tester doesn’t work night times?

9.

I do not know.

?

?

10.

Yes, finding defects more quickly and more systematically.

Do not know Not very much YES We would like to make more test daily and the only way is to make them automatic It is costly and hard to find a person to test this every day

Positive: If one part of the test goes wrong, one maybe can do a workaround to be able to continue with another part of the test at that point. Negative: The human factor; one happens to test a little careless or wrong sometimes. That can mean unnecessary and false failure investigations. I do not know, probably very little. See answer number 3.

I want a part of the Daily Test to be automatic (Executed at night and ready in the morning) I want a part of the Daily Test to be manual. (More flexible, can use workarounds.)

If we can make the test stable and reliable - yes.

11.

In the long run ABB will gain on this. Less test resources and more systematic and fast testing.

Gains on number 2.1, 3 Looses on number 4, 5

?

We will gain in product quality

12.

- Those who work with the automatic test probably should be “visible”, and spread their message. For example on TL-meetings etc. - They probably also have to work close, or be developers.

Good luck!

No.

I think that many of your questions are ones that ABB want you answer. Isn’t this what the theses is all about?

79

ABB will save time and money automating the test. Time, when we get the result of the test Time, the ability to extend the number of tests without losing much time testing every day. NO

Maintenance

I have seen it in action.

Test Automation at ABB Automation Technologies AB

80