Assessing Web Service Quality of Service with Fault Injection Nik Looker, Dept of Computer Science, University of Durham, UK

Malcolm Munro, Dept of Computer Science, University of Durham, UK

Jie Xu School of Computing University of Leeds, UK

[email protected]

[email protected]

[email protected]

Abstract This paper discusses general quality of services issues relating to SOAP based SOA and shows how our fault injection tool, WS-FIT, can be applied to this problem. We discus how WS-FIT can applied to a wide range of quality of service domains. Finally we apply WS-FIT to a sample quality of service domain to demonstrate its usefulness.

1. Introduction The proliferation of Web Service technology within the domains of eCommerce and eScience has made quality of service (QoS) issues a high priority. Our research into dependability assessment [1-4] has produced a method and tools that have wider applicability for all areas of QoS with releation to SOAP based Service Oriented Architectures (SOA). Standards like SOAP, UDDI, and WSDL [5] are being adopted by all major Web Service providers, covering a wide range of applications: financial; commerce; scientific; etc. All Web Services are going to need to establish and adhere to standards so QoS will become an important differentiating point of these services. This differentiation may become even more important in the future with technologies such as dynamic service composition being developed. In our pervious work we have shown how our method can be used to test reliability aspects of a SOAP based SOA [1, 2]. This research allowed us to uncover minor inconsistencies in the implementation of the Apache SOAP 2 protocol stack. In this paper we will demonstrate how our tools can be used to introduce latencies into a system and thus test quality of service aspects of a system.

2. QoS in relation to Web Services QoS covers a whole range of techniques that are combined to form metrics on the quality of a web service offered for by a system. This paper will concentrate on the non-functional properties of QoS

such as availability, performance, reliability, and security. Availability: the quality aspect of whether a Web Service is present or ready for use. This is represented as the probability that a Web Service will be available. This may be affected by such things as time to complete a pervious operation, loading on a particular service, etc. Accessibility: the quality aspect that represents the degree the Web Service is capable of serving a request. A service may be available but not accessible. For instance the initial request can be accepted but it cannot be processed due to some other dependency, e.g. it may depend on another unavailable service. Accessibility can be improved by improving the scalability of a system. I n t e g r i t y : the quality of the Web Service maintaining the correctness of any interaction. If a transaction fails data should remain in a consistent state. This can be achieved through mechanisms such as. atomic transactions, rollback mechanisms, etc. Performance: the quality aspect that is defined in terms of the throughput of a Web Service and the latency. Throughput is the number of requests serviced in a given period and the latency is the time taken to service a request. The aim is to produce a high throughput but low latency system. Throughput and latency can be affected by such factors as processor speed, code efficiency, network transfer time, etc. Reliability: the quality aspect that represents the capability of maintaining the service and service quality. One measurement of reliability is the number of failures during a given period. Another aspect of reliability refers to the probability that defines the dependability that a request is correctly send and serviced. Regulatory: the quality aspect that the service corresponds to rules, laws, standards and specifications. This can have an effect on areas such as availability, performance, and reliability through Service Level Agreements (SLA). SLA can define minimum levels of performance expected by a service that set levels for its dependability.

S e c u r i t y : the quality aspect that defines confidentiality for parties using a service. Again this can be influenced by regulatory factors. It can also affect performance due to the extra overhead incurred in implementing security mechanisms.

we shall recap the original work here and determine if it is significant to our experiments. We also add results from a case that we were unable to complete due to technical difficulties at the time.

4.1. Latency Model

3. WS-FIT

OGSA

SOAP API

In previous research [4] we detailed how WS-FIT would introduce a small latency into a system. Since this paper deals with deliberately introducing latencies

Client

SOAP API

4. Latency Issues

OGSA

WS-FIT (Web Service – Fault Injection Technology) is a fault injector that allows network level fault injection to be used to test SOAP based SOA. WS-FIT has been implemented specifically to test SOAP based SOA which utilize conventional RPC semantics and a detailed description of its design is given in [2]. This implementation specifically handles the problems associated with modifying SOAP messages when signing and encryption are being used. WS-FIT is implemented to inject faults using network level fault inject. Network level fault injection is a runtime injection technique that injects faults into captured network packets. Network level fault injection is usually based upon more or less random corruption of bytes within a network packet. Our method extends this technique to make meaningful perturbation to a network packet, e.g. our method can target a single parameter within an RPC message sequence. Also since our method is script based it is possible to manipulate message flows in other ways, for instance latencies can easily be added to message transfers and thus affect throughput. Our method uses an instrumented SOAP API that is made up of two small pieces of hook code. One hook intercepts outgoing messages, transmits them via a socket to the fault injector engine and receives a modified message from the fault injector. This message is then transmitted normally to the original destination. There is a similar hook for incoming messages, which can be processed in the same way. Whilst a number of existing fault injectors could be used to perturb RPC parameters, notably DOCTOR [6] and Orchestra [7], these tools are designed for general purpose protocol testing. WS-FIT was designed around an engine to decode SOAP messages and presents an interface at the script API level with the information included in a SOAP RPC easily accessible. WS-FIT also provides a number of facilities to automatically generate test script skeletons from WSDL definitions and visualize RPC parameters in real-time.

Latencies are introduced into the system by the WS-FIT hook code and the processing of SOAP messages by WS-FIT. Figure 1 details the individual latencies introduced into the system by WS-FIT. From this we have: Te a which is the time taken to execute the hook code in the SOAP library; Te t which is the time taken to execute the trigger script; Tes which is the time taken to execute the injection script; Ter which is the time taken to process operations on the result class; Te l which is the time taken to execute a log operation; Tt t which is the time taken to send the SOAP message from the SOAP API to the fault injector trigger; Ttnf which is the time taken to send the no fault message from the fault injector trigger to the SOAP API; Ttn which is the time taken to send a null message from the fault injector trigger to the SOAP API; Ttf which is the time taken to send a message containing a fault from the fault injector script to the SOAP API.

Service

Tea Ttnf Ttf

Ttn

Ttt

Trigger Tet

Script Tes Results Ter

Log Tel

Figure 1: Fault Injection Latencies There are three different paths the processing of a SOAP message can take through WS-FIT. These are: Tif - Inject a fault (a fault is injected into a SOAP message); Tinf - Inject no fault (the message is passed back to the SOAP stack unaltered); Tin - Inject null (the message is dropped). Injecting a fault into a SOAP message requires an extra step of running the script to inject the fault. This has been optimized out for the other two cases. The basic equations for the three cases are given in Equation 1. These are summations of all the timing terms encountered through the path.

Equation 1

Tif = Tea + Ttt + Tet + Tel + Tes + Ter + Tel + Ttf Tinf = Tea + Ttt + Tet + Tel + Ttnf Tin = Tea + Ttt + Tet + Tel + Ttn

€

We can make some assumptions about some of the timing terms used in these equations (See Equation 2). Firstly, we can assume that all network transfers will introduce the same latency into the system. This gives us Tt . We can also assume that since Te t and Tes both utilize a SAX parser this will be the major latency in these terms so these can be equated to a common term Tp. € Equation 2

Tt ≡ Ttt ≈ Ttnf ≈ Ttn ≈ Ttf Tp ≡ Tet ≈ Tes

€

We can substitute these factors into Tif, Tinf and Tin in Equation 1 to simplify these terms and give Equation 3. Equation 3

Tif = Tea + 2Tt + Tp + Tel + Tp + Tel + Ter Tinf = Tea + 2Tt + Tp + Tel Tin = Tea + 2Tt + Tp + Tel We can further factor out a common term Tcom from the equations given in Equation 3 to give Equation 4.

€

We can make some further assumptions about the latency introduced by some terms in relation to others. If we assume that network transfers Tt and logging functions (which include a write to disk) Te l are far greater that terms which involve just CPU processing, then we can assume that these terms are negligible as far as the model goes (See Equation 5) and these can be removed from the model.

Tif = Tcom + Tel Tinf = Tcom Tin = Tcom Tcom = 2Tt + Tel Our pervious work supports this model [4] but we were unable to verify that the terms Tinf and Tin were equivalent. Our subsequent experiments have verified that this is the case. Our original experiment gave the results in Figure 2. We ran a further experiment on the same system injecting a series of null faults and this gave the results given in Figure 3. The timings for Tinf and Tin given in these experiments compare favorably with our model given in Equation 6.

Mean Std Dev

All Write Read (micro sec) (micro sec) (micro sec) 185 78 291 221 33 272 695 16 65 58 310

Max Min Median 1st Quartile 3rd Quartile

235 60 65 62 67

695 16 294 18 563

Figure 2: Timing obtained for Tif

Mean Std Dev

All Write Read (micro sec) (micro sec) (micro sec) 174 66 282 222 34 273

Max Min Median 1st Quartile 3rd Quartile

982 3 53 48 535

500 3 53 50 55

982 4 535 6 551

Figure 3: Timing obtained for Tnf

0 ≤ Tp < Ter

Given that these timings give latencies of less that 200 micro seconds and normal Web Service timeouts are defined in the order of seconds we can safely ignore these effects in our test scenarios that follow.

0 ≤ Tp < Tea

5. Test Scenarios

Equation 5

Tp ≈ 0 Ter ≈ 0 Tea ≈ 0

€

Equation 6

Equation 4

Tcom = Tea + 2Tt + Tp + Tel

€

By removing these terms from the model we are left with the final model given in Equation 6.

Our test scenarios will be based around an electronically controlled heater system. In our scenario a ‘heater unit’ is made up of a thermocouple, a heater

coil and a control unit. The thermocouple and heater coil hardware are interfaced to two separate machines whilst the heater controller runs on a third machine.

current temperature to the client program via the getTemp operation. Measured Temperature Without Injected Faults

√

Heater

Instrumented SOAP API

Instrumented SOAP API

Send Hook

60

50

Temperature (degrees C)

Thermocouple

Receive Hook

Trigger Script Results

Non Linear Client Linear Client

30

20

10

0

Log

0

Receive Hook

WS-FIT

40

Instrumented SOAP API

HeaterController

10

20

30

40

50

60

Time (seconds)

Messages to/from WS-FIT Messages to/from SOAP Stacks

Figure 4: ‘Heat Unit’ Design Driver functionality is provided by a Web Service running on each machine: Thermocouple to drive the thermocouple hardware; Heater to drive the heater coil hardware; and HeaterController to provide a coordination facility for both the Thermocouple and Heater Web Services (See Figure 4). With embedded smart devices becoming more prevalent, this type of system may become more common in the future, hence testing reliability and security aspects will become crucial [8]. Whilst this demonstration scenario has been constructed to provide the maximum number of message transfers and so allow WS-FIT to be easily demonstrated, the system could be easily constructed with three embedded microprocessor boards running an embedded OS such as VxWorks or QNX. These could conceivably be running a full web server connected to the outside world via an Internet connection for remote monitoring and control. Many embedded devices found around the home and in industry already provide such functionality. The Heater service allows power to be applied in small increments via two operations: incPower and decPower. The heater coil power increments are logarithmic in nature (See Figure 5), therefore if linear behaviour is required a control algorithm is required to provide this. Thermocouple allows the current temperature to be read back via the getTemp operation. Since power to the heater coil can only be modified in small increments the HeaterController provides the setTemp operation that uses a time-based algorithm that issues incPower and decPower operations to the Heater to set the correct power level. The current temperature is monitored by HeaterController to provide feedback into the algorithm. HeaterController also provides the

Figure 5: Baseline Readings This ‘heater unit’ is to be used in a chemical process. A sample is to be heated over a precisely defined period with a precisely linear temperature rise to a temperature of 60°C. This constitutes the SLA for the HeaterController. Figure 6 shows the modelled behaviour. A client program is to be used to provide this behaviour. This client will send the temperature every second to HeaterController via the setTemp operation. The HeaterController will then use its time-based algorithm (10 adjustments per second) to adjust the power the heater coil supplies. The small time steps used by the client should ensure that the temperature rise is linear. This test scenario will demonstrate how WS-FIT can be used to modify latencies in a system. WS-FIT will allow this to be achieved without the need for any additional test harness or test code. By using the WS-FIT tool it is possible to introduce latencies into any RPC messages and also monitor messages sent/received. In this way we can assess the system and determine if the fault tolerance mechanisms included in the system are adequate, if the system is scalable, etc. Each machine in the system will be instrumented with a modified SOAP stack. This will allow us to inject faults on any machine but also capture and log all traffic to that server. This logged traffic can then be analyzed off line to determine latencies, etc. These tests were carried out on a mixture of Apple G3 700Mh systems running Mac OS X 10.3.5 and Intel Pentium 4 2.2Ghz systems running Redhat 9. Apache Tomcat 5.0.19 was used as the container utilizing Apache Axis 1.1. Network connections were 10Mb Ethernet.

5.1. Baseline Tests Initially we will run the system using a ‘null’ script, i.e. we will not inject any faults. We will run

Model comparied to Linear Client 60

Temperature (degrees C)

50

40

Model Linear Client

30

20

10

0 0

10

20

30

40

50

60

Time (seconds)

Figure Client

6:

Model

compared

to

Linear

A further analysis of the log files generated by WSFIT shows the frequency of incPower and decPower operations sent between the HeaterController and Heater. The maximum theoretical throughput of these messages is 10 per second. This data shows that the current system is running within this throughput constraint with an incPower/decPower message being sent on average every 1.07 seconds.

5.2. Injecting Latency Our experiment will be designed to inject a latency into the system at an appropriate point. The injection trigger must be chosen with care since we are attempting to change the maximum throughput of the system. To determine the effectiveness of a particular injection point we utilized the feature of WS-FIT which allows real-time visualization of RPC parameters. A monitor can be set on any RPC parameter contained in a SOAP message and these can be used to give a visual representation of the system as it is running. We can use this to determine that a test script is producing the desired results. Since we are attempting to affect the throughput of the system our first injection point will introduce a latency into the setTemp operation sent between the client and HeaterController. The client sends a

setTemp operation to the controller every second with the required temperature at that point in time. The HeaterController will then use a loop running in a thread to increment/decrement the H e a t e r . The threading of this algorithm allows it to run asynchronously. Our first script introduces a 10 second delay into sending the s e t T e m p operation to the HeaterController. This latency was chosen because it was large enough to be noticeable but less that the default timeout set by SOAP stack for the timeout of RPC operations. We used the visualization feature previously mentioned to monitor the thermocouple readings to determine if the test scenario had adversely affected the system. In this case the temperature increase was not adversely effected. We postulate that this is due to the asynchronous nature of the HeaterController. Since the HeaterController is capable of increasing the power to the Heater independently, introducing the latency at this point caused the client to request larger temperature increases at a lower frequency rather than smaller temperature increases at a higher frequency. The simple fault tolerant nature of the HeaterController design allowed these requests to be correctly serviced. This is born out by the information contained in the log files once they were analyzed. Our second script introduced latency into the message exchanged between the HeaterController and the H e a t e r . As explained previously the HeaterController runs in an asynchronous manner, largely independent of the other latencies introduced by other components. The HeaterController sends incPower and decPower operations to the Heater dependent on the temperature returned by the Thermocouple and the required temperature set by the setTemp operation. Our script introduced a 1 second latency into sending each incPower and decPower message to the Heater. In this way we reduce the throughput of the HeaterController to Thermocouple message exchange, and thus alter the performance of the system. Injecting a Latency Into the System 60

50

Temperature (degrees C)

the system for a length of time under normal conditions to determine baseline timings from the collected log. WS-FIT has the capability to visualize parameters in SOAP messages in real-time. The data from these visualizations can be output to file to allow offline processing in spreadsheets. We can assess the system by monitoring the getTemp operation (See Figure 6). This data shows that under normal operation, the system functions as required and the client algorithm effectively removes the logarithmic nature of the heater coil from the overall system. We can compare the operation of the system with the model to confirm this.

40

30 Linear Client with Latency Linear Client without Latency 20

10

0 0

100

200

300

400

500

600

700

800

-10 Time (seconds)

Figure 7: System

Injecting

Latency

into

the

Again we used the visualization facility provided by WS-FIT to monitor the results of this experiment. Our results are given in Figure 7 and clearly show that the SLA for this system hasn’t been met. Further analysis of the log files confirms that the throughput of messages has been significantly reduced and hence the required rise in temperature can’t be achieved.

6. Conclusions and Future Work This paper has show how network level fault injection can be used as an aid to dependability assessment of quality of service constraints. We have shown how our tool, WS-FIT, can be used to achieve this. We have used our tool to perform a quantifiable experiment on a simple SOAP based SOA and used the results to assess the impact of unexpected latencies within the system. Our future work will concentrate on three main areas. Firstly, we will conduct experiments on more complex systems. This will allow us to not only evaluate and enhance our method and tools further but will also provide us with more metrics on constructing test scripts using network level fault injection techniques. Secondly, we will examine and enhance our realtime RPC visualization method included in WS-FIT. Our preliminary experiments with visualization have provided promising results that indicate its potential usefulness. Finally, we will investigate a method that will allow us to construct test scripts for our tool. Currently our method and tool provide assistance for uses writing tests scripts. This includes generation of skeleton scripts from WSDL definitions and the generation of triggers from graphical input but the tools still require the user to enter code fragments to complete the test scripts. Ideally, our new method would be automated and would generate complex test scripts to give some defined level of coverage.

7. Acknowledgments This work was supported by the EPSRC through a DTA studentship.

8. References [1]

[2]

N. Looker and J. Xu, "Assessing the Dependability of OGSA Middleware by Fault Injection," Proceedings of the Symposium on Reliable Distributed Systems, pp. 293-302, 2003. N. Looker and J. Xu, "Assessing the Dependability of SOAP-RPC-Based Web Services by Fault Injection," 9th IEEE

[3]

[4]

[5]

[6]

[7]

[8]

International Workshop on Object-oriented Real-time Dependable Systems, pp. 163-170, 2003. N. Looker, M. Munro, and J. Xu, "WS-FIT: A Tool for Dependability Analysis of Web Services (draft)," presented at The 1st Workshop on Quality Assurance and Testing of Web-Based Applications, COMPSAC, Hong Kong, 2004. N. Looker and J. Xu, "Dependability Assessment of an OGSA Compliant Middleware Implementation by Fault Injection," presented at UK e-Science All Hands Meeting 2003, Nottingham, UK, 2003. F. Curbera, M. Duftler, R. Khalaf, W. Nagy, N. Mukhi, and S. Weerawarana, "Unraveling the Web Services Web: An Introduction to SOAP, WSDL, and UDDI," IEEE Internet Computing, vol. 6, pp. 86-93, 2002. S. Han, K. G. Shin, and H. A. Rosenberg, "DOCTOR: An IntegrateD SOftware Fault InjeCTiOn EnviRonment for Distributed Real-time Systems," presented at International computer performance and dependability symposium, Erlangen; Germany, 1995. S. Dawson, F. Jahanian, and T. Mitton, "ORCHESTRA: A Probing and Fault Injection Environment for Testing Protocol Implementations," presented at International computer performance and dependability symposium, Urbana-Champaign; IL, 1996. P. Koopman, "Embedded System Security," in Computer, vol. 37, 2004, pp. 95-97.