Testing Component-Based Software Ye Wu Information and software engineering deparment George Mason University Fairfax, VA 22030, U. S. A. (703) 9934-1651 [email protected]

Dai Pan Computer Science Department University at Albany State University of New York Albany, NY 12222, U. S. A. (518) 442-3389 [email protected]

Abstract Component-based software development facilitates software reuse and promotes productivity and quality. Although much work has been proposed for building component-based systems, techniques of testing component-based systems have not been well developed. In this paper, we present a technique for component-based software in integration testing. Our approach uses both static and dynamic analysis to guide test case generation. Static analysis captures the interaction relationship among the components, which is used to investigate what needs to be tested. Dynamic analysis keeps track of the interface invocations as well as event triggering for each test executed, and the information is used to determine test adequacy. The methodology proposed is efficient and effective, as demonstrated by the promising results obtained from an empirical study.

Mei-Hwa Chen Computer Science Department University at Albany State University of New York Albany, NY 12222, U. S. A. (518) 442-4283 [email protected]

tances; some components may be developed in-house, while others may be the third party off-the-shelf components of which the source code may not be available to the developers. A component may server as a server that provides services or as a client that requests services, some components may assume both roles, i.e, they act as a server as well as a client. Although, the components may interact in different fashions depending on the underling infrastructure, in general, a server component published its interfaces which consists of a service name, a set of parameters and a set of signatures of the functions that perform the service. When a client needs a service, it first looks up for the interfaces published by the server components in the system then it makes the request when the desired interface is found. Although the heterogeneity and the implementation transparency properties ease the development and the integration process, they introduce some difficulties when testing component-based software using traditional program-based techniques. These difficulties can be foreseen for the following reasons: First of all, if the source code of some components is not available, the program-based white-box testing may not be of use. Secondly, even the source code of every component is obtainable, some components may be written in one programming language while other are written in one or more different languages. Performing the whitebox approaches, such as Statement, Branch, data-flow based and mutation testing, often relies on some instrumentation of the source code in order to trace and record coverage information. The more languages used in the system the more complex the instrumentation can be performed. Finally, since components can be easily plug-and-play and some third party components may be frequently upgraded. The white-box approaches that use coverage information as the guide to determine test adequacy may not be appropriate for evolving systems [?]. Due to these difficulties, prevalent approaches often apply functional testing in the integration and system level testing [?]. In functional testing, test cases are generated from the input domain derived from the specification. The method directly tests a program’s behaviors from a user’s point of view regardless of how the program is written; thus it is likely to capture the most significant faults of the program assuming that the specifications are complete and correct [3, 22]. However, in spite of a few application

* Keywords Component-based software, Software testing, program analysis 1 Introduction Component-based software engineering has been increasingly adopted for software development. Such approach, which uses reusable components as the building blocks for constructing software, embellishes the likelihood of improving software quality and productivity. Much work has been devoted to developing infrastructure for the construction of component-based software[2, 12, 26, 28, 30, 39]. The aim is to achieve multiple quality objectives, such as interoperability, reusability, evolvability, buildability, implementation transparency and extensibility, to facilitate fast-paced delivery of scalable evolving software systems. To this end, a component-based software system often consists of a set of self-contained and loosely coupled components allowing plug-and-play. The components may be implemented by using different programming languages, executed in various operational platforms and distributed across geographic dis-

1

domains such as mathematical software, functional testing often requires infinite time to explore all the possible functions and their combinations [3]. Therefore, to maximize the number of faults that can be detected during a limited time period, other techniques need to be applied. In some applications such as telecommunication systems where the operational profile of the program can be closely predicted, statistical testing can be adopted [9, 29], which tests an input partition more intensively if this partition will be most frequently used in the operation. In other application domains where the operational profile is difficult to anticipate or it varies in different application environments, such an approach may be appropriate.

suggest issues of testing and maintaining component-based systems. Harrold et al.[15] proposed a testing technique that is based on the analysis of the component-based system from component-provider and component-user perspectives. The technique was adapted from the existing technique[16, 17], which makes use of complete information from the components for which source code is available and partial information from those for which source code is not available. Buy et al.[4] propose an incremental integration strategy based on the inter-component data-flow analysis. Ghosh and Mathur[14] discuss the issues in testing distributed component-based systems and suggest an interface and exception coverage-based testing strategy.

With the emerging popularity of the component-based software development, to ensure the reliability of the systems, there is a need to develop a component-based testing technique that can overcome these difficulties. In this paper we present a framework for testing component-based software at the integration level. Our focus is to detect faults residing in the interfaces and the failures encountered in the interactions among components. The technique we proposed utilizes both static and dynamic information to design test cases. The static analysis identifies the collaboration relationships among components, including interfaces, events, context sensitive paths and content sensitive paths, represented by using a component interaction graph. To capture the dynamic behavior of each test execution, we use an interface and event interaction graph to trace the interaction activities among the components for each input, thus recording the coverage information for each selected criterion.

To introduce our methodology, we first quote Weyuker’s two axioms which motivated our development of the framework for testing component-based software.

The remainder of the paper is organized as follows: In section 2, we propose a fault model that depicts the potential pitfalls of component-based software. The methodology we propose is described in Section 3, and in Section 4 we present an empirical study to demonstrate the potential strengths of the methodology. We conclude this study in Section 5. 2 Background Weyuker [?, ?] developed a set of axioms to aid in determining test adequacy and used them to evaluate several programbased testing techniques. Perry and Kaiser [?] further applied these adequacy axioms on object-oriented paradigm and suggest that with the presence of object-oriented features, in particular the (multiple) inheritance feature, testing subclasses and superclasses require special attentions. Stemming from these studies, we developed a class testing technique [?] and an object-flow based testing technique [?] for testing object-oriented classes and programs. In componentbased testing, Rosenblum [?] proposed a formal model for adequate testing of component-based software, in which a ”C − adequate” criterion is defined to determine the adequacy of a test set for a given component-based software system as well as for a component. Meanwhile, a number of studies [4, 8, 14, 15, 40] have been proposed which analyze the characteristics of component-based software and

The "anticomposition" axiom: There exists programs P and Q such that T is adequate for P and P (T ) is adequate for Q, but T is not adequate for P ; Q [i.e., to the sequential composition of P with Q]. The "antidecomposition" axiom: There exists a program P and component Q such 0 that T is adequate for P , T is the set of vectors of values that variables can assume on entrance to Q or some t of T , 0 and T is not adequate for Q. The ”anticomposition” axiom reminds us that even though each individual has been well tested when composing these components together, it is necessary to retest the behaviors of the components while they are interacting with others. Since the interfaces are the means though which the interactions take place, therefore, the interfaces published by each component need to be tested. Besides the interfaces, some components published exceptions as well to take care abnormal executions. The exceptions may be captured by the components in which they are defined or may be handled by other components. The exceptions which are handled by other components need to be tested as well, for they are considered as interactions among components. The ”antidecomposition” axiom suggest that when the enclosing context of a component is changed, the component needs to be tested. To interpret in the component-based system, when an interface/exception is invoked/raised by different components, each invocation/raise needs to be tested. Moreover, some relationships among the interfaces may cause the system behave differently when the order of the interface invocations is permuted. For example, when an interface, I1, references a variable that is defined in the interface, I2, an invocation of I1 following an invocation of I2 will likely to have different results when the order is switched. Other example, results differ when an invocation of I1 following the invocation of

I2 compare with the one following an invocation of other interface, I3. These content dependent and context dependent relationships need to be taken into account. In summary, when testing a component-based system, we suggest that every interface/exception, every invocation/raise of the interface/exception, every context sensitive execution path and content sensitive execution path need to be exercised at least once. The detailed descriptions and the implementation are provided in the following sections.

Cashier Service Component

Card Service Component

Goto Security&Transaction Component

Goto Security&Transaction Component

E_Cancel

E_Cancel I_Init

I_Init

I_GetVal

I_Validatation Security Component

I_Init

I_Transaction KeyPad Component

I_DB_Service

3 Methodology Definitions In this section we describe the terminologies used in our methodology. A component-based system is composed of a set of components which interact with each other via the means of interfaces and exceptions. Component: A component is an encapsulated unit which is self-contained and can be plug-and-play. Interface: A server component provides one or more interfaces through which a client component can request the services provided by the server component. The name of the interfaces are published when a server component registered to the system. An interface may encapsulate one or more method functions performing the service. Exception: Exceptions are similar to interfaces except that the interfaces are invoked under normal conditions, while the exceptions are raised when an abnormal execution is encountered. Some exceptions are handled by the component from which the exceptions are exposed while others may be handled by other components. Event: An event is an invocation of an interface or a raise of an exception. In addition, some implementations allow user actions directly interact with other components without making through an interface or an exception. These user actions are also considered as events. Therefore, the events can be classified into interface invocation events, exception handling events and user action events. Component interactions In order to correctelly integrate a number of components, the interactions among interfaces and events are identified and classified into the following three categories: 1. One interface when invoked generates an event. For example, an interface which is to close the window, when invoked, a close window event will be generated. 2. When one event is generated, it invokes an interface of a component. For example, pushing a button generates an event and invokes certain interfaces. 3. When one event is generated, it will trigger another event in the same or a different component. For example, in a situation where a window component contains

Database Component

Figure 1: Component Interaction Graph for ATM Simulator a sub widow component, when a resize event is generated for the main window, it triggers the resize event of the sub window. The above relationships among interfaces and events depict the direct interactions among interfaces and events, a Component Interaction Graph (CIG) can be used to represent these relationships. S In a S Component Interaction Graph(CIG) G=(V,E), V = VI VE {Ventry ,Vexit }. vi ∈ VI stands for an interface of a component, while vi ∈ VE stands for an event which may be generated in the component. Ventry andVexit are two special nodes which indicate the entry and exit of the program. eij =(vi ,vj ) ∈ E denotes one of the relationships described above. A path (or subpath) in CIG is a finite sequence of vertices (v0 , v1 , ..., vk ) such that ei,i+1 = (vi , vi+1 ) ∈ E for i = 0, 1, ..., k − 1. A simple path is a path (v0 , v1 , ..., vk ) such that for any i and j, i 6= j and vi 6= vj . In the CIG, the out-degree of a node is the number of edges leaving it and the in-degree of a node is the number of edges entering it. For every node ∈ VE, its out-degree must be greater than 0. If the out-degree of the node is 0 and the event has to be processed, report error, otherwise, it will be removed from the CIG, because no components will process the event, and the event will not have any effect on the system. Figure 1 depicts a Component Interaction Graph of an ATM simulator. In this system, seven components are developed and tested separately. A Security component obtains and verifies the account and pin number information of either a cashier or a customer, a Keypad component provides another way of processing the input of the pin number and transaction amounts, besides the direct input from keyboard, a transaction component handles all types of transactions of the ATM and a database component provides database management services. The security component has two interfaces I Init & I V alidation, and an event, Cancel. The event will terminate the current operation and start a new

operation from the very beginning. The transaction component has similar interfaces, I Init & pI T ransaction, with Cancel as an event. Keypad has two interfaces, I Init and I GetV alue. The database component has one interface, I DBservice . Interface and event dependence relationships In the CIG, the direct interaction among interfaces and events are depicted. A very basic requirement of adequate testing is that all the interfaces and events of the components are covered by test executions. When all interfaces and events are covered, it will provide us the confidence in the basic interactions among components. Whereas, two interfaces, events or one event and one interface may interact with each other through other interfaces and events. Those types of interactions may not be satisfied even when all interfaces and all events are covered. Those indirect dependence relationships have two variations: Control dependence and Data dependence. control dependence: Source code either available or not available: 1) Two interfaces I1 and I2 have a control dependence relationship if there exists a simple path n0 , n1 , n2 , ..., nk in CIG such that n0 = vi1 and nk = vi2 , ni ∈ V for i=1,2,...,k-1, and edge = (ni , ni+1 ) ∈ E for i=0,1,...,k-1; 2) An even e and an interface I have a control dependence relationship if there exists a simple path n0 , n1 , n2 , ..., nk in CIG such that n0 = ve and nk = vi , ni ∈ V for i=1,2,...,k-1, and edge = (ni , ni+1 ) ∈ E for i=0,1,...,k-1; To identify the control dependence relationship, when source code of the component is available, the direct interaction can be easily identified from the control flow graph(CFG) of than component. An additional transitive closure algorithm will find out all the control dependence relationships. Even when the source code of certain components is not available, the interfaces and events of those components have to be known in order to integrate with other components. In addition, in some cases, if these components need services provided by other components, these requirements are accessble. Thus, even in the case when source code is not available, the direct interactions can be still retrieved, so the control dependence relationships can be obtained. Because the control dependence relationships not only include the direct interactions, but include the indirect collaboration relationships among interfaces and events through other interfaces and events as well. So, the contol dependence relationships may be utilized to identify the interoperability errors, which due to the improper interaction among different components. Obviously, the control dependence relationships include more complicate relationships among interfaces and events, so if the simple interface and event coverage criteria is satisfied, the cotrol dependence relationships may not be covered.

For example, in previous ATM example, the following two test cases can cover all the interfaces and events related to the CardService, CashierService, Security and Keypad Service components. T1) CardService -¿ Sercurity Service -¿ KeyPad Component. T2) CashierService -¿ Sercurity Service(Utilize keybord to input). But the two test cases fail to cover the control dependence relationships between the interface of CashierService component and the interfaces of keypad components, and a failure will be encountered if the keypad is misinitialized in the cashier mode. Therefore, even though the interactions between the security component and the keypad component have been verified, when the security component locate in a different environment, it need to be retested. In other words, Cashierservice-¿Security Service -¿ keypad component need to be tested as well. In order to cover the control dependence relationships, we have two choices: some-control-dependence and all-controldependence. When two interfaces or one event and one interface have a control dependence relationship, the control dependence relationship can ariase in different senario, in other words, there may be more than one simple path between interfaces and events. Some-control-dependence only requries to cover at-least one of the simple path, while all-controldependence requires to cover all of the simple pathes. Theoritically, O(n2 ) number of test cases are necessary in order to satisfy some-control-dependence criterion, while exponential number of test cases are needed to satisfy all-controldependence criterion. For example, in the following graph, there are all together 2n−1 number of valid pathes between I0 and In . Theoritically, exponential number of test cases are needed. But in the real world, tree-style archetecture are usually expected, and the depth of the tree is limited. So the number of valid pathes is usually polynomial. Data dependence Both the direct interaction among interfaces and events, as well as the control dependence relationships are discuss the interactions of a component-based system from the control flow prospective. Data dependence, on the other hand, can provide additional information in generating test cases and detecting faults. When an interface is invoked, a group of methods or functions will be executed. Therefore, function dependence relationships, which have been shown to be effective in object-oriented class testing[6] and in regression testing[44], can be utilized to identify interface and event dependence relationships. The function dependence relationship is defined as follows: A function f1 depends on a function f2 if the value of a variable defined in f2 is used in f1 . Data dependence: Interfaces I1 and I2 have a data dependence relationship if one of the following conditions holds: 1. Source Code Available

(a) I1 and I2 are within the same component, and I1 calls a function f1 , which depends on another function f2 , and f2 is called in I2 ; 0

(b) I1 invokes an interface I1 and I2 invokes a inter0 0 0 face I2 where I1 depends on I2 ; 2. Source Code not Available If we do not have any information other than the interfaces, approximations of the dependence relationships have to be made. A conservative approximation considers every pair of interfaces as having a dependence relationship. Another simplified approximation will regard each interface as independent, and free of any dependence relationship with any other interface. With additional information like design documents or detailed functionality descriptions, more realistic approximations can be made. When an event triggers an interface, which has dependence relationships with other interfaces, the event is involved in those dependence relationships as well. 1. An interface I data depends on event e if e control depends on an interface I1 , where I depends on I1 ; 2. An event e data depends on interface I if e control depends on an interface I1 , where I1 depends on I; 3. Event e1 depends on event e2 if e1 control depends on an interface I1 , while e2 control depends on an interface I2 , and I1 depends on I2 . In the real systems, besides the interoperability faults, there are many faults related to the data dependence, which won’t be able to be identified by only utilizing the control dependence information. Data dependence relationships are necessary. For example, in the ATM simulator system, the I GetV alue interface depends on the I Init interface of the KeyPad component. Therefore, the I T ransaction interface depends on event Cancel of the transaction component, because Cancel may invoke interface I Init and follow by the invocation of interface I GetV alue from the interface I T ransaction. If when event Cancel is generated, the KeyPad is set to the ”Password” mode by mistake, a type I fault, which is inter-component is encountered, and the fault will be revealed only if generate an Cancel event followed by an execution of the IT ransaction, according to the data dependence relationships. Construction of CIG with source code CORBA, Enterprise JavaBeans(EJB) and COM/DCOM are the most popular techniques for component-based software development. Each of them have their unique characteristics, therefore, different approaches are necessary when build CIG. In this section, we will first discuss how to generate CIG, when the source code of the components is available.

When the source code is not available, we need to process CORBA, EJB and COM-based systems seperately. Building CIG Given a component-based system, if the source code is available, we can always easily obtain their interfaces and events. Even though for different types of componentbased systems, the retrieval method might various, we’ll discuss it later in this section. The following algorithms will build the CIG and find out all the context-sensitive path which need to be tested. Algorithm 1: CIG Construction Input: A Component-based System C and its source code. Output: CIG Step 1: Identify the interfaces of all the components. Create a corresponding interface node in the CIG. Step 2: For each interface, parse its source code, when encounter an interface invocation, create an event node, and connect the event node with both of the caller and callee interfaces. Step 3: For each interface, parse its source code, when encounter an exception which needs to be processed by other interfaces, create an event node and connect the event node with the interface which raise the event, and all other possible interfaces which may process the event. Step 4: For each component, parse its source code, when encounter a user action event, create an event node in the CIG, and connect this node with all other interface nodes which may be invoked by the event. Algorithm: Path Identification Input: CIG Output: A set of pathes P which need to be tested. Step 1: Let P =Φ; For each node v in CIG let Pv = v; Mark v unvisit; Step 2: Using Depth First Seach Algorithm seach the CIG. When expand T from node vi to vj , if 1) vj has been visited, Pvi = Pvi Pvj ; 2) Otherwise, continue with the DFS T search. When backtrack from node vj to vi , Pvi = Pvi PT vj ; Step 3: For each node v, which it’s indegree is 0, P = P Pv ; Construction of CIG without source code The algorithm presented in the prvious section provides a guildline for constructing the CIG and obtains all the paths which need to be tested. For different types of componentbased systems, the way of identifying interfaces and events may be various and in many cases the source code of a component is not available. In the following, we will discuss detail procedures of constructing CIG and how to build CIG when source code is not available for different types of component-based systems. CORBA, Enterprise JavaBeans(EJB) and COM/DCOM are the most popular techniques for component-based software development. Each of them have their unique characteristics, therefore, different approaches are necessary when build CIG. In this section, we will first discuss how to generate CIG, Enterprise JavaBeans(EJB) Bean is the basic building block of EJB-Based systems, and

EJB provides a special procedure to obtain the interfaces and events. Interfaces EJB has an essitial feature introspection, which is a precess of analyzing a Bean to determine its capability, regardless the availability of the Bean’s source code. The introspection can identify all the interfaces provided by the Bean. There are two type of interfaces, one type of interfaces are those interfaces which follow the special naming conventions, and can be retrived automatically. Other are user-defined interfaces. In order to obtain this type of interfaces, a BeanInfo class, which will decribe all user-defined interfaces, is nessary. In reality, BeanInfo classes are very often available. User action events . Besides the interfaces of a Bean, the introspection procedure will also obtain all user action events, such as mouse move and etc. Interface invocation events . For interface invocation events, when the source code of a bean is available, a simple parsing will obtain interface invocation information. When the source code of a bean is not available, which in most cases they are server beans(server components), there will be no interface invocations. If there are, a specification are usually provided to descibe the interfaces will be invoked, and therefore, the interface invocation event can be identified. Exception events Exception handling of Beans is similar to the java exception handling procedure. When the source code of a bean is available, a simple parsing will find out all the exception which will be thrown to the caller interfaces of other beans. In some cases, the caller interfaces won’t be able to handle the exception, and need to pass to its caller, when the source code of all the Beans involved in the exception is available, it can be properly handled. When the source code of some of the beans is not available, specification is necessary to correctly process the exceptions. CORBA Common Object Request Broker Architecture(CORBA) is the infrastructure which is to enable open interconnection of different languages, implementations and platforms. Therefore, CORBA-based systems are very often heterogeneous. But many of the CORBA features may greatly simplify the analysis of a CORBA-based system. Interfaces . A essential part of CORBA is the OMG interface deinition language(OMG IDL). Despite of the programming languages, platforms and availability of the source code, IDL files must be known, and these IDL files contain all the interfaces which are provided by the components. So interfaces can be easily acquired.

Exception Events . In addition to interfaces, OMG IDL files also defines the exception events, which may generated by the server components, and the exception event can be idenfied, regardless the availability of the source code, programming languages and platforms. Invocation Events . There are two types pf invocation events in CORBA-based systems. One type is traditional synchronous interface invocation events, which can be processed by using a similar procedure to the EJB-based systems, and the differences are, in CORBA-based systems, components in different programming languages may be encountered, and different parsers are needed. Another type of invocation events are asynchronous interface invocation, which perform through the CORBA Event Services event channel. Forturnately, OMG IDL also design the standard to specify which one is consumer and which one is producer and therefore, we can easily identify the invocation events under this circumstances. User action events . As to user action events, they will not be able to be sent across the network, and therefore, can be ignored. COM/DCOM Component Object Model(COM) is another software architecture that supports component interoperability across programming languages, platforms and implementations. DCOM(Distributed Component Object Model) extents COM by providing protocols to enable COM based components to commumicate directly with each other over the networks. Essentially, it makes network transparent to COM components. Interfaces . A central piece of COM is the type library. A type library is a binary description of the COM object or COM interfaces, or both, and is the compiled version of the Microsoft Interface Definition Languange(MIDL, which is a Microsoft extention to OSF-DCE IDL standard). Type library usually is part of the binary image of COM components, or should be deployed along with the binaries. From type library, it’s very easy to acquire COM component’s interfaces. Exception Events . Since there is very few explicit description of exception information in MIDL, the procedure to obtain exception events for a COM component is similiar to EJB, that if COM components’ source code are avaible, we can parse the source code to get the exception information; if not, we have to rely on specification. Invocation Events . In addition to the procedure similiar to CORBA and EJB, COM component might have another type of invocation event. In MIDL, a component can

have an event interface. The difference between a normal interface and an event interface is that the former is implemented by the component itself while the later is implemented by the other COM components connecting to it and the source component can ”fire” the specified event to the others. From the type library and the connection pattern, we can easily obtain the information for this type of event. User action events . User action events usually are handled by the COM component itself and thus can be ignored. In certain cases, one component can provide a callback handler to the other component and thus ”superclass” or ”subclass” certain events. This pattern is more common in components created with certain compiler/language, such as VC++, and less in the others. We have to rely on the source code parsing or specification to get the information in these cases. 4 Empirical study We conducted an empirical study to investigate the strength of each criterion we proposed in Section 3. We analyzed the common characteristics of most component-based software and classified the plausible faults in the componentbased software into three categories: Inter-component faults, interoperability faults and traditional faults. When developing and maintaining component-based systems, each individual component can be tested and maintained separately. While integrating a set of components, the evaluation of the interactions among the components will be the central issue. Faults related to the interactions among components can be classified into programming related, which are inter-component faults, and nonprogramming related faults which are interoperability faults. Other faults which reside within one component can very often be processed by using the traditional methodologies and therefore are classified into traditional faults. Type I - Inter-Component faults: In a component-based system, even though each individual component has been evaluated separately, when combining them together, failure may be encountered. Those programming related faults which are associated with more than one component will be considered as inter-component faults. For example, in the following code, after add i = 0 into I1 of Component C1 and test I1 and I2 , we cannot detect any problems. But when the component C1 is deployed and I1 is invoked followed by I2 in Component C2 and C3 , a failure will occur in C3 . C1 : I1 → i = 0; I2 → return i;

C2 :

I3 → C1 :: I1 ();

C3 : I4 → j = 1/C1 :: I2 ();

Type II - Interoperability faults: Many characteristics of component-based systems, such as heterogeneity, source code unavailability, reusability and etc, will generate different types of interoperability problems in component-based systems. The interoperability problems can occur at different levels, and from low to high, they may be classified into System level, programming language level and specification level interoperability faults. System level interoperability faults : In a componentbased system different components may be built under different infrastructures, for instance, different operating systems or different sets of system libraries. Programming level interoperability faults : When components are written in different programming languages, the incompatibility among the programming languages may cause the failure. For example, One of the incompatibility problems which is often encountered is due to the different float value processing of VC++ and VB. Specification level interoperability faults Specifications may be misinterpreted by the developers, and there are many different ways that the specifications may be misunderstood. • Data misunderstanding. That data which passes through the interfaces s misunderstood. For example, misunderstanding of the types and values of the input parameters and the return value and etc. • Control misunderstanding. The patterns of the interactions may be misinterpreted. This includes the misunderstanding of the execution sequences of the interfaces. For example, many components have an initiating interface which has to be invoked prior to any invocation of other interfaces, fail to do so may cause the failure. Or, misunderstanding the interaction mechanism, for example, if an interface requires an exception handler when exceptions are generated, the software will crash when the exception handler is missing. Type III- Traditional faults and other faults: For those faults which can be isolated within one component, traditional testing and maintenance techniques can be adopted. These faults will be identified as traditional faults. Other

# of Components MFC 5 System ATL 5 Visual Basic 1 SubTotal 11 Third Party 7 Total 18 Total Test Cases 1388

# of Interfaces 46 316 34 396 641 1037 Total Faults

# of Events 4 124 1876 2004 241 2245 95

test cases Faults Reported Functional Testing All-Interfaces All-Events All-Context-Sensitive / Some-Content-Sensitive / dependences

1388 76 228

I 33 27 7 19

II 38 29 12 24

III 24 16 6 15

Total 95 72 25 58

571

27

29

22+3

78+3

Table 3: Faults detection summary Table 1: System Information Summary

Version 1

Type I 33

Type II 38

Type III 24

Total 95

Table 2: Number of Faults Detected

faults, such as faults related to special input or special execution environments will also be classified into this category. Experiments To investigate the presence of faults in real component-based applications, we conducted an empirical study on an industrial system obtained from a company located in New York City. The program we used is a subsystem of a trading assistant software. Adopted by more than 100 companies and 4000 individual users, this system provides various models to help users in summarizing, visualizing, analyzing and predicting from the stock data. The math libraries of the system were implemented using both C and C++, while its GUI was implemented using C++ and VB. In addition, many third party components are used in the system. The software has been delivered for more than a year, and it has a total of 18 components, 1037 interfaces and exposes 121 different events. This information is summarized in Table 1. During the quality assurance(QA) testing of the software, 72 faults were detected. After the system was delivered, 23 more faults were corrected in the maintenance phase. We analyzed the bug reports and source code of the system, and summarized the distribution of the three types of faults in Table 2. The result shows that about 35% of these faults belong to Type I; 40% are Type II and 25% are Type III faults. Some Observations 1. From the data shown in Table 1 we can see that third party components play a significant role in the component-based system. Usually, the source code of the third party component is unknown, and therefore, many traditional testing and maintenance methodologies are not appropriate. 2. Incompatibility-related faults cannot be ignored in testing component-based systems. Some of these faults are due to the incompatibility of the development environment; others are due to the integration of third party

components. 3. Functional testing, which is used by the QA teams, can only provide a certain level of software reliability. In the component-based system which we investigate, only 75.8% faults are detected. More rigorous approaches must be applied before the the software is delivered. Two experiments were conducted in parallel: one applies the proposed strategy and the other uses the traditional functional testing. During the testing with functional testing strategies, 1388 test cases were executed, and 72 out of the 95 faults were detected. To validate the effectiveness of our strategy, we first selected test cases to cover all-interfaces, all-events criteria. By generating 76 and 228 test cases, 25 and 58 faults, respectively, were detected. Furthermore, to satisfy the all-context-sensitive / some-content-sensitivedependences criterion 571 test cases were executed, detecting 81 faults. Among these 81 faults, 78 of them were originally detected either by QA teams or customers, 3 new faults were detected by our strategy. Table 3 summarizes the test results obtained from these two experiments. From the data shown above, we derive the following observations: 1. Test case selections based on the all-interfaces, allevents criteria are simple and efficient, however they can only provide a certain level of reliability. To further improve the quality of the system, all-context-sensitive/ some-content-sensitive-dependences criteria are necessary. As shown above the all-context-sensitive/ somecontent-sensitive-dependences criteria is capable of detecting the majority of these faults, thus, even though the all-paths criterion can be used to detect more faults, more experiments are needed to investigate the efficiency of the technique to be useful in reality. 2. The result demonstrates that after enforcing the allcontext-sensitive/ some-content-sensitive-dependences criterion, we not only detect 84% of faults which have been detected, we also find 3 new faults in the system. Moreover the test effort we spend is only 41% of the functional testing approaches.

5 Conclusions and Future Work We have presented a new approach for testing componentbased software. Our empirical studies show that testing component-based software is necessary yet expensive. The technique we proposed provides several criteria for determining test adequacy. The all-context-sensitive / some-contentsensitive-dependences criterion used only 41% of test cases yet detected 84% of the faults; even the weakest criterion, the all-interface, used 26% of test cases and detected 26% of the faults. Therefore, the strengths of the technique can be expected. Our method, which can be applied on all types of component-based systems, does not rely on the knowledge of source code. Our on-going research directions on this topic are the development of a tool to support automation of the technique and enhancement of the technique for resolving problems caused by a distributed characteristic, such as synchronization. REFERENCES [1] M. Arnold, M. Hsiao, U. Kremer, and B. G. Ryder. Instruction scheduling in the presence of java’s runtime exceptions. In Proceedings of the 12th Workshop on Languages and Compilers for Parallel Computing (LCPC’99), August 1999. [2] D. Barrrett, L. Clarke, R. Tarr, and A. Wise. A framework for event-based software integration. ACM Transaction on Software Engineering and Methodology, 5(4):378 – 421, 1996. [3] B. Beizer. Software Testing Techniques. Van Nostrand, 1990. [4] U. Buy and et al. A framework for testing objectoriented components. In First International ICSE Workshop on Testing Distributed Component-Based Systems, Los Angeles, 1999. [5] R. Chatterjee and B. Ryder. Data-flow-based testing of object-oriented libraries. Technical Report DCS-TR382, Rutgers University, 1999. [6] M. Chen and M. Kao. Effect of class testing on the reliability of object-oriented programs. In Proceedings of the Eighth International Symposium on Software Reliability Engineering, May 1997. [7] P. Clements. From subroutines to subsystems: Component-based software development. The American Programmer, 8(11), 1995.

[10] G. Fowler, D.G.Korn, and K.Vo. Principles for writing reusable libraries. In Proceedings of the 17th International Conference on Software Engineering, pages 150–159, 1995. [11] P. Frankl and E. Weyuker. An applicable family of data flow testing criteria. IEEE Transactions on Software Engineering, 14(10):1483–1498, 1988. [12] E. N. G. Cugola and A. Fuggetta. Exploiting an eventbased infrastructure to develop complex distributed systems. In The 20th International Conference on Software Engineering, Kyoto, Japan, April 1998. [13] D. Garlan, R. Allen, and J. Ockerbloom. Architechtural mismatch or why it’s hard to build systems out of existing parts. In Proceedings of the 17th International Conference on Software Engineering, pages 179–185, 1995. [14] S. Ghosh and A. Mathur. Issues in testing distributed component-based systems. In First International ICSE Workshop on Testing Distributed Component-Based Systems, Los Angeles, 1999. [15] M. Harrold, D. Liang, and S. Sinha. An approach to analyzing and testing component-based systems. In First International ICSE Workshop on Testing Distributed Component-Based Systems, Los Angeles, 1999. [16] M. Harrold and G. Rothermel. Performing dataflow testing on classes. In Proceedings of the Second ACM SIGSOFT Symposium on Foundations of Software Engineering, pages 154–163, December 1994. [17] M. Harrold and M. Soffa. Interprocedural data flow testing. In Proceedings of the Third Testing, Analysis, and Verification Symposium, pages 158–167, December 1989. [18] M. Harrold and M. Soffa. Selecting data flow integration testing. IEEE software, pages 58–65, 1991. [19] M. Heimdahl, J. Thompson, and B. Czerny. Specification and analysis of intercomponent communication. IEEE Computer, 32(4):47–54, April 1999. [20] S. Horwitz, T. Reps, and D. Binkeley. Interprocedural slicing using dependence graph. In Proceedings of the ACM SIGPLAN’88 Conference on Programming Language Design and Implementation, pages 35–46, Atlanta, 1988.

[8] J. Cook and J. Dage. Highly reliable upgrading of components. In International Conference on Software Engineering, pages 203 – 212, Los Angeles, 1999.

[21] S. Horwitz, T. Reps, and M. Sagiv. Demand interprocedural dataflow analysis. In 3rd ACM SIGSOFT Symposuim on the Foundations of Software Engineering, October 1995.

[9] J. W. Duran and S. C. Ntafos. An evaluation of random testing. IEEE Transactions On Software Engineering, SE-10(4):438–444, July 1984.

[22] W. E. Howden. Functional program testing. IEEE Transactions on Software Engineering, SE-6(2):162– 169, 3 1980.

[23] T. Jeron and et al. Efficient strategies for integration and regression testing of oo systems. In Proceedings of the Tenth International Symposium on Software Reliability Engineering, November 1999. [24] D. Kung and et al. Developing an object-oriented software testing and maintenance environment. Communications of the ACM, 38(10):75–87, 1995. [25] J. Lang and D. Stewart. A study of the applicability of existing exception-handling techniques to componentbased real-time software technology. ACM Transaction on Programming Language and Systems, 20(2):274– 301, 1998. [26] G. Larsen. Designing component-based frameworks using patterns in the uml. Communications of the ACM, 42(10):38–45, 1999. [27] K. Maruyama and K. Shima. New class generation mechanism by method integration. In Fifth International Conference on Software Reuse, 1998. [28] M. Mezini and K. Lieberherr. Adaptive plug-andplay components for evolutionary software development. In Proceedings of the conference on Objectoriented programming, systems, languages, and applications, pages 97–116, 1998. [29] J. D. Musa, A. Iannino, and K. Okumoto. Software Reliability: Measurement, Prediction, Application. McGraw-Hill, New York, 1987. [30] O. Nierstrasz, S. Gibbs, and D. Tsichritzis. Component-oriented software development. Communications of the ACM, 35(9):160–165, 1992. [31] S. Rapps and E. Weyuker. Selecting software test data using data flow information. IEEE Transactions on Software Engineering, 11(4):367–375, 1985. [32] M. Robillard and G. Murphy. Analyzing exception flow in java programs. In Proceedings of the 7th European Engineering Conference held jointly with the 7th ACM SIGSOFT symposuim on Foundations of software engineering, pages 322–337, 1999. [33] D. Rosenblum. Adequate testing of component-based software. Technical Report TR97-34, University of California at Irvine, 1997. [34] D. Rosenblum and E. Weyuker. Using coverage information to predict the cost-effectiveness of regression testing strategies. IEEE transaction on Software Engineering, 23(3):146–156, 1997. [35] G. Rothermel and M. Harrold. A safe, efficient regression test selection technique. ACM Transactions on Software Engineering and Methodology, 6(2):173– 210, April 1997.

[36] G. Rothermel, M. Harrold, and J. Dedhia. Regression test selection for c++ software. Technical Report TR99-60-01, Oregon State University, 1999. [37] B. Ryder and et al. A static study of java exceptions. In 9th International Conference on Compiler Construction, March 2000. [38] S. Sinha and M. Harrold. Analysis of programs with exception-handling constructs. In IEEE International Conference on Software Maintenance, Bethesda, Maryland, November 1998. [39] K. Sullivan and D. Notkin. Reconciling environment integration and component independence. In Proceedings of the fourth ACM SIGSOFT symposium on Software development environments, pages 22–33, 1990. [40] J. Voas. Maintaining component-based systems. IEEE Software, 15(4):22–27, July/August 1998. [41] E. Weyuker. Testing component-based software:a cautionary tale. IEEE Software, 15(5):54–59, September/October 1998. [42] L. White and H. Leung. A firewall concept for both control-flow and data-flow in regression integration testing. In Proceedings of International Conference on Software Maintenance, pages 262–271, 1992. [43] Y. Wu and M. Chen. Testing and maintaining component-based software. Technical Report TR0002, State University of New York at Albany, 2000. [44] Y. Wu, M. Chen, and M. Kao. Regression testing on object-oriented programs. In Proceedings of the Tenth International Symposium on Software Reliability Engineering, November 1999.