An Evaluation

of Software Test Environment

Nancy S.

Eickehnann

Information

and

and Computer

University Irvine,

Debra

California

[email protected]

J. Richardson

Science

Of California

Department

Irvine

92715

USA

and [email protected]

Abstract

tating evaluation of developers claims concerning of STlk

So~are Test Environments (STES) provide a means of automating the test process and integrating testing tools to support required testing capabilities across the test process. Specifically, STES may support test planning, test management, test measurement, test failure analysis, test development, and test execution. The software architecture of an STE describes the allocation of the environment’s jimctions to spec.i$c implementation structures. An STE’S

Developing sophisticated testing capabilities through composition of less complex, more general components proved to be an extremely effective approach, it facilitates rapid prototyping and lower development costs for new tools as well as tool integration [17]. It has been demonstrated be evaluated

through

architecture,

where

structure

ing

activity

ally it is highly

error-prone,

Software Testing

in the development

time consuming

Environments

Environment

and costly.

and Oracle

is defined

testing tools to support a wide range

ment

the critical

of

quality

capabilities quently

made

a given

How drawn depiction

software,

has keen largely among

[6, 12, 14,5, ports

importance

rigorous

evaluation Comparisons

STES

18]. These

using

a taxonomic

illustrate

whether

task or has a specific

attribute.

well

an STE

meets

from

textual

descriptions.

complements

0270-5257/96 Proceedings

its intended

textual

$5.0001996 of ICSE-18

A

2. Create

IEEE

with

in

this

(Prolog with

Test

halysis

Integrated

to describe

that of [10],

a canonical

Test

SW

where

SAAM

steps:

functional

tem’s

their

the

structural

4. Choose

diagram

the SAAM

3. Allocate

partition

for

of each system’s

graphical

functional

struc-

notation.

partition

onto

each

sys-

decomposition.

a set of quality

attibutes

with

which

to

which

test

the

assess the architectures.

approach sup-

5. Choose

vides

graphical

a set of

quality

6. Evaluate

is usually

thereby

II

(Testing

used

and analyz-

is used

(CONVEX

of the following

a graphical

ture using

desired goals

is consistent

as consisting

1. Characterize

are fre-

a system

uniform

descriptions,

of

terminology

[8,10]. (SAAM)

the domain.

of S333s in the develop-

ignored.

TAOS

The

in this paper

the test pro-

II),

and CITE

Environment).

of test capabilities. Industrial use of STES provide significant benefits by reducing testing costs, improving test accuracy, improving software quality, and providing for [17, 20]. test reproducibility Despite

Version

SAAM

STF!S: PROTest

Support),

depiction

Method

for describing

[10].

three

of both system

such an analysis

method

can

software

graphical

Analysis

architectures

to examine

manu-

(STES) overeome the defi-

ciencies of manual testing through automating cess and integrating

paper

facilitates

an established

software

consists

A uniform

Architecture

qualities

of the system’s

an architecture

architectures

Software

provides

1 Introduction

that some system

an analysis

and functionality.

of system The

of high quality software. When testing is performed

qualities

Examples of typical claims are

The rule-based approach offers many advantages over more traditional test execution systems...which require that each test case carry with it a script or a program to perform the necessary executions and results verification [20].

architecture can facilitate or impede modifications such as changes to processing algorithms, data representation, or jimctionality. Perjorrnance and reusability are also subject to architecturally imposed constraints. Evaluation of an STE’S architecture can provide insight into modifiability, extensibility, portability and reusability of the STE. This paper proposes a reference architecture for STES. Its analytical value is demonstrated by using SAAM (So@are Architectural Anulysis Method) to compare three sojlware test environments: PROTest II (Prolog Test Environment, Version II), TAOS (Testing with Analysis and Oracle Support), and CITE (CONVEX Integrated Test Environment).

Software testing is a critical

Architectures

the degree

support

353

tasks

to which

for each task,

tion of the desired

facili-

concrete

attributes.

quality.

each architecture

thus indicating

pro-

satisfac-

To accomplish

this analysis, the SAAM

three perspectives: a canonical functional domain, system structure, and allocation to system structm. The tit

perspective,

the application

method

may be provided

2 Architectural partition

of

In this paper, the three STEs are presented as originally and then their system stmctum

A

can

then

be

attributes.

To ascertain tasks

absence

of

with

whether

quality with

partition

to

in the

quality

has a particular

quality,

The

supporta

analysis specific

attributable

partition

tbe functional

of

the application

perspective

required

or

through field.

or

extensive domain

analysis by researchers in the

Software Test Euvironmenta do not yet have a reference architecture to characterize their domain specific function-

to the sys-

tem.

ality. A domain analysis is needed to accomplish

In section 2, the three architectural analysis perspectives partition

am described.

The canonical

specific testing activities

functional

and test process automation.

notation

STEa pose a significant

that is

ing the architecture

for evaluat-

over the years the test process has increased

of STES.

the quality

of reusability

challenge for domain analysis.

as well as the evolution of the test process that should be automated by an STE. Gelperin and Hetzel point out that in scope

across the life cycle and has been refocused as to ita primary goals and objectives [9]. STH.S have not all kept up with the state-of-the-art in test process or its automation. To describe the STE’S process focus, we provide a history of test process evolution.

Section 3 describes each of the three STES to be evaluated PROTest II, TAOS, and CITE. Each is first described as originally diagramed and discussed by the authors and then nxast in the graphical notation used by SAAM along with an allocation of the canonical functional partition. Section 4 delhes

of that pro-

This is in part due to the rapid advances in test technology

used by SAAM to describe system structure is provided. Finally, the allocation of functionality to system structure is described. This section sets the groundwork

in an ideal test process

cess.

This results in a reference

for STES. Them the graphical

inclusive

[9] and the functions specific to the automation

is a result of a domain analysis of the test process

architecture

this. The

domain analysis and resulting canonical functional partition for STE5 requires that two aspects be evaluated the

1.1 Overview used by SAAM

by

ment Environments), provide fictional characterizations for their domains. In mature domains such as these, a canonical functional partition has been accomplished

detertasks

functional

provides

tems) and the Toaster model for SDES (Software Develop-

specific

STE.

the qualities

is then

the presence

Perspectives

SAAM. Reference architectures, such as the Arch/Slinky Mets-model for UIMSS (User Jnterface Management Sys-

The architecture

to demonstrate

the architecture

not in accordance

respect

if a system

are chosen

that

functional

decomposition.

evaluated

concrete mines

the canonical

to its structural

canonical

domain

the analysis. For each STE,

Analysis

2.1 Canonical Functional Partition

is recast in the

SAAM architectural notation. Using a uniform notation permits a common level of understanding on which to base

allocated

as an analysis method for software

Using SAAM, architectural analysis is approached from three perspectives: a canonical functional partition, system stmcture, and allocation of functionality to system structure. This section describes these three perspectives.

by a reference

architecture. If one is not available, a domain analysis can be done to determine the necessary partitioning of functionality. published

of SAAM

architectures.

partition of the of functionality

a canonical functional

domain,

evaluation

takes

Test process evolution

of test arti-

is shown in figure

process began with a debugging

1. The test

focus in the early 50’s,

facts analyzed in this paper. The test artifacts and the functional partition to which they are relevant are deactibed. Each of the three STES is then evaluated with respect to test artifact reusability.

where the process took an ad hoc approach of simply finding bugs. The test process then evolved to a demonstration

Section 5 concludes with a summary of results and contributions as well as a discussion of future work.

the destruction

Specific contributions of the paper include an initial reference amhitectum for So&vare Test Environments (STEs), a comparison of three WE-s using the Software an evaluation of architecArchitectural Analysis Meth@

ing.

tural constraints on test artifact reusability,

[13]. The evaluation

period, which focused on making sure the program ran and solved gram

the intended to tid Next,

integration throughout

354

which

failures

This

period

focused

through

of

methods

the software

to

period

provide

lifecycle,

Processing

was followed

on executing

as described

Systems

test-

addressed

evaluative

(FIRS)

by

a pro-

implementation-based

an evaluation-oriented

eral Information

and further

problem.

period,

the

support in the Fed-

guidelines

process focuses on detecting require

ments, design, and implementation

faults, which requires

an early life cycle testing effort. The current test process is

Test Process Evolutiou

canonical

a life cycle prevention model, which focuses on fault prevention through parallel development and test processes.

Functional Partitions

The change of emphasis has been from detection of faults to one of prevention of faults. 1950-1956 1957-1978 1979-1982 1983-1987 1988-1995

The The The The The

Debugging-Oriented Period Demonstration-Oriented Period Destruction-Oriented Period Evaluation-Oriented Period Prevention-Oriented Period

Debugging ........................ ...................... Demonstration

Figure 1. Test process evolution, adapted from [9] Our domain

analysis

evaluated

cess, test process automation,

the software

and current ST&.

....................................

test pro-

Destruction

The anal-

ysis resulted in a partitioning of the STE domain into six canonical functions: test execution, test development, test failure analysis, test measurement, test msnagemen~ and test planning. Test Execution

●

mented

source

and recording

of

Evaluation

Test Management

includes the execution of the instrucode

...........................

Test Measurement

— \

execution

traces. Test artifacts ntcorded include test output resuha, test execution traces, and test status.

Test Plaoning

/ /

................. Prevention

\ ~----------

● Test Development includes the specification and implementation of a test cotiguration. This results in a test suite, the input related test artifacts, and documentation. Specific artifacts developed include test

Figure 2. STEP Model

oracles, test cases, and test adequacy criteria.

tition resulting f%om the STE domain analysis provide

Test Failure

●

Analysis

and documentation

The test process evolution

includes behavior verification

for the Software

Test Environment

parthe

Pyramid

of test execution

(STEP) model. The STEP model, shown in figure 2, strati-

pass/fail statistics. Specific artifacts include pass/fail state and test failure reports. ● Test Measurement includes test coverage measure-

fies test functionalities from the apex of the pyramid to its base in a corresponding progression of test process evolu-

ment and analysis.

and analysis

foundation

and canonical functional

Source code is typically

tion as described in figure 1. Each period represented in the pyramid includes the functionalities of previous peri-

instru-

mented to collect execution traces. Resulting test artifacts include test coverage measures and test fail-

ods as you descend from the apex to the base.

ure measures.

of test execution. Test execution is clearly required by any

●

Test Management

persistence, execution

artifact

The top section of the pyramid

includes support for test artifact relations

state preservation.

persistence,

represents the function

test process. The test process focus of the debugging-oriented period was solely on test execution.

and test

The second segment of the pyramid,

Test process automation

from the top, is

divided into two scalene triangles. The smaller scalene triangle representa test development. The larger scalene tri-

requires a repository for test artifacts. A passive repository such as a file serves the basic need of storage. However, an active repository is needed to support relations among test artifacts and provide for their persistence. ● Test PZanning includes the development of a master

angle represents test failure analysis. The relative positions and sizes have semantic significance. Test development played a more significant role to the overall test process during the demonstration-oriented and destruction-oriented periods due to the manual intensive nature of test development at that time. Test development methods have not significantly changed, although they have improved in reliability and reproducibfity with automation. Thus, their role in test process has diminished in

test plan, the features of the system to be tested, and detailed test plans. Included in this function are risk assessment issues, organizational training needs, required and available resources, comprehensive test strategy, resource and staffing requirements, roles and responsibility allocations, and overall schedule.

significance

355

as you move ahead in test procxws evolution.

Test failure analysis was less important manually,

as interactive

checking

when performed

Components

benefit for test behavior

verification.

to test failure

analysis

have increased

sophistication,

making

The methods applied

test failure

in their level

are

specification-based

test

oracles

I

of

o

analysis more signifi-

cant to the overall test process. One of the most significant advances

which

enables early entry of the test process into the life cycle. This is a key difference in test process focus as it progresses towards the prevention-oriented period.

Computational Component

I

[

period,

(3

which represents the point of

departure from a phase approach to a life cycle approach, A significant

change in the test process focus is that testing

is applied in parallel to development of development. and improving

Test measurement

reused. Test management The base, or foundation,

is critical

Software

Active Data Repository

of the pyramid

Test Environment

is test planning.

Pyramid

presented here is an initial

to refine and improve

recognizes a number of distinct design levels, each with its own notations, models, componentry, and analysis techniques. A more detailed and complex lexicon would be required for a more focused analysis. This is not within the scope or objectives work.

reference

attempt at a dia-

2.3 Allocation ture

the STEP model and wel-

of this paper but is planned for future

of Functional

The allocation

Partitions

of domain functionality

to Struc-

to the software

structure of an STE completes the graphical representation of the STE. The allocation provides the mapping of a system’s intended functionality to the concrete interpretation in the implementation. SAAM focuses most closely on

of the term architecture, canonical functional partition is used in the remainder of the paper in deference to reference architecture.

allocation

SAAM Structure

as the primary

differentiating

factor

among

architectures of a given domain.

Structure represents the decomposition and their

by of

essary level of detail for a system level comparison. The field of software architecture, not unlike hardware design,

come other researchers to do the same. To avoid overuse

components

Uni/Bi Directional

the notation is achieved by abstracting away all but a nec-

for test process repro-

grammatic representation of the canonical functional partition of the Software Test Environment domain. We will

2.2

+

SAAM’S goal is to provide a uniform representation which to evaluate architectural qualities. The simplicity

and

Test planning is the essential component of the preventionoriented period, Test planning introduces the test process before requirements, so that rather than being an afterthought, testing is preplanned and occurs concurrently with development.

continue

Control Flow

Figure 3. SAAM graphical architectural notation

ducibility.

The

Passive Data Repository

the test process.

that is created and must be stored, retrieved,

architecture

Uni/Bi Directional

➤

4

not merely at the end also enables evaluating

Approaching the base of the pyramid, the fourth segment representa test managemen~ which is essential to the evaluative test process due to the sheer volume of information

Data Flow 4

Test measurement is represented by the third segment in the pyramid. Test measurement is required to support the evaluation-oriented

Connections

by humans added little

interconnections.

of the system The

3 Architectural

graphical

notation used by SAAM is a concise snd simple lexicon, which is shown in figure 3. In the notation used by SAAM

In this section the domain

there are four types of components: a process (unit with an independent thread of control); a computational component (a procedure or module); a passive repository (a file); and an active repository (database). There are two types of connectom: control flow and data flow, either of which may be uni or hi-directional.

Descriptions we evaluate

of Software

We &-at duplicate components

referred Next,

to the

the system in this

authors’

level

by their

from

diagram

papem

for

graphical

diagram

respective

authors.

briefly.

The reader

detailed

we recast each STE in the graphical

SAAM.

356

three test environments

Test Environments.

for each STE as published discuss

of STES

We is

information.

notation

osed by

Here, we name components of all three systems of stillar

P

functionrdity

DLT/1 Program

the same and indicate descriptions

narned components.

The re-characterization

tectures in a uniform, and common

structure

what components

the authora’ original

of the archi-

evenly abstract graphical

terminology

provides

a sound

diagram basis for

understanding and comparison. Each STE is evaluated in terms of ita stated goals and compared with the STEP model to determine its process focus. Finally, the three STE architectures are compared as represented in the notation used by SAAM.

Checker

3.1 Teat

PROTest

PROTest

II

11 Author’s

Description.

The PROTest

system, as depicted by [2], is shown graphically 4. PROTest v

u

description

\

Report

II is composed

of five components:

a test

of the system can be found in [2].

PROTest II SAAM Description and Functional Allocation. As shown in figure 5, PROTeat 11includes three of

(1 Test Rept

the canonical development,

functional

partitions

and test measurement.

test managemen~

and test planning

often typical of interactive

Figure 4. The PROTest II system, adapted from [2]

The”SAAM

test execution,

analysis,

are missing,

which is

test environments

re-characterization

test

Test failure [11].

of the PROTest 11 archi-

tecture facilitates determining g if system goals achieved. In particular, a major deficiency highlighted

,------------------,

, , , ,

------

------

;-------------------------------

-------------------------

,

Test Development

,-----~

----------------------------------

-------

Test Measurement

s I

4 --------------

t

\ ,

+ ---------------------

I 1

I t

---,

,

, I

I I

I ( 6 # 1

h

t

, 0 ,

, , , ,

# 1 # I 1 t I I I , t

o ,

,

0 $ ,

are is

#

Test Execution

I

, ,

I t

t b

r ,

1

I

t

1

I 1 t 4

I I 1 I

-----, t

I 8

I I

o I I

l ----------------------

II

in figure

driver, structure checker, test input generator, test coverage analyzer, and a test report generator. PROTest II also provides a graphical user interface. A detailed textual

Teat Report Generator

~

in

correspond to these like-

--------------------------------------

, ,--------------------------------------------------------

.

Figure 5. The PROTest II system structure and functional allocation through SAAM.

357

----------

J

II

TAOS -.,;

.-----

,..

.:. ;

I

1

=-----

: Teat

Developmen~-------------

..::. .+-

,.--”’-

m&

&’y&----M:me”t

. . . . .* Teat

Management S’=.+

Figure 6. The TAOS system, source[171 II Process Focus. The SAAM

PROTest that the PROTest II architecture

does not have structural

PROTest

separation between the functions of test development and test measurement. The test input generation process is interdependent with the test coverage process. This is not evident in the author’s diagram but is explicit in the SAAM architectural description. PROTest II accomplishes the function of test measurement by analyzing instrumented source code to determine what clauses will be covcoverage of clauses is reported

along with the test cases

The resulting

with

test development

srchitectme

as is clear in the functional

and test execution.

structural

a basis to compare

the STE

of the STEP Model.

implementation-based

entry

point

for

testing

PROTest

II system

The PROTest

II

supports

II does not support

development

life

3.2

cycle.

system,

may extend

However, mentioned

its range

diagram

of

composition. to the process II system the

life

is post-implementation.

and thus the test process

part of a larger

thus,

executing

[9]. PROTest

which

has highly coupled components,

the system’s

ports

failures,

that did not execute (called the failed to execute report). These structural choices create interdependencies of test measurement

It also provides evolution

PROTest

ered given the test cases that are chosen. The predicted

II highlights

The

a program

to cause

has a destructive

the test process the PROTest

supcycle

focus

across the II STE

but not described

is

in [2],

of support.

TAOS

TAOS Author’s Description. The TAOS system as depicted by [17], is shown graphically in figure 6. TAOS is composed of several components: a test generator, test criterion deriver, artifact repository, parallel test executor,

overlays in figure 5,

The objective of PROTest II is to provide a fully automated test environment for Prolog programs [2]. PROTest II does not achieve this goal but rather provides an interactive test environment that partially automates test execu-

behavior verifier,

tiou test development, and test measurement. The developers state that the strength of PROTest II as a system stems from the idea of delining coverage in real logic

and coverage analyzer. TAOS also inte-

grates several tools ProDAG ysis Graph), Artemis

(Program Dependenm

(source code instmmentor),

Anal-

and LPT

@@age processing Tools). The TAOS system is shown in figure 6. A complete detailed textual description of TAOS and its integration with the Arcadia SDE is given in [17]. Specification-based test oracles as automated by TAOS are described in [16].

programming terms, rather than adapting imperative pro@ amming ideas. It appears that this language focus led to coupling of components, since generic capabilities were not developed,

358

---------------------

.

\ Test Development

~

1

I

r -------------

, -----------------

--------1

------------

----

----------

*

, I , -------------------

,1

4

ProDAG

I \ ‘;:;:’; ,,:,,

$w+’:”:’::::i

:.:~&:**,;T::;:?;:’’:’; *:”,

;:f::j:;’,,

:-&*M;;,,;;::::[~’::;;,

.------:::::::-:::'_;:: (

~ml

-:-::

-_-------:::-----------------------------“’’”

)

‘g

!

Figure 7. The TAOS system structure and functional allocation through SAAM ADES

TAOS SAAM Description and Functional Allocation. As shown in figure 7, TAOS provides support for test execution,

test development,

Test planning,

egy identified

however, is

of building

not supported. As

evident

system, which supporta persisin an active reuositorv

The TAOS system was built using a development

test failure analysis, test mea-

surement, and test management.

objeet management

tence of data and data relations [19]. during the TEAM components

project

that provide

strat-

[4] with the goal

generic capabilities

and abstract interfaces to support varied testing activities in

the

SAAM

architectural

description,

and processes.

TAOS achieves loosely cnupled components through separation of concerns with respect to system functionalities. The composition of each functional partition is accomplished through the use of highly cohesive components. Test generation is a process with an independent thread of control and is not dependent on components from other functional areas. Test execution is achieved by an independent process for program and test measurement

execution.

are invoked

remain highly

cohesive componentry

tive functional

partitions.

Test failure

analysis

testing (by identifying

their respec-

The TAOS arehiteetare genetic capabilities

several

multipurpose

code affeeted by program changes). clearly

achieves its goal through

and abstract programmatic

TAOS Process Focus. TAOS

This high degree of separation of concerns is facilitated by the primary component integration mechanism, which is data integration. Data integration is facilitated by abstract programmatic

integrates

(LPT), which is also used by ProDAG and ,Artemis. Integration of generic analysis tools supports multiple test and analysis purposes. ProDAG, for instance supports the use of program dependence graphs for test adequacy criteria, test coverage analysis, and optimizing efforts in regression

by the test driver, but within

TAOS

components, including ProDAG, LPT, and Artemis. Effective data integration is achieved through a consistent internal data representation in the Language Processing Tools

supports

interfaces, specification-

bssed testing and thus an entry point into the life cycle after the requirements phase. TAOS suppoms a test process with an evaluative focus.

interfaces provided by the PLEI-

359

Test Coverage Anslysis ,- .-=--------------------

source Conml

.---z”--I

-z--z-,

---------

---------

-------,

#

.------

H-H

................................ ---------------D,--?---------------~-------=-J~~--------=

‘w=

I

!

*

i # I

xp. Files

1

TC

I

Figure 8. The CITE system, adapted from [20] mated, general purpose software. test environment.

3.3 _—— — - .- CITE

CITE Proce-= Focus. CITE supports implementationCITE

Author’s

Description.

The CITE system, as

depicted by [20], is shown graphically

baaed testing. CITE has a destructive

in figure 8 (only the

that is, it executes a program to &tect

components of CITE developed by Convex are shown). The CITE architecture consists of several components: a test driver

(’ID),

test coverage analyzer

(TCA),

test process focus failures,

which is a

limited scope for automation of the test process. However, all testing functionalities supported by CITE are fully automated,

test data

reviewer (TDReview and TDPP), test generator, and databases and a rule base. Additional tools have been inte-

3.4

STE Comparison

grated for use with the system. Specifically, the OCT test coverage tool expanda the typea of test coverage measured by the environment and the EXPECT simulator provides for special purpose interactive testing. A complete textual

process. The use of SAAM clarifies how well each STE achieves this goal and to what degree. SAAM uses a

description

canonical

The three STEa share stated goals of automating

of CITE is given in [20].

structure

CITE SAAM Description and Functional Allocation. The SAAM graphical depiction of CITE, as shown in fig-

for test execution,

test development

to characterize

the system

level. The functionalities

sup-

Test process focus was identified for each STE. PROTest II and CITE support implementation-baaed testiug

test failure

analysis, test measurement, and test management. does not support teat planning,

partition

ported and stmctural constraints imposed by the architecture are more readily identified when compared in a uniform notation.

ure 9, reflects the structural separation desired in a system that controls the test process automatically. CITE provides support

functional

at a component

the test

CITE

and have a destructive test process focus. This focus has a limited scope of life cycle applicability, as it initiates testing after implementation for the purpose of detecting failures. TAOS supporta specification-baaed testing and has an evaluative teat process focus. An evaluative test process focus provides complete life cycle support, as failure de-

CITE has achieved the desired functionalities through allocation to atructarally cohesive components. The process control exercised by the ride-base ia evidenced in direct control to each functional area. This clean separation of concerns supports the primary goal of a fully auto-

360

,Z-------------

.

------------,

,

, ,

Test ~ Measurement

--------------

ii

t # # I , -------------

,

Test

1$1

# 1 I

1 t I

Test

,.

~

~q

--------------

,

Test

:

/jl

,

1 I #

!-------------I

:

I # I

I

,

t I I

/----- -------!

,-------------.

!t N --------------------It ,-----------------------------------------------H

.,,

,

k————— “’,:’C?A4’, :,,:’” ~ ‘e’m “ ‘!: I [1 -------------------------------------------------------------------------------!

Figure 9. The CITE system structure and functional allocation through SAAM Test Measurement

tection extends ffom requirements

and design to code. Test

test development

process focus appears to dictate the scope of automation achieved across the life cycle and the types of functionality supported by an STE.

in PROTest II is tightly coupled with

and test execution. PROTest II only par-

tially automates measurement, specifically source code instrumentation is done manually. PROTest II addresses test measurement and test development by focusing on the language constructs of Prolog. The resulting monolithic,

Test planning is not supported by any of the three STES. Test planning functionality specifically supports a preven-

highly

coupled architecture

tative test process focus. We could not Iind an STE that

TAOS provide

explicitly

Some tools

CITE

supports block

as a separate issue but fail to integrate

gram

dependence

supports

address planning

a preventative

test artifacta with a parallel

process.

development

effort. A preven-

tative process focus may be best achieved through grating an STE with an SDE.

Test management

reflects this focus. CITE and

automation

of test measurement.

coverage and TAOS

coverage

as well

supports pro-

as statement

and

branch coverage.

inte-

Test Failure Aualysis be documented

Test Management is not supported in PROTest II. CITE and TAOS support test management with an active repository. TAOS provides support for object management and process management. CITE provides support for object management, rule-based test process templates and configuration management.

for full

requires that the expected output

for test case inputs.

The actual outputs

must be verified with respect to correct or expected behavior. Verification can be accomplished through a test oracle. The most accurate of oracles are specification-based test oracle~ the most error prone are interactive human oracles. Test failure analysis has increased in importance as the test process focus has changed fimn demonstrative to

is essential for full

evaluativ% it is also critical in achieving a preplanned preventative process. PROTest II does not support failure analysis, as it does not store expected outputw rather behavior verification relies on an interactive human oracle.

automation. The test process spans a temporal boundary of the life cycle and generates voluminous amounts of test artifacts with complex relations that must be supported by test management. Coordinating the test process over time is not possible without automated test management.

CJTE stores expected outputs with test case inputs, yet this

361

enhancements, and performance examined in future work.

approach still relies on a human as oracle. The process is not interactive, however, and is thus less error prone. CITE provides additional leverage for the human oracle in that

improvements

are to be

Component reusability is the attribute of facilitating multiple snd/or repeated use of components, This applies

stored expected outputs are reused extensively adding to the cotidence that they are correct. TAOS supporta specithe greatest

to both computational and data components. Wh.h respect to STES, we might evaluate reuse of the components in the

automated in PROTest II.

by the STE. Here, we choose the latter, and thus will eval-

It is an interactive process that manually develops an augmented Prolog program for test execution, manually instruments the source code, and manually checks the

uate each STB with respect to its support for reuse of the artifacts of the test process managed by the STE. An

fication-based

test oracles,

which

provide

degree of oracle accuracy and reproducibility. Test development

is partially

structure report. CITE and TAOS provide support for test development tion. Test execution

including

STE or reuse of the components managed and manipulated

test data genera-

automate the test process. Note that some of these artifacts

software

are indeed computational

is ftdly automated by all three STBS.

All thee STES share the goal of providing mated

STB’S ability to store, retrieve, and reuse test plans, test scripts, test suites, test cases, test criteria, test oracles, test resulta, and the like is central to their ability to effectively

full automated

test environment.

achieve this goal. The author’s

PROTest

a fully auto-

Reuse is further

II does not

of highly

claim their focus on lan-

cohesive components.

and provides

cycle with an evaluative sis is optimized

broader

advantage of reducing

TAOS

test effort.

modi@ the test artifact

as verbatim

Leveraged

or

reuse is of and

than to simply reproduce it.

The next section evaluates test artifact

support across the life

reusability

for

each of the three STES. The presence or absence of the

process focus. Test failure analy-

quality

in TAOS through the use of specification-

of ~usability

in PROTest II, TAOS, and CITE is

evaluated in regards to the STE’S support for test artifact reusability. 4.1

process control. Analyzing

by [Bie91]

value as long as it takes less effort to locate, select

based test oracles. CITE meets its goal of full automation. A particular strenglh of the CITE STE is its provision for automated configuration management (CM) and versioning control, Automated CM is foundational to successful

4

delineated

tion to the code. Leveraged reuse is reuse through modi~ing some portion of the code. Verbatim reuse has the clear

and CITE have highly cohesive components that are loosely coupled. TAOS clearly achieves its goal of full automation

(e.g., test scripts)

leveraged reuse. Verbatim reuse is reuse without modifica-

guage constructs is a strength despite the architectural implications in lack of support for separation of concerns and development

components

while others are data components.

Reusability

of Test Artifacts

in PROTest

II.

PROTest II does not support test artifact reuse. The lack Software

Architectural

of provision for test management is a primary factor in this deficiency. Test management is necessary to support artifact persistence and thus reuse.

Quali-

ties The goal of a software

development

The structure of PROTest II also impedes test artifact

process is to pro-

reusability.

duce software having a specified set of qualities, hopefully

Test inputs are generated based on test cover-

age. Test execution results provide clauses actually covered in the execution. Failed cases are test cases that did not execute. This functional interdependency among structural components inhibits test artifact reuse. The high degree of artifact interdependence impedes reuse as well.

quantifiable. The eleven ...ilities of software quality, delineated by [3], are correctness, reliability, efficiency, integrity, usability, maintainability, flexibility, testability, portabdity, reusabilhy and interoperabilhy. These qutilties are also desirable for Software Test Environments. Software architectural analysis is used to determine if a specific structural decomposition and the functional allocation to system structures supports or irnpedos certain qualities. Parnas identified changes to data representation

test artifact

relations

and changes to processing algorithm

TAOS

management

as particularly

4.2

Reusability

TAOS

sensi-

tive to system architectural constraints [15]. Gsrlan, Kaiser, and Notkin further identified enhancements to system functionality, improvements to performance (space and time), and reuse of components [7]. This paper examines software architectural constraints in relation to reusability. An examination of system modifications, functional

supporta

test

store, retrieve, quately

362

As

test artifact through

TAOS

graphs

for

quacy

criterion

many

uses,

provides

support

F%oDAG including

test

and maintains

object

repository.

the

Thus,

ability

TAOS

to ade-

reuse.

provides

and

persistence

test artifacts.

test artifact

an example,

in TAOS.

an active

support

and modi@

supports

Primarily, ability.

of Test Artifacts

coverage

for

verbatim

provides derivation analysis.

reus-

dependence of test

ade-

The

same

dependence

graphs can be reused iu debugging,

code slices leading

to a failure

can be determined.

where

recognizes the importance

These

STEP also provides

same graphs can be used later in the life cycle to facilitate software

maintenance,

where

code slices affected

of test process focus for STG.

insight

into an STE’S range of life

cycle support in the test process focus. The STEP model’s integration of process evolution recognizes the importance

by a

change can be determined analyzed and related test artifacts reused for regression testing. Thus, we see that

of test process automation as well as ita evolutionary nature in applicability across the life cycle,, The STEP

TAOS provides

model attempts to capture the complexity

for reuse in diverse functional

capacities

as well as in disbursed temporal contexts.

automation explanatory

TAOS also provides support for leveraged reuse as persistent test artifacts may be modified for use in slightly different contexts, such as during software evolution and

Reusability

of Test Artifacts

CITE was the most powerful environment

reuse in CITE includes reuse of test rides

test suites for a family

of related compiler

nificant

Test Artifact

Reusability.

Test artifact reusability

is of

particular importance for ST13S.Reuse of test artifacts with little or no modification is critical to providing full aut~

Work for future evaluation

systems using the architecture,

for automated

test oracles to support test failure anal-

ysis.

mation across the test process and the software life cycle.

of

Reuse is also an effective

software testing environments by developing the STEP Model for STES, validating the model by representing three different

strength of TAOS is its provision

specification-based

and updates of previously

This paper laid the groundwork

component

for configuration management and versioning control. TAOS achieves its goal of a fully automated test environment and covers test activities across the life cycle. A sig-

released or tested code.

and Future

an automated

mated oracle support for test failure analysis could be improved. A particular strength of CITE is its provision

prod-

provides support for version control, which supporta reus-

5 Conclusions

provided

tools and generic

PROTest II provides a partially automated environment. CITE is indeed powerful and complete, although auto-

Verbatim reuse in CITE includes the reuse of entire test suites as well as individual test caaes.The test management maintenance

intuitive

functionality.

ucts, where in each instance the rule required a simple modification allowing 909% of the test suite to be reused.

ing tests during

with

and complete automated test

in use, and TAOS

STE based on integrated

and test templates. The rules may be reused with slight modifications to execute tests including new capabilities. Test information can be modified for reuse by editing templates. As an example, Convex discusses the reuse of the compiler

of test process

were: the PROTest II focus on declarative language constructs would provide an advantage for such a system,

in CITE.

CITE provides extensive support for leveraged reusability. Leveraged

rich model

Comparison of Three STES. PROTest II, TAOS, and CITE all share the goal of a fully automated test environment. Claims made by the respective authors of the S’Ilk

maintenance. 43

in a semantically powers.

determinant

for delimiting

the

amount of information that must be stored and therefore managed by an STE. To support artifact reuse, an STE

comparing

three STEa using SAAM, and evaluating the architectural constraints for one quality attribute, test artifact reusability, for all three STES. This work has also provided insight into the application of SAAM for architectural analysis of

must provide test management utilities,

STES.

storage, retrieval, and modification for leveraged reuse. If this capability is not supported, leveraged reuse becomes

STEP Model. (STEP) model

The Software Test Environment was introduced

attempt at providing

Pyramid

persis-

more costly and possibly prohibitive.

in section 2 as an initial

a much needed canonical

specifically

tence of test artifacts and test artifact relations. Persistence alone, however, is not sufficient to support leveraged reusability. The artifacts must be configured to enable efficient

This paper examined PROTest II, TAOS, and CITE for

functional

partition for STEs in a semantically significant model that incorporates process evolution and focus. The six func-

the quality of reusability, in particular test artifact reusability. PROTest II does not support such reuse. TAOS and

tions identified

CITE support both verbatim and leveraged reuse. Support for test artifact reuse serves as the litmus test for an STE’S

- test execution,

test development,

test fail-

ure analysis, test measurement, test management and test planning - provide orthogonal partitiona for all STE func-

support for full automation life cycle.

tionality. The pyramid model provides semantic significance in the ordering of the sections through the test process evolution progression from apex to base. This model is unique in its incorporation of both functionality and teat process evolution.

The STEP model

of the test process across the

Evaluation of SAAM Method. The SAAM architectural analysis is approached from three perspectives: a canonical functional partition, system structwe, and allocation of functionality to structural decomposition of the

implicitly

363

References

system. SAAM uses a simple lexicon to describe all the STES in a uniform notation to depict their system structure so as to permit a common level of understanding. The systems are further evaluated in their support for some con-

[1] J.M. Beiman.“Deriving Measuresof SoftwareReusein ObjectOrientedSystems.”JtrProceedingsof theBCS-FACSWorkshopon Formal Aapcta of Measurement,pages63-83, SouthBank University, London,May 5, 1991. [2] F. Belli and O. Jack.“Jmplementstion-baaed analysissod testing of prelogpxugrsms.”JntheProceedingsof the 1993InternationalSymposiumon SoftwareTestingandAnalysis,pages70-80. Cambridge, Massachusetts, June1993. [3] J.P.CsvenoandJ.A. McCall. “A frameworkfor the measurementof softwarequaMy.” ht the proceedingsof the Software Qualb,yand AssuranceWorkshop,pages133-139.November1978. [4] L. A. Clarke,D.J.Richardson,endS.J.Zeil. “TearrxA supportenvironment for testing, evaluation, and analysis.” Jn Proceedingsof ACM SIGSOFT’88: Third Symposiumon SoftwareDevelopment Environments,pages153-162.November 1988.Appearedas SIGNotes 13(5). PLAN Notices 24(2) and Software Engineering [51 DaybreakTechnologiesInc., Datasourcessoftwareproductsguide.. CherryHill, New Jersey,DataSourcesInc., 1990. [6] R. A. Fairley. So&are testingtools. In ComputerProgramTesting, New York ElsevierNorth Holland, 1981. [7] D. Gerlan,G. Kaiser,andD. Notkin. “Using tool abstractionto composesystems.”IBHEComputer,vol. 25,June1992. [8] D. GsrlsnsodM. Shaw.“An introductionto softwarearchitecture.”. Advancesin Software Engineming and Knowledge Engineering, VolumeI, World ScientificPublishingCo., 1993. [9] D. GelperittandB. Hetzel.“The growth of softwaretesting.” Communicationsof the ACM, 31(6):687-695,June198S. [10]R. Kazman,L. Baas,G. Abowd, andM. Webb.“SAAM A method for analyzingthe propertiesof softwarearchitectures.”JnProceedings of the SixteenthInternational Conference.on Software.Engineering,81-90,Sorrento,Italy, May 21, 1994. [11]G. J. Myera.The art of acdlweretesting.New York, JohnWdey end sons, 1978. [12]E. Miller. MechanMmgsoftwaretesting.TOCG Meeting, Westlake VWage,California, April 15, 19S6. [13]Guidelinefor LifeCycleValidation,Verification,andTestingof ComputerSoftware.NationalBureauof StandardsReportNBS FIPS 101. Washington,D.C., 1983. [14]T. Nomura. “Use of softwareengineeringtools in Japan.”.Jn Proceedingsof the Ninth International Conferenceon Software13ngirteering,449,Monterey,California, March 1987. [15]D.L. Parnas.“On the criteriato beusedin decomposingsystemsinto modules.’’Communicatiooa of theACM, 15(12):1053-1058,December 1972. [16]D.J.Richardson,S.L. Aha, andT.O.O’Melley. “Specification-based teatoraclesfor reactivesystems.”In proceedingsof the Fourteenth JntemationslCottferettcoon SoftwareEngineering, 105-118,Melbourne,Auatrsti~ Msy 1992. [17]D. J. Richardson.“TAOS: Testingwith Oracleaand Analysis Suppmt.” JnProceedingsof the 1994InternationalSymposiumon Software Testing and Analysis, pages 13S-153. Seattle, Washington, August 1994. [181Sotlwsre Quality Engineering.Survey of Software Test Practices. Jacksonville,Florida SoftwareQuality Engineering19SS. [19]P.Tarr and L.A. Clarke.“An Object ManagementSystemfor SotlwareEngineeringEnvironments.”JrtACM SIGSOF’r’93: Proceedingsof the Symposiumon theFoundsdonsof SoftwareEngineering, Los Angeles,California, December1993. [20] P. A. Vogel. “An integrated generat propose automated teat environment.” In the proceedingsof the 1993JnternatiottrdSymposiumon SoftwareTestingandAnatysis,pages61-69.Cambridge,Massachusetts,June1993.

crete task that would determine whether or not the system has a quality such as modifiability or reusability. SAAM enabled our analysis of all three STES and revealed system strengths and weaknesses inherent to structural

decompositions

tional allocations, from the original

and choices

concerning

func-

The differences noted were not apparent author’s descriptions but were revealed

by using SAAM. A more detailed lexicon in SAAM would support more in-depth analysis of test artifact reusability. The SAAM lexical notation

supports analysis of data and control flow

among system components data relationships.

but does not address complex

Future work

will

look at augmenting

the SAW lexical notation to provide greater detail and support complex analysis of data components. Future Work. The STEP model reference architecture for STES is to be further refined. Specific improvements planned are adding a third dimension to the pyramid and providing

the required

semantic

closely correlate the canonical process evolution. Additional

to more

partitions

to test

STES will be compared to the current work

and evaluated insight into approaches.

significance

function

with

SAANL

WI%

beyond

This will that

provide

gained

by

additional taxonomic

The three STES analyzed in this paper will be further evaluated for architectural impact on enhancements to system functionality. Such analysis is supported by SAAM and will be the next area of investigation. In addition, system modifiability as evidenced in changes to data representation, optimization lexicon

processing

algorithms,

will be examined.

and

However,

performance

SAAM’S

simple

does not support such analysis and will therefore

require a more detailed lexical notation be developed.

Acknowledgment Thank you to the referees for their insightful suggestions and to Peter Vogel for his corrections to the CITE diagrams. This work was sponsored in part by the Air Force Material Command, Rome Laboratories, and the Advanced Research Projects Agency under Contract # F’30602-94-C-0218 and the University of California MICRO program and Hughes Aircraft Company under Grant #94- 105. The content does not necessarily reflect the position or policy of the U.S. Government, the University of California, or Hughes Aircraft Company, and no official endorsement should be inferred.

364