NSLoadGen– A Testbed for Notification Services Diego Palmisano and Mariano Cilia Argentina Software Development Center - Intel Corp. Cordoba, Argentina [email protected]

Abstract. During the past years a lot of work on Notification Services has been focused on features like scalability, transactions, persistence, routing algorithms, caching, mobility, etc. However, less work has been invested on how to evaluate or compare such systems. The selection of the most appropriate Notification Services for a particular application scenario is crucial and today available tools are bound to a particular implementation. If the Notification Service under test does not fulfill the application requirements then a new try with other Notification Service needs to be started from scratch: the description of the workload characterization and its injection cannot be reused. In this paper we introduce NSLoadGen (Notification Services Load Generator), a testbed platform that supports the definition of real-life scenarios, the simulation of these scenarios against notification services, and finally generating vast data that can be used to precisely evaluate it. NSLoadGen is not targeted at any specific Notification Services, but rather is generic and adaptable. It has been designed to support a wide variety of Notification Services characteristics, hiding the many differences among messaging products/specifications (e.g. Java Message Service [1]) and, at the same time, it is easily extensible to support new implementations. This paper covers the different steps the tool follows (scenario definition, scenario simulation and result collection), the proposed approach, as well as relevant design and implementation details. Keywords: Message-Oriented Middleware, Notification Services, Publish/Subscribe, JMS, Benchmarking, Distributed Testing, Simulation.

1

Introduction

Message-Oriented Middleware (MOM) refers to the software layer, between applications and the network protocols, that supports software engineers in developing distributed applications. Historically, MOM has been used to address issues related to heterogeneity, communication, and distribution of software components, relieving software engineers from the burden of solving low-level, network issues, such as lower-level communication protocols, concurrency control, transaction management, distributed object location, among others. Notification Services are a kind of MOM that implement the publish/subscribe paradigm: a message is published by a producer in the Notification Services. R. Meersman and Z. Tari et al. (Eds.): OTM 2007, Part I, LNCS 4803, pp. 790–807, 2007. c Springer-Verlag Berlin Heidelberg 2007 

NSLoadGen– A Testbed for Notification Services

791

Consumers express their interest by issuing subscriptions. The Notification Services is responsible for the transportation of the published messages to the matching consumers. Normally, Notification Services platforms provide several QoS alternatives/dimensions such as reliability, security, delivery order, transactions, and so on. In spite of the availability of standardized solutions such as CORBA-NS (CORBA Notification Service) [2] or JMS (Java Message Service) [1], new kind of these infrastructures continue to be developed to address the needs of novel applications. In the past years a lot of work has been focused on Notification Services features like scalability, transactions, persistence, routing algorithms, caching, mobility, etc. However, less work has been invested on how to evaluate or compare such systems. When surveying the existing implementations of standards (such as JMS) and publish/subscribe infrastructures, we found that they have several differences in the way they communicate with the client applications and the features they support. The selection of the most appropriate Notification Services for a particular application is crucial for its success. First of all, the workload characterization needs to be described and, afterwards, injected into the Notification Services under test. Unfortunately, available tools for this purpose are restricted in functionality and/or bound to a particular Notification Services. Consequently, if the Notification Services in question does not fulfill the application requirements, then a new try with another needs to be started from scratch. In this paper we briefly introduce NSLoadGen (Notification Services Load Generator), a testbed platform that supports the definition of real-life scenarios for Notification Services, the simulation of these scenarios against notification services, and finally generating vast data that can be used to precisely evaluate it. NSLoadGen is not targeted at any specific Notification Services, but rather is generic and adaptable. It has been designed to support a wide variety of Notification Services characteristics, hiding the many differences among messaging products (even the many differences among JMS implementations) and, at the same time, it is easily extensible to support new Notification Services. NSLoadGen allows a high level definition of scenarios, independent of the Notification Services under test: a scenario can be described and conducted against different notification services. Moreover, it supports distributed simulation making it possible to assign injection of messages and subscriptions to different Notification Services where all the NSLoadGen instances will be coordinated among them. The rest of the paper is organized as follows: related work is presented in section 2, focusing on Notification Services features relevant to NSLoadGen, benchmarking and testing of NSs. The proposed approach is described in more detail in section 3, presenting a brief description of the three steps supported by the platform. The main design decisions are sketched in section 4: how Notification Services independence was achieved, the architecture of producers and consumers and finally a description about the NSLoadGen distributed environment. Relevant implementation details are presented in section 5. Section 6 presents conclusions, open issues and areas of future work.

792

2

D. Palmisano and M. Cilia

Related Work

Since this work deals with various issues, like Notification Services platforms, distributed testbed, and benchmarking, this section serves as an introduction/overview of these topics including related work. 2.1

Notification Services

Our work relates to a subset of the features available in a Notification Services that are relevant in the context of load generation and injection. This features are: – Interface between Notification Services and Client Applications: a Notification Services offers services to client applications by means of functions. At least it exposes two functions: (i) subscribe allowing consumers express their interest; (ii) publish allowing producers inject messages. Several Notification Services platforms extend this interface with additional functions called advertise, which a producer uses to inform the messages it will generate, unsubscribe and unadvertise. Subscriptions can be matched repeatedly until they are canceled by a call to unsubscribe. Advertisements remain in effect until they are canceled by a call to unadvertise. – Addressing Models: Subscribers are usually interested in particular messages and not in all messages. The different ways of specifying the messages of interest have led to several distinct addressing models: channel-based addressing where the subscriber receives all messages that are posted to the channel; subject-based addressing [3] which is based on the notion of “topics” or “subjects” that represents a hierarchical organization of keywords to which participants can publish notifications and subscribe to; contentbased addressing which allows consumers to express interest on the content of the messages; and concept-based addressing [4] that allows consumers in an heterogeneous environment to express their interest considering their local context thus delivering to them as a result ready-to-process data which does not require further conversions. – Architecture: There are two major architectural alternatives: 1. Centralized: The Notification Services has one point where the matching of messages with subscriptions is evaluated, 2. Distributed: cooperating components (called brokers) evaluate the matching in a distributed way building a network of brokers. The centralized approach is simple to implement but it has a single point of failure and it could be a bottleneck. The distributed approach is more complicated to implement since a single global view on all subscriptions needs to be supported, but it scales much better. Actually, there are several research projects involving Notification Services (e.g. REBECA [5], JEDI [6], SIENA [7], HERMES [8], GRYPHON [9],

NSLoadGen– A Testbed for Notification Services

793

standard specifications (e.g. CORBA-NS [10], JMS [1]) and commercial products (e.g. TIB/Smartsockets and TIB/Rendezvous) that focus on different aspects of data dissemination, like efficient routing algorithms, optimal filter placement, minimization of resources usage, etc. Examples of JMS providers are FiornaoMQ and SoniqMQ (commercials products) and ActiveMQ, OpenJMS, JBossMQ, and Presumo (open source projects). 2.2

Benchmarking of MOM

Benchmarking Notification Services is a process that involves two main goals: validation of the service interface, and performance evaluation [11]. The choice of applications, configurations, and workloads is clearly crucial for the formulation of the benchmark. Part of the benchmark must be an expression of real, current applications. In addition, configurations and workloads must reflect real computing environments and actual use scenarios. Under the same scenario, the test system could be executed over several notification services, allowing to know how well they perform under the scenario in question. The results can be used to compare performance of different Notification Services solutions, helping in the selection process of a messaging system. Key areas in Notification Services benchmarking definitions are deployment topology (definition of distributed scenarios of consumers and producers), messaging domain (point-to-point or publish-subscribe systems1 ), duration, behavior of producers and consumers (how many messages, the distribution of injections), messages data and type, and so on. Furthermore, related to consumers, which messages they want to receive (i.e., subscription) and if they must be active at the moment of receiving messages are key points. In the literature there are research projects in the field of load injection in Notification Services, which are mainly focusing on specific Notification Services rather than general. Further, they almost do not provide a way to describe reallife scenarios. Sonic Software Test Harness [12] presents a simple test application that allows to create simple scenarios and test it over any JMS implementation. The behavior of TestHarness can be influenced by means of command-line parameters such as where is the JMS implementation running, number of producers and consumers to be created, number of connections to be used by producers and consumers, messages’ length and test duration. Beside that, producer or consumer behavior can not be defined at all. KOM ScenGen [13] relies on different steps in generating a scenario: topology creation, properties setting, network load or workload creation, plausibility check (where several things critical for the scenario can be checked for plausibility), scenario exportation to a collection of scripts and configuration files that are used to setup the scenario in a testbed, simulation and evaluation. Even thought it covers most of the necessary steps to follow a testing, the user has to have a deep understanding about the used technologies and settings. 1

Point-to-point products are built around the concept of message queues.

794

D. Palmisano and M. Cilia

Kuo and Palmer present a test harness that automates the testing of JMS implementations for correctness and performance [14]. The paper describes a methodology for black-box testing of a JMS provider. A test harness exercises the JMS server and a model of the expected behavior of this server is built; the model can then be compared against the actual behavior of the provider. However, like Sonic Software Test Harness, the user has not the possibility of setting producer/consumer behavior and workload characterization. The idea behind Service Integration Test Tool (SITT) [15] is to test services and their workflow by analyzing the message flows. In this way, message injections and receptions are logged in a standardized manner. As the previous JMS test harness, SITT supports distributed testing and the execution is managed by a master node. Further, it allows to define scenarios by means of writing XML scripts. However, it falls in the same deficiencies: no description of producers and consumers behavior, no characterization of workload, lack of using data services to fill messages out. The scenario descriptions of the tools mentioned above do not provide enough characteristics to be defined properly and, as a consequence, it is difficult to set up real-life behaviors. In addition, sometimes they can be defined with reduced power of expression, via command line arguments or directly hard coded in the application, having to recompile it every time a change is produced in the scenario. Besides that, producers are limited to simply inject messages as quick as possible with no chance to describe a pattern nor distribution of message injection. Messages are filled out with constant or even without data. Most commonly, consumers are restricted to subscribe to all messages and, as a consequence, the evaluation of the subscription functionality is left out. Regarding the result presented during the test execution, most of the test tools generate a fixed number of metrics without allowing to create new ones. Even, the results are represented as plain numbers in the execution console. Consequently, the evaluation/comparison of Notification Services becomes very difficult.

3

Proposed Approach

This section presents the approach followed by NSLoadGen. The idea is to provide users the ability to (quickly and easily) express their specific scenarios and, as result, obtain information related to important events carried out during the simulation of the scenario. An experiment (a.k.a., test run) can be simply formulated as a three-phase process that is described in detail below. 3.1

Scenario Definition

NSLoadGen supports the definition of scenarios by means of a description language. The scenario definition and the platform functionality has been separated and, thereby, behavior of producers and consumers is user-defined and independent of the logic of the platform. To adjust the scenario, the user only has to modify the scenario description and not the code of the platform. The tool allows the definition of a wide range of properties related to producers (where and when to inject messages), consumers (where and when to inject subscriptions)

NSLoadGen– A Testbed for Notification Services

795

Fig. 1. Load Description, Run Description and Configuration File

and messages (types, data and sizes), and assignment of them to distributed nodes in a distributed environment. Therefore, the scenario definition was separated into three parts, each of them containing related data that, together, shapes a real-life scenario. An example of the general structure of these three sections is depicted in figure 1. The first section defines data and behavior, containing producer, consumer and message definition, and how these three parts relate to each other. The second set up how producers and consumers are assigned to distributed nodes, i.e., binding information, and the last one is related to the configuration of the platform. The idea behind dividing the scenario description in three parts is not only to make its definition clearer and easier, but it is also founded in the fact that normally only the data and behavior is modified during several test runs under the same binding setup. For instance, once the network location of producers/consumers is defined, they are likely to be maintained unchanged while messages structure and its assignment to producers and consumers may vary to evaluate different executions. Conversely, if the test goal is to use the same load but creating more producers and consumers along different tests, data and behavior information will be maintained unmodified and the binding part would be modified. With respect to the configuration part, it is usually stable across test runs. This section will change only if a different Notification Services is going to be tested. 3.2

Scenario Simulation

One of the most important requirements related to the simulation of the scenario is that it can be driven in a distributed environment. This is one of the characteristics that allows to create realistic scenarios for real-world application deployments. When multiple distributed producers and consumers are active at

796

D. Palmisano and M. Cilia

Fig. 2. Example deployment diagram

the same time, the notification service will be stressed from multiple directions, potentially causing the server to behave differently. NSLoadGen allows to deploy a test across a number of distributed machines. According to the scenario description, a group of NSLoadGen instances are started in these computers, each of them simulating the part of the scenario that correspond to it. Figure 2 shows an example deployment diagram. The experiment is coordinated by a Master node, responsible to start, stop and synchronize the set of NSLoadGen instances. Each NSLoadGen instance has assigned the instantiation of producers that inject messages (also advertisements/unadvertisements depending on the Notification Service) and/or the consumers that inject subscriptions (and unsubscriptions). These events are registered for later evaluation. 3.3

Result Collection

Commonly, users want to analyze different metrics of test runs, such as producer injection, message latency or even which was the injection distribution of a set of producers during a time windows. They may even want to have the same results using different charts. Thereby, instead of presenting specific metrics, NSLoadGen provides a very convenient way to catch events containing the most important actions carried out during the test run. This information can be used later to analyze in deep the behavior of the System Under Test (SUT).

4

Architecture and Design

This section sketches the principal architectural design presenting a brief description of the main building blocks. 4.1

High Level Architecture

The NSLoadGen platform is intended to be a testbed for notification services, maintaining it abstract enough to test different Notification Services. In this

NSLoadGen– A Testbed for Notification Services

797

Fig. 3. Logical view of NSLoadGen

way, it can be seen as an independent piece of software that instantiates NSindependent (Notification Services independent) producers, consumers and inject messages and subscriptions. At the moment of interacting with the SUT, these pieces are transformed from NS-independent to NS-dependent (Notification Services dependent). Users only have to declare the scenario. NSLoadGen will simulate it and transform involved components (producers, consumers, messages, and so on) to communicate with the underlying Notification Services. In this way, the architecture of NSLoadGen is logically split into two layers as can be seen in figure 3. The upper layer stays unaware of any concrete Notification Services functionality, while the lower one is in charge of interacting with the SUT. The idea was to encapsulate common functionality, needed for every Notification Services, to be used by the core of NSLoadGen, but letting specific functionality to be implemented by NS-dependent components. This is achieved by well-known pieces of software called adapters. The platform has specific interfaces defined as an integral part of its core. A library of adapters for several notification services is available in NSLoadGen. Additionally, users can write adapters for those Notification Services not supported yet. At the producer side, producer adapters are responsible for issuing advertisements, unadvertisements and injecting messages. All of them are NSindependent activities. Only at the moment of carrying out some NS-dependent task, the platform will involve the respective dependent component. In the same way, NSLoadGen is also split into two parts at the consumer side; consumer adapters are in charge of making subscriptions, receiving messages and unsubscriptions. This component will call to specific consumers when necessary. Additionally, there is a third kind of adapters called administer adapter and, as its name suggests, it has the responsibility of carrying out management activities like creating connections, message destinations2 or queues. Producer. Producers are mainly in charge of injecting messages according to an injection distribution specified by the user in the scenario description. As a 2

A Destination is an abstract name used to mean “subject” or “topic”.

798

D. Palmisano and M. Cilia

consequence, several steps have to be carried out to generate ready-to-send messages. The producer component design follows a pipeline architecture: Message Factory → Message Filled Out → Message Transformer → Injection. Producers have a behavior assigned, as defined in the scenario, that drives the message injection. In addition to this behavior, they have set up specific messages to inject. When the inject distribution demands the injection of a message several steps that are described later take place to get a ready-to-send message. First, the Message Factory creates a new message according to the message structure and the message type to be injected (name-value pairs, a XML message or a serialized object). A single message structure can be linked with any of these types: a single producer can be assigned to inject the same message structure but with different types or even data. The Message Factory creates cleaned messages, that is, messages without data, and are NS-independent, that is, messages that are not specific to any notification service. Created messages are the input to the Message Filled Out component. This component makes an analysis of the information the message should have, fetches the corresponded data from the source specified by the user in the scenario and fills the message out. Afterwards, a Message Transformer is needed to transform messages from NS-independent to NS-dependent in the third step. Fourth, the filled NS-dependent message is received by the Producer Adapter that finally, as it was previously explained in section 4.1, it has encapsulated common functionality needed for every Notification Services. This last component injects the messages into the Notification Services, delegating this task to the NS Interface component that contains functionality specific to the SUT. Although this approach gains in Notification Services independence it has an important drawback: every time a message has to be injected a three-step process take place: message creation → fill it out with data → transformation. What is worse, these steps are executed after test run start up. As a consequence, it may become a bottleneck on NSLoadGen side and its performance can be seriously affected. In order to improve the performance of producer component, a Pool of ready-to-send messages was added to the previous solution. Figure 4 shows how the previous approach was adapted to support this pool. Note that messages are defined by means of a called message structure that is sketched with a tree.

Fig. 4. Producer component: proposed approach

NSLoadGen– A Testbed for Notification Services

799

Fig. 5. Consumer component: proposed approach

As in the previous solutions, the producer component activates itself to inject a message. However, instead of starting the three-step sequence, it only picks a ready-to-send message from the pool. Once the pool returns a message, the producer gives it to the adapter that injects it into the SUT, always relying on the Notification Services Interface. The pool acts as an active component that runs at background, when the system is not overloaded. A mark is used to maintain this pool with a minimal number of instances during runtime. When this mark is reached the pool will asks for new fresh message instances to the factory, triggering the three-step process. Notice that messages stored in the pool are ready-to-send. The producer only has to pick one and deliver it to the adapter. Additionally, NSLoadGen provides also configuration mechanisms to skip the step of filling out messages with data, becoming the three-step process in only two steps. This is very useful when message data is not important in the evaluation. Consumer. Consumers are mainly in charge of issuing subscriptions and receiving messages. Each consumer has a filter3 that was assigned by the user in the scenario. Like messages, filters go through a sequence of steps to convert it into subscriptions that the consumer can issue. Figure 5 shows the architecture of the consumer component. The Filter Factory component creates a NS-independent filter according to the Filter Structure. The platform supports the dynamic creation of filter data, i.e., the user can express the filter structure and the source of data to fill it out, and NSLoadGen will create it. In this way, once the filter has been created it is moved to the Filter Filled Out component, which returns a filter with data, but still Notification Services independent. After that, the filter is transformed to a subscription and assigned to the consumer. During simulation and according to its specified behavior, a consumer will give the subscription to the Consumer Adapter, who relies on the NS Interface component to issue it. Later, the notification service will deliver messages to that consumer when matching messages are published. Notice that filter steps are similar and different to message steps presented in the producer section. A filter has to traverse several components to become a 3

A filter is an abstract name used to mean NS-independent subscription.

800

D. Palmisano and M. Cilia

subscription. But in contrast to producers, this is not a cyclic process. Filters, and as a consequence subscriptions, are created once during the initialization of the test run. The same approach is used for advertisements on the producer side. 4.2

Distributed Testbed

NSLoadGen was designed to allow automated testing of notification services across a number of machines connected by a network. A test run is related to three primary concepts: the System Under Test), one Master node and, commonly, several Slave instances. Test runs are coordinated by the Master node, who starts, stops and synchronizes the set of Slave instances. The Slave is in charge of instantiating producers that inject messages, and/or consumers that issue subscriptions. Its purpose is to wait for Master high-level commands and to trigger activities (start producers, consumers, collect data, etc.). The Master node has the ability to collect run-time data from all slave instances. This is done by simply sending a command after execution. Figure 2 in section 3.2 shown an example deployment diagram of a test run. At the moment of starting the simulation, the input scenario (composed by the three parts presented in section 3.1) is received by the Master instance and sent to the Slave instances. Once it has been received, the Salve instance loads it and, after an initialization process where producers, consumers and the pool of messages are created, the Master is responsible for coordinating the Slave instances acting as initiator of the test run. At this time, the Slave(s) begin processing their part of the scenario: producers will begin sending advertisements, messages and unadvertisements; and consumers will issue subscriptions, consume messages and unsubscriptions. As each message is sent or received, these events are registered with relevant information such as a unique message ID and timestamp4 . Individual producers and consumers can be configured with different message production, persistence, durability, subscriptions and other characteristics, as well as connection and disconnection behavior, for every Slave instance. At the end of the execution the Master node triggers the “stop” signal to the Slave instances, stopping producers and consumers. 4.3

Logging Approach

During the test run all significant events inside the platform are registered. This information can be obviously used to analyze the notification services under test, not only performance conclusions but also producer and consumer behavior, exception conditions, etc. The user has the possibility of defining which are the events that should be registered along the test run, as well as, the amount of information for every event (debug, info or minimal). Significant events are advertisements, unadvertisements and injections (in the producer side) and subscriptions, unsubscriptions and received messages (in the consumer side). 4

In case of performance analysis want to be considered, NSLoadGen is dependent on all system clocks being synchronized.

NSLoadGen– A Testbed for Notification Services

801

However, it is clear that registering important events of an application is a well-known crosscutting concerns. We use aspect-oriented programming (AOP) to flexibly store relevant events during the experiment. Since the evaluation step is carried out relying on this data, supporting a configurable recording of it was very important. This approach allows maintaining independence between NSLoadGen and the tools to analyze the log. Aspects can be configured to manipulate the data and with different alternatives to store/disseminate it. For example, these events can be sent to files, or stored in a database, or sent them directly to a monitoring application that presents real-time results about the test run. Furthermore, the user can decide which events to register. Depending on the metrics of interest, not all events are necessary. For example, if throughput is going to be analyzed, then only message injections and receptions should be caught, but not producer and consumer connections and disconnections. Also, the user can modify the format of the events. As an example, the user could define a XML format, a plain-string format, or SQL sentences, by means of extending the components that are in charge of event logging. Additionally, the user can deactivate the logging module altogether if these events are not important. For example, Salvucci+ [16] rely on NSLoadGen as the load generation tool. This particular project focused on analyzing the internal of Notification Services at runtime where NSLoadGen logging features where disabled.

5

Implementation

NSLoadGen has been implemented in Java and also uses XSLT transformations. This section introduces the steps of scenario definition and scenario emulation in more detail, focusing on the implementation view. 5.1

Scenario Definition: Meta Language

As presented in section 3.1, a NSLoadGen scenario is composed by three parts. To describe it, a Meta Language in XML was defined where each scenario issue was assigned to a separated XML file: Load Description contains data and behavior, Run Description contains binding information and Configuration contains information to configure NSLoadGen and the SUT.

Fig. 6. Testbed steps

802

5.2

D. Palmisano and M. Cilia

Scenario Simulation

In this step the defined scenario is simulated and the SUT is put under pressure. Along this step NSLoadGen passes through a two-phase process: initialization time where the platform is set to simulate the defined scenario, and run time where the simulation itself takes place. Initialization Time. One of the main activities carried out in the initialization step is the conversion from {Message Structures,Message Type} to Message Factories. The Load Description part of the scenario contains information about the message structure, the source and data that the messages have to be filled out, and the message type that every producer should inject. Figure 7 illustrates the idea with an example. In the example there are three producers, P1, P2 and P3 and three message structures declared. The message structures are declared as a tree, and, within every element of the tree there are information about how it has to be filled out with data (for example, node c must be a 20-length string). NSLoadGen allows different sources of data, such as from a data base, a custom source defined by the user, CVS files, or fixed data, to fill out a message. Furthermore, NSLoadGen has a library of functions that return common data used in different domains, such as city names, country names, zip code numbers, to name a few. Additionally, the figure shows that producer P1 is assigned to generate namevalue pair messages according to the structure M2, producer P2 is assigned to create XML messages following the structure M3 and, finally, producer P3 should generate messages M1 and M3 of types XML and serialized, respectively. By means of separating the message structure from the assignation of them to producers, the same kind of message can be transformed to different types. In the example, producer P2 and P3 should generate messages according to the structure M3 but with type XML and serialized, respectively. The activity of transforming a message structure with an assigned type to a real NS-independent message is carried out by specifics Message Factorys, as explained in subsection 4.1. Therefore, and following the previous example, the platform should create four Message Factory, one for each {message type,message structure} pair, such as “LoadGeneratorFactoryMessage XML Message1” and “LoadGeneratorFactoryMessage NVP Message2”.

Fig. 7. Load declaration

NSLoadGen– A Testbed for Notification Services

803

Other activities that take place at initialization time are producers and consumers creation, message pool creation and subscriptions creation. Run Time Message Transformer. The system uses a generic message type along the test execution but, at specific times, it transforms this NS-Independent message to NS-Dependent message. This activity is carried out by a component called Message Transformer. In case a new notification service should be supported by NSLoadGen, the MessageTransformer interface has to be implemented. One of the pre-defined message transformer provided by NSLoadGen is JMSMessageTransformer which transform from NSLoadGen to JMS messages, and vice versa. Message Types. NSLoadGen supports three types of messages: XML, Map and Serializable (See figure 8). These three types cover most of the message types used in Notification Services. LoadGeneratorMessage is the message superclass and contains a set of properties. As was previously explained, these messages are created by factories. In this way, LoadGeneratorFactoryMessage is the superclass of the factories containing an abstract method getMessage that returns a LoadGeneratoreMessage. Notification Service Independence. In order to fulfill with the requirement of Notification Services independence, NSLoadGen was desgined and implemented using the Abstract Factory Design Pattern [17]. This pattern allows to create families of related or dependent objects without specifying their concrete classes, which in turn, will be NS-dependent classes. As can be seen in figure 9, the AdapterAbstractFactory provides the methods to create the adapter entities (e.g., LoadGeneratorProducer, LoadGeneratorConsumer or LoadGeneratorAdminister). This factory is created at initialization time and is NS-dependent, so it knows what kind (subclasses) of entities will be needed to interact with the SUT. It is in charge of creating NSdependent entities. NSLoadGen interacts with these components without even knowing which Notification Services are they communicating with. In order to extend the platform to support new notification services, specializations of these classes should be written.

Fig. 8. Message class diagram

804

D. Palmisano and M. Cilia

Fig. 9. Adapter abstract factory class diagram

Distributions to Simulate Behavior. Distributions are a way that the user can use to vary the producer and consumer behavior. At the producer side, they have the ability to connect to and disconnect from the system, or, in other words, they can issue an advertisement, inject messages for a while, and then disconnect from the system issuing an unadvertisement. As a consequence, they only can inject messages when their state is active. During this state, producers can emit messages following, probably different, inject distribution. For instance, let’s suppose that a producer was assigned to a loop where it maintains enable during 20 seconds and disable for 10 seconds. In addition, when enable, it has an inject distribution that corresponds with the “sine” function. To cope with this requirement, producer components have been designed with two clocks, one to drive the enable/disable function distribution, and the other to follow the injection rate. Obviously, the second clock only works when the first is active, so enable/disable timer is the “master” of the injection timer. Figure 10 (a) shows both timers and the events carried out by a producer, i.e., enable, inject and disable. This important feature in producers of enabling and disabling (or advertisements and unadvertisements) is given by a distribution function that the user defines in the Load Description part of the scenario. A library of pre-defined functions are available but, additionally, users can define their own functions. Besides that, sometimes there are NSs that do not support advertisements and unadvertisements. In these cases, NSLoadGen has a default timer that wakes up the injection timer at the beginning of the test run, and sleeps it at the end. An important example of such systems is the JMS specification, which does not define the advertisement behavior.

NSLoadGen– A Testbed for Notification Services

805

Fig. 10. Distribution timers: (a) producer side, (b) consumer Side

At the consumer side, among the main parameters is the rate of subscriptions and unsubscriptions. Unlike producers, consumers only have one timer associated that drives this rate. Figure 10 (b) shows the main concepts. This enable/disable (subscription/unsubscription) clock drives the complete consumer’s life-cycle. Like producers, this timer uses a enable/disable (subscribe/unsubscribe) distribution which is given by a function defined in the Load Description part. There are a library of pre-defined functions but, additionally, users can define their own function. In case the user does not want a subscription/unsubscription behavior for some consumer, the platform has a default timer that wakes up the consumer at the beginning of the test run, and sleeps it at the end.

6

Conclusions and Future Work

This paper presented NSLoadGen, a testbed for notification services that automates the process of stimulating a Notification Services. In other words, NSLoadGen is in charge of injecting load according to a scenario description with multiple purposes: 1. supporting the selection of the most appropriate Notification Services for a specific application (scenario); 2. analyzing a Notification Services under stress conditions for code optimization; 3. analyzing a Notification Services under typical and peak conditions for tuning purposes. For doing so, we first concentrated on defining a language to describe the scenario to be simulated, covering as many Notification Services features as possible and demonstrating how that representation can be done easier by splitting it in three parts: one for the load characterization, one for running the experiments and one to configure the platform and Notification Services. After the scenario is specified, it is then taken and simulated in order to evaluate the characteristics

806

D. Palmisano and M. Cilia

of the Notification Services under test. Detailed data during simulation is generated for later analysis. In this way, the platform covers scenario description, as well as load generation, deployment, execution and data collection during the experiment. NSLoadGen was constructed with the idea to be generic, meaning with this that it should be independent of the Notification Services under test. NSLoadGen has been successfully used with several JMS implementations and also with other NSs that do not provide a JMS interface [18]. Furthermore, the platform provides several facilities in which new messaging systems can be rapidly tested by only specializing a small set of classes. The scenario description language was founded on XML . From the point of view of the end user, the interaction with the platform through the use of XML can be cumbersome, even more when complex scenarios have to be represented. Hence, in future versions a graphical editor will be provided to specify the scenario with a graphical (easy to use) interface generating as a result at the end the language already defined. One interesting direction to be investigated is to extend NSLoadGen to stress Web Services and other SOA-related technologies. We are also analyzing the possibility to incorporate the notion of mobility of consumers and producers into the platform. Besides, studies on more complex scenarios are needed to validate the scenario description language, extending it if necessary.

References [1] Inc. Sun Microsystems. The Java Message Service Specification. Technical report, Sun Microsystem Technical Report (2002) [2] Object Management Group. Notification service specification (July 1999) [3] Oki, B., Pfluegl, M., Siegel, A., Skeen, D.: The Information Bus - An Architecture for Extensible Distributed Systems. In: Proceedings of SIGOPS, pp. 58–68 (1993) [4] Cilia, M., Antollini, M., Bornh¨ ovd, C., Buchmann, A.: Dealing with heterogeneous data in pub/sub systems: The Concept-Based approach. In: DEBS 2004, [5] M¨ uhl, G., Fiege, L.: The REBECA Notification Service (2001) [6] Cugola, G., Di Nitto, E., Fuggetta, A.: Exploiting an event-based infrastructure to develop complex distributed systems. In: ICSE 1998 [7] Carzaniga, A., Rosenblum, D.S., Wolf, A.L.: Design and evaluation of a wide-area event notification service. ACM Transactions on Computer Systems (2001) [8] Pietzuch, P.R., Bacon, J.: Hermes: A distributed event-based middleware architecture. In: ICDCSW 2002 [9] Banavar, G., Chandra, T., Mukherjee, B., Nagarajarao, J., Strom, R.E., Sturman, D.C.: An efficient multicast protocol for content-based publish-subscribe systems. In: ICDCS 1999 [10] Object Management Group. Common Object Request Broker Architecture specification [11] Carzaniga, A., Wolf, A.L.: A benchmark suite for distributed publish/subscribe systems. Technical report, Department of Computer Science, University of Colorado (2002)

NSLoadGen– A Testbed for Notification Services

807

[12] Sonic Software. Benchmarking e-business messaging providers. Technical report, Sonic Software (2004) [13] Heckmann, O., Pandit, K., Schmitt, J., Steinmetz, R.: KOM ScenGen - The Swiss Army Knife For Simulation And Emulation Experiments. In: Ventre, G., Canonico, R. (eds.) MIPS 2003. LNCS, vol. 2899, Springer, Heidelberg (2003) [14] Kuo, D., Palmer, D.: Automated analysis of java message service providers (2001) [15] Dustar, S., Haslinger, S.: Testing of Service Oriented Architecture - A practical approach (2004) [16] Salvucci, S., Cilia, M., Buchmann, A.: A Practical Approach for Enabling Online Analysis of Event Streams. In: DEBS 2007 [17] Gamma, E., Helm, R., Johnson, R., Vlissides, J.: Design Patterns, Elements of Reusable Object-Oriented Software. Addison Wesley Longman, Inc, Redwood City (1998) [18] Palmisano, D., Cilia, M.: NSLoadGen, a testbed for Notification Services. In: partial fulfillment of the requirement for the degree of Systems Engineer (August 2006)