Progress Actional

Progress Orbix

Progress Sonic

TECHNICAL Progress Orbacus

®

®

®

®

®

®

Progress Apama ®

™

PAPER

Progress DataXtend

®

®

Progress ObjectStore ®

®

®

Progress Artix ®

®

®

Progress Artix Data Services ®

®

> INTERACTION MODEL Enterprise Integration Whiteboard Series: Progress Sonic Technical Brief Series Number Eight ®

www.progress.com

®

TABLE OF CONTENTS Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Mediation and the Problem of Impedance Mismatch . . . . . . . . . . . . . 2 A Simple Example: Mediating Two Interaction Models with the Sonic ESB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 A Real-world Example: Meditating Several Interaction Models with the Sonic ESB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

www.progress.com

2

INTRODUCTION This paper focuses on the seventh point of mediation : the interaction model. Most code gets written to mediate the interaction model because people don’t think of it in a normative way. Rather they see file processing, publication subscription (PubSub or P/S), and request response, whether it’s a database or a type of RPC (remote procedure call) functionality, as different things. In fact, every generation of technology—going back to DEC architectures, OSF, CORBA, and, most recently, J2EE—has dealt with request-response, PubSub, and file-oriented batch movement as completely separate things. They generally tend to be different technology types; they are clearly different protocols. They tend to be offered in largely different components of the stack and oftentimes in completely different product lines. In short, no one has ever seen all of these communication modalities as the same, and, therefore, no one has seen their need to be mediated—which has exacerbated the problem of mediation.

MEDIATION AND THE PROBLEM OF AN IMPEDANCE MISMATCH

The best way to see this is on a grid, as shown. On one side is the provider and on the other is the consumer, or, in more old-fashioned terms, the server and the client. You can also see that there are really four models of interaction.

www.progress.com

CLIENT/ CONSUMER

Every solution ends up having to mediate these things. Here’s a classic example: a system receives a batch file, but it can only process transactions. So developers write code that hacks the file up into messages and posts each of those messages, one at a time, to the transactional system in the background. People don’t think of that SERVER/PROVIDER as mediation because it’s probably written as an REQUEST FIRE & PUBLISH RESPONSE FORGET SUBSCRIBE import routine or a gateway, or else it’s an entirely different piece of application code. This now becomes part of the maintenance portfolio and the corporate estate of IT infrastructure, whereas it really is a mediation methodology.

BULK & BATCH

3

>>

Request response. This is the most conventional interaction in which a user invokes or looks up a table on a database or does something similar.

>>

Fire and forget. In this interaction a user fires a message off to a queue and doesn’t have to think about it being delivered.

>>

PubSub. A single message is pushed out and communicated to hundreds of consumers.

>>

Bulk or batch processing. Files or big sets of data are moved together.

If this picture is squared and the four models SERVER/PROVIDER REQUEST FIRE & PUBLISH are duplicated on the other side as shown, then on the RESPONSE FORGET SUBSCRIBE diagonal, there’s no impedance mismatch. A requestREQUEST RESPONSE response consumer should be able to get to a requestresponse producer with no problem. However, the 12 FIRE & other squares are a big question mark. For information FORGET delivered PubSub there are two services: one designed to produce customer records and the other designed PUBLISH SUBSCRIBE to consume them. A client desktop has customer records coming to it, and a service out there produces BULK & BATCH customer records. But imagine that the producer makes changes to customer records every time they occur and publishes them to all the interested parties. However, the desktop client wants to ask for a customer whenever it needs the customer’s information. Even though the service provides the customer data the way the client wants it—with the right format, protocol, data structure and so forth—if the client desktop wants to ask for the records when needed but the server wants to push them out any time it has a change to one of them, there is an impedance mismatch in the interaction model. Then, even though everything else in the case represents addresses the points of mediation and so might work fine, the two services still can’t talk to one another. Jon Udell, Senior Editor at InfoWorld several years ago, once put the idea in a nutshell, ““Request-response software speaks when spoken to whereas event-driven software speaks when it has something to say.” In addition to the suggestion that event-driven software is bit smarter than

www.progress.com

BULK & BATCH

4

request-response software, he makes a fundamental point: there are two types, at least, of communication and a fundamental impedance mismatch between the two. If one system wants to send a batch of information, but another wants to take it in one message at a time—for example, in a fireand-forget interaction—there is an equally difficult impedance mismatch to mediate. So these 12 cells that represent the impedance mismatches are, in fact, the 12 cells where integration code lives today. It’s in these places where we would want to mitigate the mismatch within the ESB, removing them from the picture.

A SIMPLE EXAMPLE: MEDIATING TWO INTERACTION MODELS WITH THE SONIC ESB To understand how interaction models can be mediated on the Progress Sonic ESB, consider this: BATCH TO REQUEST/RESPONSE MEDIATION a batch file arrives and needs to go to a mainframe that wants to have each of the records in this file as a MAINFRAME single transaction, in a request-response manner. The REQUEST/ RESPONSE file PAR service can ingest the file and divide it into individual records on the bus. The ESB then transmits CALL OUT ESB SERVICE to a callout service, which takes each of those records in turn and posts it to the mainframe, as a single PAR SERVICE transaction. Notice the mediation of batch processing BATCH to a request-response interaction occurs without FILE having to write a single line of code. The mediation has been achieved entirely through configuration-driven means and orchestrated by a simple sequencer that moves these messages through the system accordingly. The next section will discuss a more complex, example illustrating a typical mediation as it occurs in the real world. ®

®

A REAL-WORLD EXAMPLE: MEDIATING SEVERAL INTERACTION MODELS WITH THE SONIC ESB Consider this example of a Progress customer, a manufacturer and one of the earliest Sonic ESB users, who had a problem that is representative of manufacturers and of many other businesses as well. The company had

www.progress.com

5

a product master in which all products sold are recorded. These records included a product’s pricing; SKU; relevant information about where the product is made, stored and warehoused; what divisions carry the product; and so on. Like most product masters, this was a universal source of information, used throughout the business in 13 different places, starting with the Web site, where it was used to sell products. It was also needed in the call center for servicing requests; in the order processing system for orders; in the inventory, warehousing, and distribution systems to keep track of products; all the way back to the general ledger to record revenue by product. Specifically, a form of the DATABASE product master was required at each of these locations. In other words, while PRODUCT MASTER the actual structure of the product PM1 PM2 PM3 PM5 PM6 PM4 master was in entity relationship terms at the master site, the form was WEBSITE CALL CENTER ORDER INVENTORY WAREHOUSING DISTRIBUTION represented differently in each of the PROCESSING operational data stores to serve the BUSNINSS WORKFLOW purpose of the application supported there. But as the graphic of this relationship reveals, there is also a process pipeline that shows how work actually flows through the business. Customers see a product on the Web site. They place an order for it. Inventory gets pulled, then assembled, distributed, and, finally, billed. However, workflow exceptions were occurring all the time. Orders would drop out of systems at various places. The call center wouldn’t know how to process an order because the product information wasn’t correct at that location. Or the inventory couldn’t be pulled because there wasn’t a current product master identifying where the product was located or, in the case of a new product, that the product existed. Distribution would have problems shipping products, and the general ledger might not be able to record some products that were sold. The common denominator among these problems was inconsistent views of the truth. Each of these sites often did not have the most current version of the product master, which caused orders to drop out.

www.progress.com

PM7

GENERAL LEDGER

6

ETL PRODUCT MASTER

INCONSISTENT PRODUCT MASTER(S)

FTP

PM1

WEBSITE

PM2

CALL CENTER

PM3

ORDER PROCESSING

PM4

INVENTORY

PM5

PM6

WAREHOUSING DISTRIBUTION

PM7

GENERAL LEDGER

Why weren’t all versions consistent at the same time? In this operation the product master team would re-master the file every night. It would use ETL (export, transform, and load) techniques to extract the file. Then it would use FTP to send the file to each of the consuming locations that wanted it. Now here’s where the problems began. For a variety of reasons, the updates couldn’t occur at the necessary endpoints. One reason was that the Web site, for example, was a 24-by-7 site. It had to be up all the time and so couldn’t be taken down to import the refreshed file. Consequently, imports were confined to a few hours, the wee hours of Sunday night, and done as quickly as possible. As a result, the Web site got updated only once a week. In other cases, there might be transmission failures that caused the file not to get to a data store in time, so that the system would run with the prior day’s data. Or the import routine for the file would fail for one reason or another. In many cases, the problem arose because the file was batched. A single failed record would cause the whole batch to fail. Because the IT staff handling the job wouldn’t know what other inconsistencies had been introduced, rather than accept a single failure and try to reconcile it, they would reject the whole batch and use the current day’s data, known to be safe. In short, imports were not occurring for a variety of issues—transmission problems, batch import failures, site availability and so forth—and the inconsistencies were great enough that the process failures were real. As a result, the owner of the product master decided to put a new technology in place, one that could update all systems in near real time. The

www.progress.com

7

idea was that any time there was a single change to the product master, this change could be communicated to everyone, and they could apply that update quickly, like a transaction, rather than having to bring the whole site down. If they would acquiesce to doing the import, then maybe all systems could run in synch. The product master owner looked at JMS PubSub technology with the idea of taking every event as it occurs in the product master and publishing it onto a JMS topic. Everyone concerned with the product master could then quickly consume from that topic, updating their systems in real time so they would all be perfectly synchronized. In other words, this appeared to be a bona fide, state-of-the-art, loosely coupled technology that would enable everyone to make an event-driven update. However, when the product master owner finally got everyone involved to meet together—about three months later than the meeting was first scheduled—there was a range of obstacles. One site that received the product master was built by a systems integrator that went out of business. The team was tied up for the next 18 months and couldn’t do anything to modify it. Another obstacle was the mainframe system that personnel in charge didn’t want to change because they lost the source code, which was written by a company that’s had gone out of business. Yet another system was written in-house, but there were no longer people on staff who knew the code. In other words, for all kinds of reasons, none of these systems could be changed to accept JMS messages in any reasonable amount of time: because they’re packaged applications, because they’re written by third parties, because they’re off the development lifecycle in which there are higher maintenance priorities on those applications. As a result, it would be three years before everybody could cut over to JMS PubSub. In fact, there was still a problem. Even though the product master owner intended to use the most loosely coupled, state-of-the-art technology available, the organization was still tightly coupled. Everybody would need to move in lockstep from a batch, ETL-FTP methodology to JMS PubSub at the same time. Otherwise, they’d be cut off from the updates of the product master. In other words, even though the technology was loosely coupled, the entire organization was tightly coupled. Getting everyone to move in unison, enterprise-wide, would take too much coordination.

www.progress.com

8

What was the alternative? Imagine that an interaction model could be delegated in such a way that each endpoint didn’t care what interaction model the other endpoints were using. This could happen if, instead of using JMS alone, the company used an ESB (specifically, the Progress Sonic ESB) and delegated the interaction to it. The graphic shows this alternative. Remember the purpose of moving from JMS PubSub into the ESB is to bring about organizational loose coupling as well as technological loose coupling. This requires two steps. The first step is to introduce an ESB to the picture, continuing to do what has always been done in distributing the product master. The second step is to extract, transform, and load (ETL) the information out of the product master, producing a batch file. However, at this point, with the introduction of Sonic ESB, there is a container with the file PAR service that will take in or consume that batch file and chop it up into little messages. The PARS service will then push those messages out to all of the various endpoints, each of which has another BATCH FILE

PRODUCT MASTER (PM)

ETL PAR SERVICE CONTAINER

ESB

FDS

FDS

PM1

WEBSITE

FDS

FDS

PM2

INVENTORY

FDS

PM3

WAREHOUSING

FDS

PM4

GENERAL LEDGER

container that hosts a file drop service. This service receives all e-messages and plunks them out to a file. As a result, all relevant systems and sites—the Web site, the inventory system, the warehouse system, all the way through to the general ledger—receive the same file they were accustomed to getting, but now over an ESB, and continue to ingest that file as they always had. Although an ESB was installed into this environment, none of the endpoints

www.progress.com

9

was the wiser. As far as they were concerned, they were still getting the same FTP-ETL created file that they always got. What was accomplished—especially since the result is the same functionally as before? The benefits are threefold. First, there is now a recoverable FTP because this model allows for sync point restart. In other words, if 90 percent of the file were transmitted in FTP and then the transmission were to fail, previously you would have to retransmit the whole file. In contrast, the ESB will pick up where the failure occurred and retransmit only the remaining 10 percent of the messages. Consequently, the file is more recoverable, more resilient than with FTP by itself. Second, FTP is secure. FTP is an unsecure protocol. Normally it takes a lot to make FTP secure, so that when you make it secure, the resulting FTP is no longer standard. In contrast, Sonic ESB is highly secure. The third advantage gained by introducing the ESB is more visibility. Because the large file has been hacked up into messages, there is better fidelity when you look inside them. With the instrumentation of the ESB you can view things about this data that you wouldn’t see if it were transmitted as a big file. At this point, you may well ask if these are benefits worth the kind of conversion necessary to get them. After all, replacing the FTP infrastructure with an ESB and using file drop and file PAR services will result in slower performance than the FTP infrastructure at the start. So you have a slower infrastructure at greater expense, and all you have gained are recoverable FTP, secure FTP, and a little more visibility. In other words, if this is all you get, the conversion isn’t worthwhile. However, the real benefit at this stage is organizational loose coupling—as shown in the new, overlaid graphic. What is happening here is that the system dropped its file PAR service and immediately switched over to a JMS PubSub model. Then it began pushing the messages directly onto an ESB topic. Also, a new service—for caching—has been introduced. The caching service takes those messages like any other consumer and keeps a state table, basically the current state of all the product information. Each time a new update occurs, the correct information is updated in the caching service. At the end of the day, the file is shipped to each location. Once again, the environment really isn’t affected. In fact, everybody is still consuming

www.progress.com

10

BATCH FILE

PRODUCT MASTER (PM)

ETL

1 RECOVERABLE FTP

PAR SERVICE CONTAINER

ESB

2 SECURE FTP

FDS

FDS

FDS

PM1

FDS

PM2

FDS

FDS

PM3

3

PM4

VISIBILITY

WEBSITE

INVENTORY

GENERAL LEDGER

WAREHOUSING

ETL files. But, the very next day, one system could eliminate its ETL file drop service and begin consuming the JMS messages directly while everybody else continues to receive the ETL file drop they were getting before. In other words, each consuming system in the organization can convert on its own timetable, changing to the JMS PubSub when and if it’s able and ready to do so. The head end was able to change right away while all the others systems can take their time, eliminating organizational tight coupling. As a result, the organization can begin to change on its own timetable in a systematic and incremental way, which the former picture of everybody cutting en masse over to JMS PubSub wouldn’t have permitted. The real power of the ESB, however, is that it doesn’t just mediate JMS PubSub to batch. It allows for all the forms of mediation to take place. In fact, BATCH FILE

PRODUCT MASTER (PM)

ETL ESB TOPIC

PAR SERVICE CONTAINER

JMS PUBSUB MODEL

1 STATE TABLE

CACHING SERVICE

ESB

RECOVERABLE FTP

2 SECURE FTP

FDS

FDS

FDS

FDS

FDS

FDS

WEB SERVICE CALLOUT

JMS PM1 PUBSUB

WEBSITE

www.progress.com

PM2

INVENTORY

WEB SERVICES ENDPOINT

PM3

WAREHOUSING

PM4

3 VISIBILITY

GENERAL LEDGER

11

the inventory system owner didn’t want to receive JMS messages. Instead, he exposed a Web service endpoint to accept changes. Now, every time the product master publishes a message (a new service on the ESB), the inventory system can accept that new message as a Web service via its exposed endpoint. As a result, the ESB is mediating PubSub to request-response. Yet another system owner rejects both exposing a Web service endpoint and taking JMS messages. He wants only to ask for product information when the system is ready. This is possible as well with Sonic ESB. Every time this system requests product information, yet another Web service on the ESB goes up to the cache and returns that information directly this system—mediating PubSub to request-response in the other direction. Of course, the product master system can still output messages using the file drop services batch.

CONCLUSION As we’ve seen in the previous example with the Sonic ESB, one PubSub producer can deliver messages in PubSub, in request-response in, request-response out, and in batch—handling all four mediation types without requiring IT to write a single line of code to do it. Just as important, the Sonic ESB allows the organization to remain loosely coupled, so that system owners can select the right interaction model for their application and not have to deal with writing new code to adapt to what the producer or consumer might ideally want. This is interaction model mediation at its finest, creating loosely coupled technology as well as loosely coupled organizations that—with all seven points of mediation discussed—now can move in a more agile way than possible with other technologies. The final brief in this series will talk about the service lifecycle.

www.progress.com

PROGRESS SOF T WARE Progress Software Corporation (NASDAQ: PRGS) is a global software company that enables enterprises to be operationally responsive to changing conditions and customer interactions as they occur. Our goal is to enable our customers to capitalize on new opportunities, drive greater efficiencies, and reduce risk. Progress offers a comprehensive portfolio of best-in-class infrastructure software spanning event-driven visibility and real-time response, open integration, data access and integration, and application development and management—all supporting on-premises and SaaS/cloud deployments. Progress maximizes the benefits of operational responsiveness while minimizing IT complexity and total cost of ownership.

WORLDWIDE HEADQUARTERS Progress Software Corporation, 14 Oak Park, Bedford, MA 01730 USA Tel: +1 781 280-4000 Fax: +1 781 280-4095 On the Web at: www.progress.com For regional international office locations and contact information, please refer to the Web page below: www.progress.com/worldwide Progress, Actional and Business Making Progress are trademarks or registered trademarks of Progress Software Corporation or one of its affiliates or subsidiaries in the U.S. and other countries. Any other marks contained herein may be trademarks of their respective owners. Specifications subject to change without notice. © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Rev. 2/11 | 6525-131886