Hanging Together on the SOA Platform - Introduction to the ESB-jBPM Integration

One of the great strengths of the SOA Platform is its wealth of software integrations. In this post, I'll examine the JBoss ESB - jBPM integration in the SOA Platform and how it can be used to connect business processes to services.

JBosssESB

Service Oriented Architecture (SOA) is not a single program or technology. It’s really a matter of software architecture or design. In hardware terms, a “bus” is a physical connector that ties together multiple systems or subsystems. Instead of having a large number of point-to-point connectors between pairs of systems, you connect each system to the bus once. An Enterprise Service Bus (ESB) does the same thing, logically, in software.

Instead of passing electric current or data over the bus to and from the connections (or “endpoints”) on the ESB, the ESB logically sits in the architectural layer above a messaging system. The messaging system allows for asynchronous communications between services over the ESB. In fact, when you are working with an ESB, everything is either a service (which in this context is your application software) or a message being sent between services. It’s important to note that a “service” is not automatically a web service. Other types of applications, using transports such as FTP or JMS, can also be services. Is an ESB the same thing as SOA? (Service Oriented Architecture) Not exactly. An ESB does not provide a Service Oriented Architecture, but it does provide the tools than can be used to build one–especially loose-coupling and asynchronous message passing. SOA is a series of principles, patterns, and best practices.

JBossESB [2] is an open source ESB implementation that supports multiple transports, protocols, a listener-action model for loose coupling of services, content based routing with JBoss Rules (Drools), and workflow management with jBPM.

JBoss jBPM

Business process management (BPM) is an often misused or misunderstood term.[3] There are many definitions for BPM, but the best and simplest that I can think of is that it is a systematic approach to improving business processes. What's a business process? It's important for us to understand that in this context, a "process" is not an instance of a running software program, such as the set of processes that are displayed when you type "ps" on a Unix system. In this context, a business process is the set of tasks that are performed by the systems AND the people in an organization in order to achieve that organization's goals. For example, some business processes for an insurance company would be determining risk for selling hurricane insurance in the south eastern USA, preparing actuarial tables for different demographic groups, or deciding on which pieces of junk mail to send to other demographic groups. These processes can be performed by a person, groups of people acting together, people and software services working together, or groups of services working together.

JBoss jBPM [4] is an open source business process management development environment and workflow management system. jBPM includes: a process definition language (jPDL) that combines Java and process definition, a graphical process design tool (GPD) that is implemented as an eclipse plugin, a process engine runtime (deployed to an application server) that supports long-running tasks in business processes, and a web-based console for end users to access the business process' supporting application. The GPD enables users to create process definitions by manipulating a graphic representation of the process.


A business process definition as defined in jPDL and represented in a graph in the GPD consists of multiple types of nodes that can represent tasks or a state in the process, actions which are Java code that is executed when events happen in the process, and transitions between nodes.

It's important to note the importance of state and waiting as supported by jBPM. In a long-running business process, where actions are performed by people and systems, it's often the case that the application program that supports the process has to wait for a person to do something. For example, if the process governs a company's purchasing department, the process may have to pause and wait for a person to manually approve a large purchase. Typically, when you write an application, you want function #1 to call function #2 and so on without introducing wait times. Bur, what if your application has to wait for a person to review some information and do something? We'll walk through an example of how jBPM handles waiting later on in this post. The JBossESB - jBPM integration also enables you to to "orchestrate" ESB services by invoking those services from a process definition. In this context, "orchestration" refers to combining services into a greater whole (application) while retaining some central control. Sort of like how an orchestra is comprised of many different instruments, all under the leadership of a conductor. In contrast, service "choreography" is more in the form of a collection of equal peers or partners.

In reviewing the JBossESB - jBPM integration, the three topics to consider are:

* Making Calls From the ESB to the process definition in jBPM
* Orchestrating ESB Services From jBPM to the ESB
* Exception Handling

JBossESB and jBPM Integration - Making Calls From the ESB to jBPM

Calls from the JBoss ESB into jBPM use the BpmProcessor action. This action actually uses the jBPM command API [5] to execute the calls into jBPM. Of the commands in the command API, the following (3) are available for use from ESB:

NewProcessInstanceCommand creates a new ProcessInstance using a process definition that has already been deployed to jBPM. The process instance is left in the start state so that tasks referenced by start node are executed.

StartProcessInstanceCommand is the same as NewProcessInstanceCommand, except that the process instance that is created is moved from the start position to the first node in the process graph.

As its name implies, CancelProcessInstanceCommand cancels a process instance.

All of these commands requires ESB action attributes such as process ID to be defined. The action definitions can also specify additional information in action configuration properties. For example, the "esbToBpmVars" property defines the variables that are to be extracted from the message from the ESB and set in the jBPM context. We'll look at this property when we examine a JBossESB-jPBM quickstart later in this post.

JBossESB and jBPM Integration - Orchestrating ESB Services from jBPM

Remember how we described jBPM actions as Java code that is executed when selected events happen in the business process? Well, the ESB-jBPM integration handles orchestrating ESB services from a jBPM process with the EsbActionHandler and EsbNotifier action classes. The classes are similar in that they each send a message to the target ESB service. (Don't forget that in the ESB, everything is either a message or a service.) Where they differ is that the EsbActionHandler class blocks the process and waits for a response to the message that it sends to the service. The EsbNotifier follows an asynchronous model and does not block the process and wait for a response.

How are EsbActionHandler and EsbNotifier action classes used? Since the EsbNotifier does not wait for a response to the message that it sends to a service, it should be attached to a jBPM process node's outgoing transition. This will allow the jBPM process to proceed while the message (request) sent to the target ESB service can be processed by a different thread. The EsbActionHandler, in contrast, uses a request/response asynchronous model and should be attached to a jBPM process node. When the EsbActionHandler is called, the node that is attached to waits for it to respond with a transition signal. When the node receives this signal, usually from the JBossESB callback service, the jBPM process can then continue.

JBossESB and jBPM Integration - Exception Handling - When Something Goes Wrong

A common source of problems in when you integrate products together is handling errors. In the JBossESB-jBPM integration, exceptions are handled in different ways, depending on whether a message is being passed from the ESB to jBPM, or from jBPM to the ESB.

JBossESB to jBPM exception handling is the simpler of the two. The jBPM command API (remember that the integration includes these 3 API calls: NewProcessInstanceCommand, StartProcessInstanceCommand, and CancelProcessInstanceCommand) an throw a JbpmException. The exception is not caught by the integration, but rather it goes to the ESB action pipeline, which sends the letter to the DeadLetterService and logs the message. An error message can also be sent to the ESB faultTo EPR if that property is set on the message. You define the handling of the exception by configuring an aptly-named ExceptionHandler to the jBPM process node.

jBPM to JBossESB exception handling is more complicated. If the delivery of the message to the ESB fails, then a MessageDeliveryException is thrown by the ESB ServiceInvoker. Both the EsbActionHandler and EsbNotifier actions can throw this type of exception. If the message is received by the target ESB service, but then a failure happens when it is being processed by the service, then what happens to the exception depends on whether we used an EsbActionHandler or EsbNotifier action. Remember that an EsbNotifier does not wait for a response to the message that it sends to a service. Instead, the jBPM business process continues after the message is sent to the ESB service. So, for an EsbNotifier action, the exception is not sent back to the jBPM process. If, however, an EsbActionHandler was used to send the message to the ESB service, then the jBPM business process is waiting for a response. You define the handling of the exception with a jBPM exception-transition for the EsbActionHandler.

Let's look at this integration in action by examining one of the JBossESB-jBPM quickstarts. The quickstart programs that ship with JBossESB are a great resource for learning about how to use the ESB and its integrations. At first glance, these programs may look like simple examples. But, they are much more than that. They are an extensive set of well documented models of working code that you can use as the starting point for writing your own SOA applications.

Before we examine the quickstart, we have one more setup task to perform. Now, we can run the quickstart with ESB alone installed, but anytime that you work with jBPM, you really have to look at the Graphic Process Design (GPD) tool. One of the most useful and instantly recognizable features of jBPM is how it enables you to construct a visual (graphic) representation of your business process. The process is defined in a file named "processdefinition.xml" You can always view and edit the process definition in the XML source, but its much easier to do so with the GPD.

The quickstart that we'll look at is bpm_orchestration2. This quickstart illustrates how jBPM can control a business process that executes multiple JBoss ESB processes. This quickstart makes use of synchronous calls from jBPM into the JBoss ESB processes. Let's start by looking at the quickstart's source code and its process diagram. To begin, import the quickstart as an existing project into your workspace in JBoss Developer Studio (JBDS). After the project is created, open the processdefinition.xml file to view the process design in the GPD.


One thing that you should remember when you view the process design in the GPD is that you are not looking at a static picture of the process. What you are looking at is a dynamic graphic representation of the process. Any changes that you make to the process in the GPD are reflected in the process definition source.

As I mentioned a minute ago, the bpm_orchestration2 quickstart demonstrates jBPM controlling the execution of JBossESB processes. The quickstart does this by simulating a purchase order placement and approval process. Let's examine just how this happens. To do this, we'll look at two files in the quickstart; processdefinition.xml and the JBoss ESB configuration file, jboss-esb.xml file:

jboss-esb.xml, lines 74-82:

      <action  name="start_a_new_process_instance"
                 class="org.jboss.soa.esb.services.jbpm.actions.BpmProcessor">
                  <property name="command" value="StartProcessInstanceCommand" />
                  <property name="process-definition-name" value="bpm_orchestration2Process"/>
                   <property name="esbToBpmVars">
                      <mapping esb="eVar1" bpm="counter"  value="45" />
                      <mapping esb="BODY_CONTENT" bpm="theBody" />
                  </property>
      </action>

The execution of this quickstart, like almost all of the JBossESB quickstarts, is started when a message is received by a listener defined in the ESB. In this quickstart, when that message is received by the listener, a jBPM process is started and moved from the start position to the first node in the process graph when the StartProcessInstanceCommand jBPM API call is executed.

At this point, let's switch over the the processdefinition.xml to see what happens in the start node. Open the processdefinition.xml file and switch to the "source" tab.

processdefinition.xml, lines 5-21

   <start-state name="start">
    <transition name="" to="Receive Order"></transition>
 </start-state>

 <node name="Receive Order">
       <action name="esbAction" class="org.jboss.soa.esb.services.jbpm.actionhandlers.EsbActionHandler">
          <esbCategoryName>BPM_Orchestration2_Service1</esbCategoryName>
          <esbServiceName>Service1</esbServiceName>
          <bpmToEsbVars>
              <mapping bpm="theBody" esb="BODY_CONTENT" />
          </bpmToEsbVars>
          <esbToBpmVars>
              <mapping esb="BODY_CONTENT" bpm="theBody" />
          </esbToBpmVars>
       </action>
      <transition to="Credit Check"></transition>
  </node>

The start node simply transitions to the "Receive Order" node. This node defines an action using the EsbActionHandler action class. This tells us that the invocation of the action is synchronous in nature and that the jBPM process will wait for the service referenced in the action (in this case, the service named "Service1") to complete before the jBPM process continues. The rest of the quickstart continues this pattern where the jBPM process executes actions that in turn invoke JBossESB services in a synchronous manner.

At this point, let's run the quickstart and see what happens. To do this, the steps are:

Deploy the JBoss ESB archive (/build/Quickstart_bpm_orchestration2.esb) - to do this, execute "ant deploy"
Deploy the jBPM process archive (the contents of the /processDefinition directory) - to do this, execute "ant deployProcess"
Start the process - to do this, execute "ant startProcess"

After the process is started, you should see this in the server log:

20:04:36,886 INFO  [STDOUT] ** Begin Receive Order - Service 1 **
20:04:37,399 INFO  [STDOUT] In: Getting Started
20:04:37,400 INFO  [STDOUT] Out: Getting Started 'Receive Order'
20:04:37,400 INFO  [STDOUT] ** End Receive Order - Service 1 **
20:04:38,952 INFO  [STDOUT] ** Begin Credit Check - Service 3 **
20:04:38,952 INFO  [STDOUT] In: Getting Started 'Receive Order'
20:04:38,953 INFO  [STDOUT] Out: Getting Started 'Receive Order'  'Credit Check'
20:04:38,953 INFO  [STDOUT] ** End Credit Check - Service 3 **
20:04:40,443 INFO  [STDOUT] ** Begin Validate Order - Service 2 **
20:04:40,444 INFO  [STDOUT] In: Getting Started 'Receive Order'  'Credit Check'
20:04:40,444 INFO  [STDOUT] Out: Getting Started 'Receive Order'  'Credit Check'  'Validate Order'
20:04:40,444 INFO  [STDOUT] ** End Validate Order - Service 2 **
20:04:42,704 INFO  [STDOUT] ** Begin Inventory Check - Service 4 **
20:04:42,704 INFO  [STDOUT] In: Getting Started 'Receive Order'  'Credit Check'  'Validate Order'
20:04:42,705 INFO  [STDOUT] Out: Getting Started 'Receive Order'  'Credit Check'  'Validate Order'  'Inventory Check'
20:04:42,705 INFO  [STDOUT] ** End Inventory Check - Service 4 **
20:04:45,093 INFO  [STDOUT] ** Begin Los Angeles - Service 5 **
20:04:45,093 INFO  [STDOUT] In: Getting Started 'Receive Order'  'Credit Check'  'Validate Order'  'Inventory Check'
20:04:45,094 INFO  [STDOUT] Out: Getting Started 'Receive Order'  'Credit Check'  'Validate Order'  'Inventory Check'  'Los Angeles'
20:04:45,094 INFO  [STDOUT] ** End Los Angeles - Service 5 **
20:04:45,381 INFO  [STDOUT] ** Begin Atlanta - Service 7 **
20:04:45,382 INFO  [STDOUT] In: Getting Started 'Receive Order'  'Credit Check'  'Validate Order'  'Inventory Check'
20:04:45,382 INFO  [STDOUT] Out: Getting Started 'Receive Order'  'Credit Check'  'Validate Order'  'Inventory Check'  'Atlanta'
20:04:45,382 INFO  [STDOUT] ** End Atlanta - Service 7 **
20:04:45,660 INFO  [STDOUT] ** Begin Dallas - Service 6 **
20:04:45,661 INFO  [STDOUT] In: Getting Started 'Receive Order'  'Credit Check'  'Validate Order'  'Inventory Check'
20:04:45,661 INFO  [STDOUT] Out: Getting Started 'Receive Order'  'Credit Check'  'Validate Order'  'Inventory Check'  'Dallas'
20:04:45,661 INFO  [STDOUT] ** End Dallas - Service 6 **
20:04:49,357 INFO  [STDOUT] ***** Ship It *****
20:04:49,369 INFO  [STDOUT] In: Getting Started 'Receive Order'  'Credit Check'  'Validate Order'  'Inventory Check'
20:04:49,369 INFO  [STDOUT] Out: Getting Started 'Receive Order'  'Credit Check'  'Validate Order'  'Inventory Check'  'Shipped'
20:04:49,370 INFO  [STDOUT] ***** End Ship It *****
20:04:49,392 INFO  [STDOUT] SUCCESS!:
20:04:49,392 INFO  [STDOUT] [Getting Started 'Receive Order'  'Credit Check'  'Validate Order'  'Inventory Check'  'Shipped' ].

Now, remember that what's happening is that the jBPM process is invoking a JBossESB service, waiting for the process to complete, and then invoking the next JBossESB services until all the services have been invoked and the jBPM process completes. Hmm. I don't know about you, but since I started working in QE, I've become skeptical of just about anything when it seems to "just work." The JBoss ESB services in this quickstart are implemented in simple groovy scripts. The scripts all run very quickly, so it's a little difficult to verify that the jBPM process is actually waiting for each service to complete before invoking the next service in the sequence defined in processDefinition.xml

So, let's make the delay to the jBPM process a little more obvious. If we add the following lines to scripts/service1.groovy and then deploy the ESB and jBPM process archive and run the quickstart again, we should see more obvious delay in the log:

println "Added new println - waiting .........."
for ( i in 0..1000 ) {
  print i + " "
}
println "Added new println - done waiting .........."

To make the change effective, execute "ant undeploy" then redeploy the JBossESB archive and jBPM process archive and run the quickstart again. The server log shows:

21:45:04,966 INFO  [STDOUT] ** End Receive Order - Service 1 **
21:45:04,966 INFO  [STDOUT] Added new println - waiting ..........
21:45:05,105 INFO  [STDOUT] 0
21:45:05,105 INFO  [STDOUT] 1
21:45:05,105 INFO  [STDOUT] 2
21:45:05,106 INFO  [STDOUT] 3
21:45:05,106 INFO  [STDOUT] 4
.
.
.
21:45:05,312 INFO  [STDOUT] 997
21:45:05,312 INFO  [STDOUT] 998
21:45:05,312 INFO  [STDOUT] 999
21:45:05,313 INFO  [STDOUT] 1000
21:45:05,313 INFO  [STDOUT] Added new println - done waiting ..........

And then, the rest of the JBossESB services are invoked in sequence, so, yes, the call to the service was synchronous and the process waited for the service to complete before it continued.

Closing Thoughts

As we discussed in the introduction to this post, one of the great strengths of the SOA Platform is the set of integrations that it supports. With its integration with jBPM you can combine a service oriented architecture implemented through the ESB with the flexibility of building applications based on your business processes designed with the jBPM graphical process designer. And, by using the JBossESB-jBPM integration, your applications can handle awkward situations where the application has to wait for a service, or a person, to perform an action. (So, unlike what Tom Petty said, waiting is not the hardest part... ;-)

References

[1] http://www.ushistory.org/franklin/quotable/quote71.htm
[2] http://www.jboss.org/jbossesb/
[3] Business Process Management definitions: http://searchcio.techtarget.com/sDefinition/0,,sid182_gci1088464,00.html
http://en.wikipedia.org/wiki/Business_process_management
[4] jBPM: http://www.jboss.org/jbossjbpm/ and the JBossESB jBPM Integration Chapter in the JBoss ESB Services Guide: http://www.jboss.org/jbossesb/docs/4.5.GA/manuals/html/ServicesGuide.html
[5] http://docs.jboss.com/jbpm/v3.2/javadoc-jpdl/

Acknowledgements

I want to thank the members of the JBossESB (see http://anonsvn.labs.jboss.com/labs/jbossesb/trunk/Contributors.txt) and jBPM projects and the SOA Platform - especially Burr Sutter, Mark Little, Kurt Stam, and Tom Baeyans - for their help and review comments. Also, this post relies heavily on the extensive JBossESB and jBPM user documents and the quickstarts. (And, I'd also like to thank Bascha Harris for her feedback on the post!)