Neuron ESB Workflow Environment

Intro

Neuron ESB 3.5 Workflow brings long-running transactions and complex business processes to the enterprise service bus. Using workflow, it is possible to build business processes that can span days, weeks, or even months coordinating business activities, responding to business inputs, and integrating business systems. Neuron ESB 3.5 provides a complete Workflow hosting environment for running workflows as part of, or independent of, your ESB messaging solution.

Features

Figure 3 illustrates the features of Neuron ESB 3.5 Workflow. The blue highlighted elements are the core features of the Microsoft .NET Workflow Foundation (WF) that come as part of the .NET Framework. The green highlighted elements are the additional features that Neuron ESB provides on top of WF for use in enterprise environments.

Neuron ESB’s 3.5 Workflow is built on WF that was originally introduced in .NET 4.0 and improved upon in .NET 4.5. Although Neuron ESB uses WF to manage workflow execution and persistence, significant work was undertaken to make WF manageable, fault tolerant and truly enterprise-ready, including the development of the following:

• Workflow Designer
• Workflow Types
• Workflow Execution Engine
• ESB Message Integration
• Workflow Tracking and Playback
• Workflow Control and Monitoring
• Workflow Samples

Workflow Designer

Neuron ESB hosts its own Workflow Designer within the Neuron ESB Explorer. The Workflow Designer hosted in the Neuron ESB Explorer is the same designer that developers use inside of Microsoft Visual Studio to design and build workflows for .NET applications. With Neuron ESB, developers do not need to leave the Neuron ESB Explorer environment or have Visual Studio installed in order to build and edit workflows. Neuron ESB maintains compatibility with Visual Studio workflows allowing workflows to be imported and exported between both environments.

Using the Neuron ESB Workflow Designer, users can create new Workflows or import existing Workflows previously created in Visual Studio by selecting either “Create Workflow” or “Import Workflow” from the Processes toolbar.

Workflow Activity Toolbox

The Workflow Activity Toolbox is located to the right of the Workflow Designer and contains 79 Workflow Activities. Although many are the standard Workflow Activities that ship as part of WF, others are Neuron ESB specific that enable interaction with Neuron ESB Messaging, Adapters and Service Endpoints or provide some additional level of interaction with the Neuron ESB Messaging system.

Neuron ESB also supports WF custom developed activities as well as external assembly references which can be added through the Neuron ESB Workflow Designer toolbar.

Custom activities can be added to the Toolbox by copying activity assemblies and dependent assemblies to the Workflows folder under the Neuron ESB instance installation folder and restarting Neuron Explorer (ex: C:\Program Files\Neudesic\Neuron ESB v3\DEFAULT\Workflows ). When custom activities are added, they will show up in the Toolbox where they can then be dragged onto the surface of the Workflow Designer. The out-of-the-box Neuron ESB specific Workflow Activities are listed in the table below.

Comment Out	The Comment Out activity is useful when you need to disable the execution of one or more activities, but do not want to delete the activities from the workflow. The Comment Out activity is a container activity that can hold a single activity or multiple activities contained in a Sequence activity. At execution time, the Comment Out activity will prevent the activities that it contains from executing.
C#	The C# activity executes a C# code fragment. The code fragment is compiled at runtime into a dynamic assembly. The C# code activity can reference classes in the .NET Framework or reference third-party libraries. The C# code activity can also interact with any variables of the workflow that are accessible to the activity. This is identical to the C# Process Step and supports full IntelliSense, and opens into a Visual Studio style editor.
C# Class	The C# Class activity gives the developer more freedom than the C# activity and allows the developer to define a custom class that will be executed. The C# Class activity has the ability to get or set the values of variables in the activity’s scope. The main advantage of the C# Class activity over the C# activity is that the developer can define methods and make larger code fragments more readable than the C# activity. This is identical to the C# Class Process Step and supports full IntelliSense, and opens into a Visual Studio style editor.
JavaScript	The JavaScript activity allows the user to execute dynamic code written in JavaScript. The JavaScript activity uses the Google v8 JavaScript engine to execute workflow-specific custom logic. This is identical to the JavaScript Process Step and opens into a Visual Studio style editor.
Visual Basic .NET	The Visual Basic .NET activity executes a Visual Basic .NET code fragment. The code fragment is compiled at runtime into a dynamic assembly. The Visual Basic .NET code activity can reference classes in the .NET Framework or reference third-party libraries. The Visual Basic .NET code activity can also interact with any variables of the workflow that are accessible to the activity. This is identical to the Visual Basic .NET Process Step and supports full IntelliSense, and opens into a Visual Studio style editor.
Execute Process	The Execute Process activity allows developers to reuse existing message processing that has been built using Neuron ESB’s Business Process designer. With the Execute Process activity, a workflow developer can choose an existing Business Process to execute and can send a message into the process and capture the message output by the process.
Deserialize Data Contract	The Deserialize Data Contract activity will accept an XML message and will use the .NET DataContractSerializer class to deserialize the XML into an object. The Deserialize Data Contract activity is a generic workflow activity and the workflow developer will be prompted at design time to choose the class type for the deserialized object.
Serialize Data Contract	The Serialize Data Contract activity will take a DataContract object and will serialize the object to XML using the .NET DataContractSerializer class.
Clone Message	The Clone Message activity will accept an ESBMessage object and will create an identical copy of the ESBMessage. This activity can be useful when publishing the message to other topics or when it is necessary to change the body or headers of the message during workflow processing.
Create Binary Message	Create Binary Message will create an ESBMessage object containing a binary body. The Create Binary Message activity accepts the body as an array of bytes and will output an ESBMessage object.
Create Reply Message	The Create Reply Message activity will accept an ESBMessage object and will generate another ESBMessage that can be used to return a reply for the original message. The Create Reply Message activity is useful when building Request/Reply workflows.
Create Text Message	The Create Text Message activity will create an ESBMessage object containing a text body. The Create Text Message activity accepts a body as a string and will output an ESBMessage object.
Publish Message	The Publish Message activity will let a workflow publish a message to a Neuron ESB topic. The Publish Message activity can specify correlation settings that the workflow runtime will use to find and route reply messages back to the workflow. If a Correlation Set is used, the Semantic property must be set to Multicast. When testing at design time the solution must be loaded into the local Neuron ESB Runtime. The runtime must be started. Lastly, a valid Source ID and Topic must be entered into the Edit Message Dialog when initiating the test.
Receive Message	The Receive Message activity is used in correlated workflows to receive messages that have been routed to the workflow based on configured correlation settings. When running correlated workflows, messages will be queued in the Neuron ESB database for the workflow instance. The Receive Message activity will retrieve the next message from the queue and will return the message to the workflow for processing.
HTTP GET	The HTTP GET activity will execute an HTTP GET request against an HTTP service such as a web service or website. The HTTP GET activity will return the body of the data that was received from the remote web service.
HTTP POST	The HTTP POST activity will execute an HTTP POST request against a web service or website. The HTTP POST activity will allow the workflow to specify the body of the request to be sent to the remote web service.
Write to Event Log	The Write to Event Log activity allows the workflow to report a message to the Windows Event Log.
Invoke PowerShell	The Invoke PowerShell activities are used to execute PowerShell scripts inside of a workflow. There are two PowerShell activities. The second activity is capable of returning a result from the PowerShell script to the workflow. Each requires the installation of PowerShell 3.0 or greater.
Is Match?	The Is Match activity evaluates a string value using a regular expression to determine whether the string is a match. The Is Match activity will return a Boolean result.
Matches	The Matches activity will execute a regular expression against a string value and will return the string fragments that match the specified regular expression.
Replace	The Replace activity will execute a regular expression against a string value and will replace the matched text with an alternative string or value.
Database Query	The Database Query activity will execute a query against a SQL database and will return the result of the query.
Query DataSet	The Query DataSet activity will execute a query against a SQL database and will return the results of the query in a .NET DataSet object for processing.
Query Scalar	The Query Scalar activity will execute a query against a SQL database and will return the single result of the query.
Database Update	The Database Update activity will execute a SQL update command against the database. The Database Update command will typically be used to perform INSERT, UPDATE, or DELETE SQL statements.
Get Workflow Instance ID	The Get Workflow Instance ID activity is useful when the workflow needs to know the identifier for the executing workflow. Each instance of a workflow has a unique UUID value that is generated automatically when the workflow instance is created. The Get Workflow Instance ID activity will return the identifier for the current executing workflow. This activity can be useful for logging or other diagnostic uses.
Deserialize From JSON	The Deserialize From JSON activity will accept a JSON string and will deserialize the JSON into an object. Deserialize From JSON uses the JSON.NET library to deserialize the JSON object into a .NET object.
Serialize To JSON	The Serialize To JSON activity will take a .NET object and will generate the JSON representation of the object. The Serialize To JSON activity uses the JSON.NET library to serialize the object to JSON.
Publish Request Message	The Publish Request Message activity will publish a message as a request to a Neuron ESB topic and will wait for the reply to be returned. While waiting for a reply, the workflow may become idle and unload to allow another workflow to execute. When the reply is received, the workflow will be reloaded from the database and will continue execution. When testing at design time the solution must be loaded into the local Neuron ESB Runtime. The runtime must be started. Lastly, a valid Source ID and Topic must be entered into the Edit Message Dialog when initiating the test.
Adapter Endpoint	The Adapter Endpoint activity allows a workflow to call a configured Neuron ESB adapter endpoint directly without the need of publishing a message to an ESB topic. The Adapter Endpoint activity will allow the workflow to send a message to or receive a message from a configured Neuron ESB adapter endpoint.
Service Connector	The Service Connector activity allows a workflow to send a message to a configured Neuron ESB service connector directly without the need of publishing a message to an ESB. The Service Connector activity is useful when interacting with web services where a service connector is already configured. When testing at design time the solution must be loaded into the local Neuron ESB Runtime. The runtime must be started. Lastly, a valid Source ID and Topic must be entered into the Edit Message Dialog when initiating the test.
Audit Message	The Audit Message activity functions identically to the Audit Message Process Step. Allows messages to be selectively audited into the Neuron ESB message history or failed message reports. When testing at design time the solution must be loaded into the local Neuron ESB Runtime. The runtime must be started. Lastly, a valid Source ID and Topic must be entered into the Edit Message Dialog when initiating the test.
Split Xml Message	The Split Xml Message activity allows users to split incoming batch files into their individual records, storing those records into a List<ESBMessage> collection that can be operated on. Users can configure the Split Xml Message with an XPATH statement to determine the boundary of individual records within the batch file.
Join Xml Messages	The Join Xml Messages activity compliments the Split Xml Message. It takes the collection of messages (i.e. List<ESBMessage>) and outputs them as a single aggregated message. Users can define the root node as well as namespace for the outgoing batch file.
Validate Xml	The Validate Xml Activity can be used to validate an Xml message against a set of Xml schemas. This activity is identical in both functionality and property configuration to the Validate – Schemas Process Step. If validation is not successful, an exception is thrown indicating the reason why validation failed.
Transform Xml	The Transform Xml Activity can be used to apply an Xslt/Xsl transform to an XML message. Additionally parameterized Xslt/Xsl is supported. Parameters can be useful when the same value must be repeated many times within the document. This activity is identical in both functionality and property configuration to the Transform – Xslt Process Step. If the transform is not successful, an exception is thrown indicating the reason why transformation failed.

Workflow Simulation

Neuron ESB’s Workflow Designer supports similar testing/simulation features as the Neuron ESB Process Designer. Using the Neuron ESB Workflow Designer, users have the ability to simulate workflow execution for testing at development time. Neuron ESB Explorer’s test runner allows developers to pass messages to the workflow and to monitor the workflow’s execution in real-time. Several of the workflow activities also support the simulated runtime environment. For example, the Receive Message activity will allow developers to test receiving messages during a workflow’s execution. There are other Workflow Activities that support a combination of design time and run time testing. Specifically, the Publish Message, Publish Request Message, Audit Message, Adapter Endpoint and Service Endpoint Workflow Activities can all be tested at design time while interacting directly with the local Neuron ESB runtime solution. To test these, the local Neuron ESB runtime service needs to be loaded and started with the current solution and a valid Source ID and Topic must be provided in the Edit Test Message dialog.

Simulation/Testing is started by pressing the Test Workflow button on the Workflow Designer’s toolbar as shown in Figure 6. This will prompt the user for a message via the Edit Test Message dialog (the same dialog used in the Business Process Designer). When the user presses the OK button, the simulation will begin. Each step that executes will be highlighted in Yellow, with any outputs (WriteLine workflow activity or anything that writes to System Console) being written to the Output window located at the bottom of the Workflow Designer. Any errors that occur will be written to the Errors window located at the bottom of the Workflow Designer.

Workflow Types

When users select the “Create Workflow…” option from the Neuron ESB Processes menu, they are presented with a prompt that allows them to choose from 3 basic types of Workflows i.e. Normal, Request/Reply or Correlated Workflow shown in figure 8.

Normal Workflows are used primarily where a response is not expected to be returned from the Workflow. Request – Reply Workflows are what they infer, where a Request message starts a Workflow and that Workflow instance sends back a response to the client/system that initiated the original Request. Correlated Workflows are a special type that defines a unique set of messages to be processed by a single instance of a Workflow.

Normal Workflow

Normal Workflow types are most commonly used to create Workflows that subscribe to messages and execute an instance of a Workflow for each message received. When a user selects to create a Normal Workflow, a Workflow definition is created within the Workflow Designer. Once the Workflow is completed, it’s must be saved and associated with a Workflow Endpoint. A Workflow Endpoint is used to associate the Workflow definition with a Neuron ESB Subscriber, Topic and Availability Group.  Workflows essentially “subscribe” to messages published to the bus.

When a Normal Workflow is created within the Workflow Designer, the Neuron.Esb namespace as well as 3 arguments specific to Neuron ESB Messaging are added to the Workflow definition, allowing any activity within the Workflow to interact directly with Neuron ESB Messaging or Configuration.

As shown in Figure 9, these arguments can be used directly within any activity, including all the Neuron ESB Code Activities such as the C#, C# Class, JavaScript and Visual Basic.NET Activities.

The message argument represents the original Neuron ESB Message that the Workflow will be initiated by at runtime. All of its content and properties are accessible both during design time testing as well as runtime as shown in Figure 10.

The environmentVariables argument can be used to retrieve values specific to the runtime environment that the Workflow instance is running in (i.e. Production, Staging, QA, etc.). Neuron ESB Environment Variables are defined and managed within the Neuron ESB Explorer and located under Deployment->Environments->Environment Variables section and can be used to configure any Business Process step, Database connection, Adapter and Service Endpoints. Many developers will create application specific Environment Variables, retrieve them at runtime and use their values to drive the business logic within their custom Business Processes or Workflows.

Lastly, the configuration argument provides access to the entire Neuron ESB solution configuration. Almost all elements of a Neuron ESB solution can be accessed through this object. This can be useful to retrieve XSLT or XML documents, encryption keys, certificates and any other entity contained within the Neuron ESB Solution.

Request-Reply Workflow

Request-Reply workflows are easy to build using the Neuron ESB Workflow Designer by selecting Request-Response Workflow in the prompt displayed in figure 11.

By following the pre-defined pattern of accepting a request as an input argument and outputting a message when the workflow terminates, the Neuron ESB workflow engine will automatically return the output message as a reply to the Neuron ESB Party that originally sent the request message. This could be a remotely hosted Neuron ESB Party, an Adapter or even a Neuron ESB Client Connector. The Neuron ESB workflow engine was designed to fit into the messaging patterns employed by Neuron ESB users.

Request-Reply Workflow arguments differ slightly from Normal Workflows. The inbound Neuron ESB Message argument is named request (named message in Normal Workflows), while the outbound Neuron ESB Message argument is named reply. Use the Create Reply Message activity to set the reply argument and additional code to set its properties. The reply argument contains the final Neuron ESB Message that is returned to the original calling Neuron ESB Party as shown in figure 12.

Correlated Workflows

The last type of Workflow that can be created is a Correlated Workflow as shown in figure 13.

Correlated Workflows are a special type of Workflow that employs Custom Correlation at the workflow endpoint level. Custom Correlation is user-defined criteria that controls what set of messages a single instance of a Workflow will process. This set of messages will be routed to a specific instance of a Workflow (i.e. Singleton pattern), whether that Workflow is running or currently dehydrated in the database. Custom Correlation determines the “uniqueness” of a set of messages. Although many instances of a Workflow may still execute, each instance could be processing a set of messages received from the bus, rather than just one, as in the case of Normal Workflows. The first message that launches the instance of the workflow is used to initialize the values of the Correlation Set.

For example, if all messages from a single publisher needed to be processed by the same workflow, a user can choose to correlate messages based off of the session identifier and source identifier. This will result in all messages sent from the same party on the same connection to be processed by the same workflow instance.

Another example would be where a large file was previously split into individual records and published to the bus. A Workflow could be used to aggregate all of the related messages (the individual records) into a new outgoing file. In this case, any combination of message content, custom header, SOAP or HTTP header or even regex and XPATH expressions against the body of the message could be used to define the correlation set for a Workflow instance.

A Receive Message Workflow Activity is used within a Correlated Workflow to receive all messages that match the Correlation Set definition on the Workflow Endpoint. Usually this is placed within a While or similar loop within the Correlated Workflow. Hence the Receive Message Workflow Activity “follows” the Correlation Set that was initialized when the Workflow Instance was first started (Figure 14).

When a user creates a Correlated Workflow and assigns it to a Workflow Endpoint, the Correlation Set tab of the Workflow Endpoint is enabled. This allows the user to define the necessary properties that control what messages are processed together by a single instance of a Workflow as shown in the figure below.

Correlation is not limited to Correlated Workflows. Correlation sets can also be defined within any type of Workflow using the Publish Message Workflow Activity followed by the Receive Message Workflow Activity.

Workflow Execution Engine

Neuron ESB introduces Workflow Endpoints and Availability Groups for hosting workflows. A Workflow Endpoint is a first-class citizen in the Neuron ESB ecosystem, similar to Service Endpoints or Adapter Endpoints. A Workflow Endpoint is associated with a specific Workflow Definition created with the Neuron ESB Workflow Designer. It acts in a subscriber role and is configured to receive messages from a Topic by a specific Subscribing Party. Lastly, it’s assigned to a specific Availability Group which serves as the runtime host (Neuron ESB workflow engine) for managing the execution of the Workflow Definition associated with the Workflow Endpoint. Users can create any number of Availability Groups which in turn can be assigned to any number of Workflow Endpoints.

Neuron ESB’s Workflow Endpoints also employ configurable limits on workflow execution in order to not overload host servers. The Neuron ESB workflow engine utilizes a persistent queue built inside of SQL Server to schedule the execution of workflow instances. The persistent queue allows workflows to be interrupted if it is necessary to restart the ESB Service or the host service, and the workflow engine can restart work processing when the Workflow Endpoint is restarted.

While Workflow Endpoints contain the configuration specific to setting up the environment for executing a Workflow, Availability Groups use the configuration to provide the actual runtime hosting environment for the Workflow. Availability Groups introduce a new form of isolation and reliability inside of Neuron ESB’s Runtime Windows service. Availability Groups are isolated into their own process space and execute as child processes of the Neuron ESB Runtime Windows service. If a fatal error occurs that causes the Availability Group to fail, the Neuron ESB Server can restart the Availability Group on the same server, or on a different server when Neuron ESB is being run in a clustered configuration (see Figure 2).

Through the Deployment Settings tab of an Availability Group, users can configure on what servers, in which specific Deployment Groups they want an Availability Group host to run on. When in a clustered configuration, the Neuron ESB Workflow Engine will schedule execution of Workflows to the selected Primary servers within the clustered configuration, monitoring CPU, Memory and the configurable Max Concurrent Workflows property on the Settings tab of the Workflow Endpoint. Workflow execution will be evenly spread across all configured servers depending on their resource limits.

Deploying Workflows

Neuron ESB 3.5 makes deploying Workflows just as easy as deploying most other entities within Neuron. Once a Workflow Definition has been created and saved (i.e. click the Apply button followed by the Save button), an Availability Group must be created to serve as the runtime host for the Workflow. If one already exist, it may be practical to reuse it. Once the Availability Group has either been created or identified, a Topic and Subscriber must be created. Lastly, a Workflow Endpoint must be configured for the Workflow Definition using the Availability Group, Subscriber and Topic. Deploying a Workflow involves these 5 entities:

• Workflow Definition
• Availability Group
• Topic
• Subscriber
• Workflow Endpoint

In Neuron ESB 3.5, each Workflow entity created within the Neuron ESB Explorer (i.e. Availability Group, Workflow Endpoint, Workflow Definition) are saved like all other entities; as individual XML files within dedicated folders on disk. The same deployment methods are used with these as are used with other Neuron ESB entities such as Topics and Endpoints. This also means that when changes are saved, or these entities are deployed to another server, the Neuron ESB Runtime and Workflow host will automatically detect the additions, deletions or changes and load them for execution.

ESB Message Integration

The Neuron Workflow engine and Designer provide a number of integration points into the Neuron ESB Messaging system, some of which are:

• Failed Message Reporting
• Publishing to Topics
• Service Endpoints
• Adapter Endpoints
• Auditing Messages

Failed Message Reporting

Neuron ESB’s workflow engine integrates directly with Neuron ESB’s auditing service to report on messages that failed while being processed by workflows. This furthers Neuron ESB’s goal not to lose messages during transport and processing of messages. Messages that fail during workflow execution can be viewed using Neuron ESB Explorer’s Failed Messages report. Additionally, all the messages of a failed or in flight workflow can be inspected through the new Neuron ESB Workflow Tracking console.

To support integration with Failed Message Reporting, additional properties were added to the Failed Message report, specifically the Workflow Name, Workflow Instance ID and Workflow Endpoint Name. This allows users to correlate failed messages back against the specific instances of a Workflow that generated them.

Publishing to Topics

Just as in the Neuron ESB Business Process Designer, users can directly publish messages to Neuron ESB Topics using the Publish Message and Publish Request Message Workflow Activities.

The Publish Request Message Workflow Activity allows users to make request/response type calls over the bus by publishing the request message to a Topic (must be Topic other than that configured in the Workflow Endpoint). Neuron ESB will route the message to a subscribing party which could either be an Adapter, Service Connector or a remotely hosted Party (endpoint). That party will process the request and publish the response back to the bus at which point Neuron ESB will return the response back to the Workflow instance that initiated the request. This can be used for bus driven, decoupled, request/response calls.

Alternatively, the Publish Message Workflow Activity can be used to publish messages to a Topic using any of the other Neuron ESB message semantics such as Multicast (fire and forget), Reply, Direct or Routed. Once published, the Neuron ESB Messaging system will route the message to all eligible subscribers.

The Publish Message Workflow Activity can also be used for Correlated request/reply scenarios. Users can define a Correlation Set (a property on the Workflow Activity) and set the semantic to Multicast. Once this is done, the outgoing message that is published to the Topic from the Workflow will be used to initialize the values of the Correlation Set. The initialized values for the Correlation Set are persisted to user defined correlation variable. To receive the expected response message, a Receive Message Workflow Activity must be added to the Workflow following the Publish Message Workflow Activity and be configured with the correlation variable.

For example, imagine a common partner scenario where a request is sent to a vendor. However, the response may not be received immediately. In fact, it could be hours or days before the response is sent back from the Vendor. In that time, the Workflow would have been unloaded from memory and persisted to the database. The runtime environment may have also been restarted or even moved. Once the response message is received, it’s the job of the Neuron ESB runtime environment to route the response back to the correct “instance” of the Workflow, i.e. the Workflow instance that sent out the original request. In many scenarios like this there will not be a unique identifier within the response message to correlate on. In those cases users can define on the outgoing request message a Correlation Set. Correlation Set is a property exposed by the Publish Message Workflow Activity. When activated it will display the Correlation Set dialog as shown in the figure below. This is identical to the user interface of the Correlation Set tab of the Workflow Endpoint.

In the case of a purchase order, a user may define a Correlation Set as a combination of customer number, vendor id and purchase order number. When a response message is published to the bus to the expected topic, those property values will be retrieved and evaluated and if a match is found, the response message will be routed to the Receive Message Workflow Activity that directly follows the Publish Message Workflow Activity on the correct instance of the Workflow. The Receive Message Workflow Activity must be used either with a Correlated Workflow, or it must follow a Publish Message Workflow Activity.

Service Endpoints

Neuron ESB includes a Service Broker that enables organizations to deploy Neuron ESB as a Service Gateway, providing mediation, security, hosting and a number of other services. Service Connectors are essentially registrations within Neuron ESB that point to existing services hosted within an organization, partner or cloud domain. In previous versions of Neuron ESB, the only way to route a request message to a Service Connector (externally hosted service registered with Neuron ESB) was by publishing the request to a Topic, whose underlying publishing service would then route to the Service Connector. This meant that every web service request received by Neuron had to go through the pub/sub infrastructure if an externally hosted service had to be called. Although the pub/sub Topic Transport of choice would usually be an in-memory option, this still led to additional overhead and configuration at runtime.

Neuron ESB 3.5 provides both a Service Endpoint Workflow Activity and Process Step that can be used with either the new Neuron ESB Workflow Designer or the existing Business Process Designer. Either one allows a user to directly call any Service Endpoint registered with Neuron, without the need to publish a request to a Topic, eliminating all pub/sub overhead.

This allows users to create either a Workflow or Business Process that defines an end-to-end solution without the pub/sub abstraction in the middle.

Adapter Endpoints

A core feature of Application and Data Integration servers is their ability to bridge 3rd party applications, databases, technologies, protocols or transports. Neuron ESB provides this through either its library of built in adapters and by allowing users to build their own adapters using the Neuron ESB Adapter Framework. In many ways, Adapters provide capabilities similar to those found in Neuron ESB’s Service Broker specifically:

• Bridging external endpoints
• Functioning as subscribers

Just as with Service Connectors, Adapter Endpoints would normally need to be called through the Neuron ESB Messaging system where a message is published to a Topic and then routed to eligible subscribers, one of which could be an Adapter Endpoint.

Neuron ESB 3.5 provides both an Adapter Endpoint Workflow Activity and a Process Step that can be used with either the new Neuron ESB Workflow Designer or the existing Business Process Designer. Either one allows a user to directly call any Adapter Endpoint registered with Neuron, without the need to publish a message to a Topic, eliminating all pub/sub overhead (Figure 21).

This allows users to create either a Workflow or Business Process that defines an end-to-end solution without the pub/sub abstraction in the middle. These activities and process steps should always be used with any request/response type of messaging since there will never be more than one subscribing endpoint fulfilling the request. For fire-and-forget messaging (i.e. multicast/datagram), unless there is a need to decouple using the topic-based pub/sub engine as in the case where the publishing process should not know who the subscribing endpoints/parties are, then using these activities and process steps would be a preferred approach.

Auditing Messages

Neuron ESB provides message tracking capabilities at several levels, commonly referred to as Message Auditing. When used, messages either published to or subscribed to through the Neuron ESB Messaging system are placed in the Neuron Audit tables (database) where they can be viewed, searched for, edited and resubmitted back to the bus within the Message History report. The first level most commonly used is configured at the Topic level as shown in the Figure below.

By default, this provides blanket auditing. In other words, all messages that are published to or subscribed to from the Topic will be stored in the Neuron Audit tables. The properties on the Auditing tab of the Topic control the time and granularity of the messaging auditing feature. For example, users can choose to “audit” either before or after a process executes or determine whether or not either the message data or custom properties will get tracked, along with the exception stack. We call this level of Auditing “Global” since it tracks everything that flows through a Topic.  In many cases though, users require a more granular level of control over message Auditing. Specifically, the ability to Audit a message at a point in a process where it matters most. In some cases it may be desirable to audit just a fragment of the message rather than the entire message. For those more common scenarios Neuron ESB provides a configurable Neuron Audit Process Step that users can add to an existing Neuron ESB Business Process to strategically Audit a message as part of a defined process.

The Neuron ESB Audit Process Step can also be used to capture an exception in a Business Process and store the affected message, its context and the exception information are placed in the Neuron Audit tables (database) where they can be viewed, searched for, edited and resubmitted back to the bus within the Failed Message report. This similar feature is also exposed by Service and Adapter endpoint policies, whereas if a delivery failure occurs the policy can be configured to capture the message, context and exception information and place all into the Neuron Audit tables, managed through the Failed Message report.

In Neuron ESB 3.5, the Neuron ESB Audit Process Step has been completely ported to a Workflow Activity, where it can be used in exactly the same way as it is used today in a Neuron ESB Business Process. Users can drag the Audit Message Workflow Activity onto the Workflow designer, configure it to capture a specific message, set the properties. Now that message will be viewable, searchable and editable within either the Message History or Failed Message reports (depending on context) as shown in the figure below.

Workflow Tracking and Playback

Neuron ESB’s workflow engine implements proprietary tracking providers that are used to collect and report execution history for the workflow. The primary tracking provider stores workflow execution tracking events and data into a Microsoft SQL Server database. Once collected in the database, Neuron ESB Explorer provides the monitoring interface that developers or administrators can use to observe the status of each executing workflow, and if desired, play back the workflow’s execution. Using Neuron ESB Workflow Tracking, users can observe the execution history and state transitions of the workflow. Users can also step through the execution of the activities for the workflow, viewing both the input arguments and the values that are output by each workflow activity. Users are also able to explore error conditions and view exception messages for errors that occurred during the workflow.

Neuron ESB Workflow Tracking (seen in the Figure below) is the central user interface for querying and viewing the tracking information for all workflow instances; those that are in process as well as those that have completed.

The Workflow Tracking interface allows users to group, sort, filter, search and delete all information related to the execution of Workflows. Each row in the table represents a unique instance of a Workflow. Double-clicking on any row will bring up the detailed tracking profile of the Workflow Instance. Users can examine the state of the messages, variables and arguments at each stage of the Workflow Instance by navigating through the Tracking events (Tracking Tab), or highlighting a Workflow Activity in the Workflow instance diagram displayed in the main pane. Users can open any ESB Message variable into the Edit Message dialog where it can be edited and resubmitted back to the Neuron ESB Messaging system.

Lastly, all the rows in the current view can be exported to Excel by right clicking the grid and selecting “Export to Excel…” from the context menu.

Workflow Control and Monitoring

Neuron ESB 3.5 provides Workflow control and monitoring through the new Workflow Tracking, Neuron ESB Endpoint Health and WMI Performance Counters. Workflow Tracking provides users with control commands such as Start, Suspend, Cancel, and Abort or Delete that can be used against any selected Workflow instance or group of Workflow Instances. Control commands for Workflow instances are context sensitive, depending on their current state. For example, a Workflow Instance that is in an unloaded or suspended state can be Started, Cancelled or Aborted. An Aborted, Completed or Cancelled Workflow Instance can be deleted while a Started one can be suspended. Control commands are accessed through the right click context menu of the Workflow Tracking grid (as shown in the figure below).

While Workflow Tracking provides detailed information for each Workflow instance, both executing and completed, a summary view of real time Workflow execution activity can be viewed through the Neuron ESB Endpoint Health interface.

Neuron ESB Endpoint Health has been refactored to accommodate real time activity monitoring for the Workflow hosting environments (Availability Groups) as well as the Workflow Endpoints that they host. The new user interface provides users the ability to group and sort, and now lists all machines in the selected deployment group. The Rate measurement has also been changed.  In previous versions of Neuron ESB, the Rate column was calculated based on the last time the Neuron ESB runtime was started. However, this meant that the value reflected could become less useful over time if there was not a steady continuous stream of processing. The Rate is now calculated based on the refresh window (default is 15 seconds which can be changed in Zone settings). Effectively we calculate a new rate every 15 seconds based on what was processed in that window.

The Neuron ESB Endpoint Health interface has a scalable horizontal divider, separating Neuron ESB Messaging entities such as Topics, Adapter and Service Endpoints from Workflow entities such as Availability Groups and their associated Workflow Endpoints. A context menu is exposed at the entity level that allows users to restart any selected entity.

Lastly, two new WMI Performance Counter groups are introduced with Workflow; “Neuron AvailabilityGroups” and “Neuron Workflow Endpoints”. Although “Neuron AvailabilityGroups” only exposes counters to track Errors and Warnings, “Neuron Workflow Endpoints” exposes a number of counters, including:

• Aborted
• Active
• Cancelled
• Completed
• CompletionRate
• Errors
• Idle
• PendingEvents
• PendingTime
• Persisted
• Terminated
• WaitTime
• Warnings

These WMI counters can used by third party tools for remote monitoring solutions as well as used within Microsoft Performance Monitor.

Neuron ESB WMI Performance Counters Installation

The WMI Counters for Workflow are created by the Setup.exe installer. This feature is represented on the Feature Install page of the installation wizard by the “ESB Service Management Objects” and is disabled (unchecked) by default. Neuron ESB Workflow, specifically the ability to monitor Workflow activity within Endpoint Health, requires that this feature be selected and installed.

Workflow Samples

Neuron ESB 3.5 ships several new Workflow samples accessible through the Neuron ESB Explorer’s Sample Browser, shown in the figure below:

There are many Workflow Foundation (WF) samples available on the Microsoft Developer Network. Since Neuron ESB 3.5 hosts its own Workflow Foundation runtime, many samples from Microsoft can be used in Neuron ESB with little or no modifications.

Sample workflows can be imported into Neuron ESB Explorer.  This has the advantage of being able to reuse existing workflows and allows users to modify the workflow in the Neuron ESB Workflow Designer.   

Note: The Microsoft Windows Workflow (WF) Samples can be downloaded and installed from the download link found on the Windows Workflow Samples page.

The new Neuron ESB Workflow samples that are included in 3.5:

Developing a Custom Workflow Activity

This sample demonstrates how to create a custom workflow activity that generates lorem ipsum text.  To test, compile the solution, copy the assembly to the \Program Files\Neuron ESB v3\DEFAULT\Workflows folder and restart the Neuron Explorer. Includes: Visual Studio solution and documentation.

Importing a Microsoft Workflow Sample

This sample demonstrates how to import the Switch activity that ships with the Microsoft samples into Neuron ESB Explorer.  This sample contains only documentation and requires that you install the Windows Workflow (WF) Samples from the Microsoft site.

Microsoft Workflow Sample as a Custom Workflow Activity

This sample demonstrates how to use the FlowChartWithFaultHandling activity that ships with the Microsoft samples into Neuron ESB Explorer.  This sample contains only documentation and requires that you install the Windows Workflow (WF) Samples from the Microsoft site.

Normal Workflow

This sample shows a simple Normal workflow that logs the text of the incoming ESBMessage as well as the Workflow Instance ID. A Normal workflow is executed once for each message sent to the topic associated with the Workflow Endpoint.

Request-Reply Workflow

This sample shows a Request-Reply workflow that accepts a promotion code on the text property of the incoming ESBMessage and replies with a new ESBMessage where the text property contains the discount corresponding to the promotion code. A Request-Reply workflow is executed once for each message sent to the topic associated with the Workflow Endpoint.

Correlated Workflow

This sample shows a Correlated Workflow that joins order records after a Process has split them into separate messages. A Correlated-type workflow is executed once for a unique set of messages determined by the correlation set properties configured on the Workflow Endpoint. This is sometimes referred to as a Singleton pattern. In this sample, orders are assigned to batches and the batch ID is used to initialize the correlation set. All orders in batch ‘B001’ are handled by one instance of the workflow while all orders in batch ‘B002’ are handled by another workflow instance.

Correlated Send and Receive

This sample shows an order message sent to a normal-type workflow that then publishes it to another topic for further processing. When that processing completes the order is sent back to the existing instance of the workflow that published it. Routing to the original workflow is done by setting the correlation set on the Publish Message Activity to the order ID.