Introducing the new Neuron ESB Connector Metadata Generation Wizard

The Neuron ESB 3.5.3 Cumulative Update (CU) 3 released a host of new features, enhancements and critical bug fixes. For more information on CU3 and all that is going on in the Neuron ESB World, take a moment to read July, 2015 newsletter: https://www.peregrineconnect.com/support/downloads/newsletter/2015/july/.

For all the details check CU 3 release document.

One of the most exciting new features and one that we’re sure you will like is the Connector Metadata Generation Wizard. Neuron ESB ships with a number of connectors including Salesforce, Dynamics CRM and NetSuite to name just few. CU 3 introduces the new SAP Connector. My colleague Joe Klug has written a number of blog posts about the new SAP Connector (https://www.peregrineconnect.com/article/using-neuron-esb-sap-connector-part-1/).

Among other things, connectors make it easier for organizations to connect and integrate with different protocols, transports, databases and line of business systems. Connectors abstract away the communication and connectivity details of the system for you, allowing you to focus on the business solution. Neuron ESB ships with connectors that support dozens of different protocol, transports, databases and line of business systems that organizations commonly use.

For this post, I’m going to concentrate on connecting to an external SAP instance. Just connecting however is not enough. For many line of business systems like Salesforce, Dynamics CRM, NetSuite and SAP, there are many strongly typed entities (like an “Account” or “Contact”) that you can execute a variety of operations against. In order to effectively work with and integrate with these systems, you need to know what these types are, the operations specific to them and how to quickly find and manage them. At runtime, these operations are represented by messages (usually XML) which correspond to their respective type. An Connector Endpoint (a subscribing line of business system) will receive and convert theses message into the line of business specific API (Application Programming Interface) calls, handling all the required proprietary protocol interactions. In this way, connectors allow you to focus on the message and solution rather than the proprietary APIs and Protocols of the various systems you interact with.

Tracking down, managing and creating messages for all the types and operations that line of business systems support can be a time consuming and arduous challenge. With the new Connector Metadata Generation Wizard, Neuron ESB makes this a walk in the park.

Metadata: it’s place in integration and Neuron ESB

Metadata is data that describes other data. In this case, we seek to generate data (metadata) that describes (defines) the messages we intend to send in an operational call to some external system (ex: SAP, Salesforce, Dynamics CRM, NetSuite, etc.).  These definitions are represented as XML Schema Documents (XSD’s).

We may also want to generate sample messages based on the generated XSD’s. These messages are represented as XML Documents and in most cases represent the operation you want to execute against the external system.

One of Neuron ESB’s chief design goals is to make integration and working with external systems simple, but more so, enabling you to make changes to existing solutions even easier. To do this, Neuron ESB was designed to be type agnostic. The direct consequence of this is that at runtime Neuron ESB doesn’t require WSDL or XSD based Schemas to be present before services can be called or messages can be published or subscribed to. There is no tight coupling between the message and any artifact, either external or internal, unless you specifically declare it. For example, if a service contract changes, you only need to change the transformation logic; there are no other dependencies that you have to worry about unwinding, un-deploying and de-referencing first. This also makes it easier for the Neuron ESB runtime to be self-aware to changes to the solution, automatically loading them without the need to restart services or runtimes.

Even though Neuron ESB is type agnostic by design, many line of business systems are not, and hence the connectors for those systems require messages that are strongly typed. Imagine for a moment that you have a generic on ramp pattern deployed using Neuron ESB. Any number of canonical messages could be published to the bus through the on ramp. Those canonical messages would usually be represented by one or more XML Schema documents. Once published, any external system (represented by an Connector Endpoint) could subscribe to those messages. Before being consumed, those messages would need to be converted to the format expected by the connector representing the external system. To make that happen, you need metadata (XSD, XML and XSL files) and the new Connector Metadata Generation Wizard makes it easier than ever to acquire that metadata.

Getting started

While the Connector Metadata Generation Wizard can leverage an existing Connector Endpoint’s connection details, you don’t need to have any Neuron ESB artifacts in place to get started with the wizard. For purposes of this post, an existing Connector Endpoint will be used. If you wish to start the same way, you’ll need at least one registered connector and at least one Party (Publisher or Subscriber) and that in turn means you need at least 1 Topic. For purposes of this post, I’ve created the following:

• Topic: Test Topic
• Subscriber: Test Subscriber
• Connector Endpoint: SAP Endpoint (consumes the registered SAP Connector and the Test Subscriber Subscriber)

Note: While this post focuses on the SAP Connector, you may not have access to an SAP instance. The Connector Metadata Generation Wizard works with the following additional connectors:

• Dynamics CRM 2013/2015
• NetSuite
• Salesforce

Adapt the steps below accordingly.

Running the Connector Metadata Generation Wizard

You can access the wizard from the Connections Connector Endpoints Section of the Neuron ESB Explorer:

The following graphic illustrates how to launch the wizard by clicking the “Generate” toolbar button:

Note: The wizard can also be launched from the Repository section (XML Schema and Document items) and the Connections Connector Registration section.

Once the wizard has launched, click the Next Button.

Selecting an connector and Connector Endpoint to retrieve connection details

The following graphic illustrates where and how to select the connector and the Connector Endpoint connection details you want to use in order to connect to the external system.

Note: The metadata artifacts we will retrieve don’t have a direct relationship to the selected Connector Endpoint. We are selecting an Connector Endpoint so that we can use its connection details to both test a connection and to generate metadata. The following graphic illustrates that relationship:

In cases where you are not starting with an existing Connector Endpoint

In many cases you may not want to use the wizard with an existing Connector Endpoint or, you may not have one configured. In those cases you can manually enter the credentials you need to connect to the system so that you can generate metadata. This may be preferable as most systems will require different levels of authentication for runtime operations versus browsing metadata. Generally, browsing metadata requires elevated permissions which you would not grant for runtime operations.

Once you have tested your connection, click the Next Button

Selecting available operations

Now that you have successfully connected to your system, the next step is to generate metadata for the operations you wish to carry out. The following graphic illustrates how to begin that process:

One look at the system names, it becomes clear how productive this new wizard is. Depending on the system, the operation names can be quite cryptic!

Take note of the two search boxes. The following graphic illustrates how those tools can filter a list:

To filter a list, once the list is populated, you can type the text you are looking for. Immediately, the list contents will reflect those items that match your search criteria.

Once you have selected the operations you wish to add, click the Add Button

You can add multiple operations as needed:

Once you have selected your operations, click the Next Button

Generating the schema (XSD’s) and sample data (XML)

The following graphic illustrates how to specify Neuron ESB data. Here, you can specify:

• Category Name (In Neuron ESB, we can tag artifacts with a category and filter accordingly)
• Whether or not to overwrite items that have been previously imported into the Repository Section in Neuron ESB
• Whether or not to generate sample XML
• Description (The wizard uses defaults as specified by the external system. You can alter this text as required)

Click the Next Button.

The final step summarized what you are about to create. If you need to change something, simply click the Back Button.

Viewing the metadata in Neuron ESB

Once you have completed all of the wizard steps, navigate to the Repository Section in the Neuron ESB Explorer and click the XML Schema item. The following graphic illustrates the new items that can be used in a Neuron ESB Business Process or Workflow:

Using the metadata in a Business Process or Workflow

A Business Process is a packaged set of steps (business logic) that you can apply to a Party (Publisher or Subscriber). For a Publisher, the Business Process will act on the message before the message is set across the bus. For a subscriber, the Business Process will act on the message after it has been received from the bus.  Business Processes are in-memory, graphical representations of business logic in Neuron ESB. They are not meant for long-running operations. For that, Neuron ESB supports long-running fault tolerant Workflows that are expressed as Workflow Endpoints. For more information on these items, please consult the Neuron ESB help documentation.

Now, you can leverage this metadata in a Business Process or a Workflow as part of a Validate XML step. In the graphic below, the Validate – Schema step is used in a process. The Validate – Schema step can leverage one or more XSD’s stored in the Neuron ESB Repository. In this case, we are leveraging the SAP XSD’s imported through the wizard.

No Compilation and tight coupling!

Neuron ESB’s connectors make it easy for you to connect to and integrate with various technologies and systems, relieving you from needing to become an expert in every system and technology you have to work with. The Connector Metadata Generation Wizard makes crafting the messages for line of business systems that much easier, eliminating the guesswork out of the integration. The wizard generated XSD Schemas can also be used as the foundation for XSLT to transform the canonical formats being published to the messages expected by the subscribing line of business systems.

In other environments, the middleware solution is often made up of tightly coupled and compiled components that have to be re compiled and re-deployed as part of a complicated process. With Neuron ESB, all you need to do is distribute the new XSD. There is nothing to recompile and the deployment process couldn’t be simpler!