Database Polling Listener and Watermark Configuration in MuleSoft Anypoint Studio

Many a times, there is a need for retrieving the newly inserted database records for synching or business process purposes.

MuleSoft makes it easy for us to process the records easily using Poll, Database with the Batch processing components.

Let us see how to design the Mule Flow - 

Step - 1 Drag and drop 'Batch' object into Anypoint studio canvas
Step - 2 Drag and drop 'Poll' object into Input section of batch process.
Step - 3 Place business process objects as appropriate under  Batch_Step section.
Step - 4 Place the objects as required in 'On Complete' section.

Now, you can see the template as below - 

Next step is to configure the DB query under Datbase object as below -

In Connector configuration configure the Database and query details as below - 

Please note that we need to update the Required Dependencies with the Database JAR file to enable communication to Database.

Next step is to configure the DB Query as below -

You may note that the query has the WHERE clause based on the criteria such that only the latest updated records are fetched.  So, we need to introduce a mule flow variable accordingly.  It could be based on the ID, date or any other data that suits to select only the latest updated records.

We will now see how MuleSoft helps us automatically update the variable to set the watermark of the data that eliminates the records that are already processed.

Based on the query, MuleSoft automatically udpates the watermark variable and processes the records only when there is at least one satisfying the criteria.

The below is the complete Mule Flow for your reference -

<?xml version="1.0" encoding="UTF-8"?>

<mule xmlns:mulexml="http://www.mulesoft.org/schema/mule/xml" xmlns:batch="http://www.mulesoft.org/schema/mule/batch" xmlns:dw="http://www.mulesoft.org/schema/mule/ee/dw" xmlns:db="http://www.mulesoft.org/schema/mule/db" xmlns:file="http://www.mulesoft.org/schema/mule/file" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
xmlns:spring="http://www.springframework.org/schema/beans" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/file http://www.mulesoft.org/schema/mule/file/current/mule-file.xsd
http://www.mulesoft.org/schema/mule/db http://www.mulesoft.org/schema/mule/db/current/mule-db.xsd
http://www.mulesoft.org/schema/mule/ee/dw http://www.mulesoft.org/schema/mule/ee/dw/current/dw.xsd
http://www.mulesoft.org/schema/mule/batch http://www.mulesoft.org/schema/mule/batch/current/mule-batch.xsd
http://www.mulesoft.org/schema/mule/xml http://www.mulesoft.org/schema/mule/xml/current/mule-xml.xsd">
    <file:connector name="File" outputPattern="Outfile.xml"  autoDelete="true" streaming="true" validateConnections="true" doc:name="File" workFileNamePattern="*.xm" writeToDirectory="C:\FileCopySourceLocation"/>
    <file:connector name="File1"   autoDelete="true" streaming="true" validateConnections="true" doc:name="File" readFromDirectory="C:\FileCopySourceLocation" workFileNamePattern="Myfile.xml"/>
    <db:mysql-config name="MySQL_Configuration" host="localhost" port="3306" user="root" password="root" database="test" doc:name="MySQL Configuration"/>
    <batch:job name="Process_Employee_Records">
        <batch:input>
            <poll doc:name="Poll">
                <fixed-frequency-scheduler frequency="1" timeUnit="MINUTES"/>
                <watermark variable="id" default-expression="0" selector="LAST" selector-expression="#[payload.id]"/>
                <db:select config-ref="MySQL_Configuration" doc:name="Employee Repository">
                    <db:parameterized-query><![CDATA[SELECT ID, NAME, UPDATED_TIME FROM EMPLOYEE WHERE ID > #[flowVars.id]]]></db:parameterized-query>
                </db:select>
            </poll>
        </batch:input>
        <batch:process-records>
            <batch:step name="Batch_Step">
                <logger message="#[payload]" level="INFO" doc:name="Log Retrieval Status"/>
                <mulexml:object-to-xml-transformer doc:name="Convert to XML"/>
                <file:outbound-endpoint path="C:\FileCopySourceLocation" outputPattern="Myfile.xml" connector-ref="File" responseTimeout="10000" doc:name="Persist Payload"/>
            </batch:step>
        </batch:process-records>
        <batch:on-complete>
            <logger message="Completed Successfully" level="INFO" doc:name="Record Completion"/>
        </batch:on-complete>
    </batch:job>
</mule>

Hope you enjoyed reading this blog.  For any queries, please feel free to write to me @ sivakumar.th@gmail.com.

How to assign multiple 'Soap Action' names to same 'Operation' in WSDL?

Recently we had a situation in a web service migration, we faced a situation where we need to route to the same operation in a proxy service from requests with the different soap action names.

This is particularly useful when designing the web service invocation to different web service clients.

Let us see how the WSDL is designed:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>

<wsdl:definitions xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/" xmlns:tns="http://www.example.org/EmployeeService/" xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" name="EmployeeService" targetNamespace="http://www.example.org/EmployeeService/">

  <wsdl:types>

    <xsd:schema targetNamespace="http://www.example.org/EmployeeService/">

      <xsd:element name="EmployeeInput">

        <xsd:complexType>

          <xsd:sequence>

            <xsd:element name="firstName" type="xsd:string"/>

            <xsd:element name="lastName" type="xsd:string"/>

            <xsd:element name="address" type="xsd:string"/>

          </xsd:sequence>

        </xsd:complexType>

      </xsd:element>

      <xsd:element name="EmployeeOutput">

        <xsd:complexType>

          <xsd:sequence>

            <xsd:element name="fullName" type="xsd:string"/>

            <xsd:element name="address" type="xsd:string"/>

          </xsd:sequence>

        </xsd:complexType>

      </xsd:element>

    </xsd:schema>

  </wsdl:types>

  <wsdl:message name="EmployeeRequest">

    <wsdl:part element="tns:EmployeeInput" name="parameters"/>

  </wsdl:message>

  <wsdl:message name="EmployeeResponse">

    <wsdl:part element="tns:EmployeeOutput" name="parameters"/>

  </wsdl:message>

  <wsdl:portType name="EmployeeService">

    <wsdl:operation name="GetEmployeeDetails">

      <wsdl:input message="tns:EmployeeRequest"/>

      <wsdl:output message="tns:EmployeeResponse"/>

    </wsdl:operation>

  </wsdl:portType>

  <wsdl:binding name="EmployeeServiceBinding" type="tns:EmployeeService">

    <soap:binding style="document" transport="http://schemas.xmlsoap.org/soap/http"/>

    <wsdl:operation name="GetEmployeeDetails">

      <soap:operation soapAction="GetEmployeeDetailsActionOne"/>

      <wsdl:input>

        <soap:body use="literal"/>

      </wsdl:input>

      <wsdl:output>

        <soap:body use="literal"/>

      </wsdl:output>

    </wsdl:operation>

    <wsdl:operation name="GetEmployeeDetails">

      <soap:operation soapAction="GetEmployeeDetailsActionTwo"/>

      <wsdl:input>

        <soap:body use="literal"/>

      </wsdl:input>

      <wsdl:output>

        <soap:body use="literal"/>

      </wsdl:output>

    </wsdl:operation>

  </wsdl:binding>

  <wsdl:service name="EmployeeService">

    <wsdl:port binding="tns:EmployeeServiceBinding" name="EmployeeServiceBinding">

      <soap:address location="https://localhost:8243/services/EmployeeService.EmployeeServiceHttpsEndpoint"/>

    </wsdl:port>

  </wsdl:service>

</wsdl:definitions>

From SoapUI, you can see the same operation name pointing to different SoapAction names.

Exception Management in SOA – Loosely Coupled vsTightly Coupled

The Background:

The convention dictates that the exception handling in a typical OOPS should be centralised and all the errors would need to be converted or translated to human readable form.  This is often true because of the fact that the user would get the opportunity to correct the data and hence recover from the error scenario. This requires the developers to catch all the possible exceptions, convert the technical error into human readable form and send across different layers cohesively until the user sees in the view layer.

However, this topic will let us analyse in detail the possible error handling principles in the SOA world and will leave the decision and the best practice to the choice of the readers themselves.

The Importance of Loose Coupling:

The services developed in the SOA world are aimed to be atomic to perform a single task of the underlying business and hence promoting the reusability.  At the same time, we need to remember that these services could be accessed or triggered by one off the below ways:

•  By a batch process  (In a regular interval)
• By another web-service (Used by another application Soap or REST based)
• By an orchestration service (A service that calls 2 or more services in a definite sequence)
• By a Business-Event (A special criteria formed based on 2 or more data)
• By a Rule-Engine (A special service that takes series of decision based on service calls)
• By a Messaging System (A service call based on the incoming event/message placed in the Queue)

This is the key in having the service design with the loosely coupled architecture.  Look at the Actors involved in the above services invocation; it could either be a system or user or event.  Do we expect these actors to correct the error and call the service back?  No, it is not the best that the system corrects every error by altering the original message and re-issue or re-invoke the same service to get it processed successfully.  But the major focus in SOA is the service enablement such that it can be used by any services, any technology, any number of times.

How do we achieve these loose-coupling and yet have the best exception handling strategy. Let us analyse our options.

Introduce Consistency in Error Handling

Considering the services being invoked from any boundary (across and outside the Enterprise) and triggered by any actor (system or event), it is important to maintain the consistency in indicating the errors.

The consistency can be achieved by the below key principles:

• Use of Common Error Codes for all services
• Optionally provide the error descriptions
•  Uniform Message Structure (Canonical)

Consider the below sample of such a ‘Fault’ Message:

                <error>

                   <code>10000</code>

                   <type>Network Communication</type>

                   </description>Service or Network Access Error</description>

          </error>

Centralize the Error Handling

The centralization in SOA would mean to place the error identification logic within the individual services but decision making around the error using the isolated and centralized logic via a dedicated service or set of services.  This is important so that the exception handling code is not duplicated inside different services in a redundant way.  Where there is a duplication of logic, more the possibility for overhead of maintenance, defect fixing, testing and in turn more errors.

Connect the Business to Error Handling:

It is important for every enterprise organization to analyse the errors occurred in the specific period of time for the below reasons. 

• To identify the pattern of errors occurred.
• To identify the frequency of errors occurred.
• To identify the services causing the errors.

The above listed paradigm helps the organization to evolve SOA as a continuous improvement of the different systems that in turn evolved over a period of time.  This also helps to build a strategy around the results, find the root cause and thereby taking effective decision and cost-reduction by reducing the maintenance. 

Build Effective Error Handling Strategy

The error handling needs to have a definite strategy, reasons for every action behind it. For example, the error handling needs to be built considering the below basic error handling principles or policies:

• Support for retrying the failed messages that are valid.
• Support for Notification and Alerts.
• Deactivating the Service events during failure of dependent services.
• Persisting/Storing the Failed Messages for future Manual Correction.
• Logging the Error and the Related Events.

Build High Performance Architecture

The power of SOA is to be able to scale the performance if needed.  The reason for centralizing the error handling and making it consistent is to make the system not getting affected due to number of errors.

The error handling strategy should focus on the performance and not support for the user to correct the error. The asynchronous messaging pattern can be introduced, optionally, so that these errors are handled without affecting the actual service performance.   The asynchronous way of error handling is building the services such that they are processed offline without making the calling services wait for the error handling logic to be completed.

Build Monitoring Process around Error Handling Strategy

Now that we have the strategy in place to gather and process all the details about the errors occurred.  Also it is now important to build a monitoring process to gain the business benefits out of it.

For example, the notification service and event service can be effectively utilized to notify the loss of significant business due to communication error or other unexpected event.  For example, the loss of sales above 5000$ can be notified to the CxO officials when such combination occurs or when the business event occurs worth 10000$ which resulted in error due to unexpected error such as network outage or password expiry. Building such a design to recover the selected errors would often result in favour of the business which otherwise is possible with the dedicated infrastructures such as BHM or Complex-Event processing.

Conclusion

The purpose of this blog is to introduce these frequently followed principles of error handling in SOA. However, the choice of the combination of the suggested strategies depends upon different factors involved in the enterprise such as available SOA infrastructure, cost and time, roadmap and the development plan.

So, the success lies in the right choice of these design aspects at the right time for the enterprise to be able to scale in future.

Constructing 'Contract-Last' web-service using MuleSoft.

In the previous blog, we discussed about what contract is and how to build contract first service using MuleSoft.

In this blog, we will see about the method of constructing contract-last web-service.

What is 'Contract-Last' Scenario?

When 2 systems are to be integrated and both systems are with their components already built in are just completed it.  They were earlier used for internal business process and the same needs to be exposed as a web-service for the external vendor needs to access it. In these scenarios, we will go with this approach.

MuleSoft uses Spring and CXF frameworks internally, to create stubs and publish the web services without any manual efforts in Mule Anypoint studio. They are created for you automatically.

The below steps are needed to create a web-service in Mule Anypoint Studio:

1. Develop 2 POJO Java classes one for request format and another one for response.
2. Develop a POJO Java class for web-service interface with @WebService annotation.
2. Configure CXF object with the necessary parameters.
3. Access the WSDL which is internally constructed automatically by Mule.
4. Optionally, add any additional functionalities to be implemented.

Step-1: Develop POJO Classes:

We will take a simple use case for demo purpose.  The input will be the name of the user and the response will be to add Greeting text 'Hallo ' as the response.

Let us create a simple Java POJO classes for both request & response as below -

package wsdlfirst;

public class Employee {

String name;

public String getName() {
return name;
}

public void setName(String name) {
this.name = name;
}

}

Now, we need to create an interface for an actual web-service stub:

package wsdlfirst;

import javax.jws.WebService;

@WebService
public interface IService {

public Employee greetEmployee(Employee name);


}

Note the annotation @WebService which internally helps to create the necessary artifacts with the help of Spring framework.

We now need to implement the web-service interface 'IService' as below:

package wsdlfirst;

public class GreetService implements IService{

@Override
public Employee greetEmployee(Employee name)
{
name.setName("Hallo " + name.getName());
return name;
}

}

All set.  Now we will get into Mule flow as below:

    <http:listener-config name="HTTP_Listener_Configuration" host="0.0.0.0" port="8081" basePath="service" doc:name="HTTP Listener Configuration"/>

    <spring:beans>
        <spring:bean id="greetService" class="wsdlfirst.GreetService" scope="singleton"/>
    </spring:beans>

    <flow name="webserviceFlow">
        <http:listener config-ref="HTTP_Listener_Configuration" path="/greetUser" doc:name="HTTP"/>
        <cxf:jaxws-service doc:name="AddressService" serviceClass="wsdlfirst.IService"/>
            <component  doc:name="Java">
                <spring-object bean="greetService" />
            </component>
    </flow>

The first part is used to create a simple java object greetService.  The key part is to create a web-service CXF component with the webservice interface.

Take a look at the follow up with the component tag where we mention the object to be used for web service response.

The Mule Flow looks as below -

We can execute the SOAP service using Soap UI as below:

Hope you enjoyed going through this blow.  Should you have any questions, please feel free to reach out to me @ sivakumar.th@gmail.com 

Constructing 'Contract First' web-Service using MuleSoft

Before we see MuleSoft specific method of implementation, let us investigate what a web-service is.

Web-Site vs Web-Service:

A web-site is a HTTP transport based communication between a client (technically called browser used by human beings) and the server components.  The server components can be of different technology platforms such as .NET, Java or PHP based ones. 

A web-service is a HTTP transport based communication between 2 different application systems where manual intervention is completely avoided to bring automation.  These are aimed to have 2 different systems exchange data automatically.

What is Contract-first web-service?

A 'contract' is an agreement or format between 2 systems that intends to exchange data. In order to avoid assumptions and later confusions, the schema is identified before hand (before 2 systems start to implement their server side components).  A WSDL is a meta-data that comprises of schema, operation and address.

A contract-first approach of designing a web-service is very helpful if both publisher and consumer applications do not have any components developed yet. With the manual construction of WSDL, one can agree the structure of data and different operations involved in the data exchange between them before hand. 

Constructing Contract-First web-service using MuleSoft:

Step-1 Create a new project in Anypoint Studio with HTTP inbound:

Create a new Mule Project and introduce a inbound HTTP service by selecting HTTP component into a design canvas section.

Step-2 Import WSDL:

Choose the project, right click and select 'Import' option to get the 'Select' dialog window and select 'File System' and then choose 'Next'. Once you have the next window, choose the WSDL file and select the folder where the WSDL file to be imported.  Probably create a folder 'wsdl' under src/main/resources folder to get the WSDL imported.

Step-3 Create CXF object for web-service:

The basic inbound HTTP service will need to be wrapped as a CXF in order to publish the WSDL and bind it to the associated Java class. 

Now, drag and drop 'CXF' object into the canvas.

Step-4:  Associate the Java Classes into web-service:

Select the 'CXF' object to see the properties window and choose the operation 'JAX-WS service' and click 'Generate from WSDL' against the 'service class' option.  Now, in the dialog window, choose the wsdl that is earlier imported and mention the package name where the stub classes are to be created.

Step-5: Now add business logic in Mule Flow:

We are all set to go.  Now following the CXF introduce any business logic to process the incoming data in the format specified in WSDL.  

Once these steps are completed, you may use any tools like SoapUI to test the functionality.

In the next blog, we will see 'Contract-Last' webservice and way to implement it using MuleSoft.

How to manage 'Transaction Events' in Middleware Integration?

Design Techniques behind 'Audit Trail' in Integration Services :

When the two heterogeneous applications are integrated in an enterprise, it is very important to log the details for future reference.  

This is because, collecting the data before managing it for analytics is the important piece of design.  These data if made structured to the maximum would give competitive edge to the retailer for future BI analytics and decision making process.

Is it just logging of information or there is more?

Let us step a bit farther to see what the CXOs want when they rapidly expanding their business when they are in the growth path. The below listed analytics are critical for decision making.

1. Current Business Trend.
2. Different dimensions of Business Data.
3. Growth dimensions
4. Effective Customer Support
5. Avenues for business expansion

Well, how are these related to a very basic logging which is known as audit trail?  Lets see.

During the system integrations, millions of elementary data are flowing between different systems which is of huge help to construct the high level business analytic details.  The integration solution would need to address the above by an effective logging of critical events.

What are the basic and key information that forms an effective logging?

1. Transaction Time and Date.
2. Last updated Time and Date.
3. The system and person involved in the particular transaction.
4. Status of the transactions.
5. Messages associated to transactions.
6. Critical details about the transaction entity.
7. Information about the purpose of the transaction.

The audit trail can contain any additional piece of information that would be useful for future reference.  However, it is better to separate them into a secondary location and have them referenced from the above listed key information.  In a nut-shell, the metadata details about every message needs to be collected and consolidated even before planning the BI and Big data needs.

Rather than aiming to address the immediate integration needs, the system has to be designed to log the events and the associated details irrespective of the future analytics and business needs.  

This is because, the data about transactions are very valuable and will be very expensive if missed out (it cannot be retrieved later if not done from the very beginning) and hence it is essential to consolidate them from the beginning and decide how these can be used based on the business expansion needs.

When these data are made available and consolidated, it can serve as the input for its future business analytics needs and hence the business can make strategical and tactical adjustments.

Realizing the visions of the 'Retail Enterprise' by effective Integration Architecture

How should the 'Retail Enterprise' envision their 'System Integration' Architecture?

During the retail integration for a medium to big enterprise, the integration architects need to focus on the architecture that would not just help developing the web services to connect 2 systems, but rather construct the solution that would help grow their business and achieve the future roadmap for significant period of time ranging from 5 to 10 years. 

The enterprise integration architects need to focus the below aspects -

1.  Identify the standards in schema for the interfaces. 
2.  Identify the SLAs and QoS (Service Level Agreement & Quality of Service)
3.  Architecture to audit enough information about the errors.
4.  Designing the integration system to persist the payload that fails.
5.  Interfaces with a capability to identify the invalid payload vs failed payload.
6.  Recover from the error when the communication is restored.
7.  Ability to sustain for near and future Roadmap of the enterprise.
8.  Handle large and large-volume of messages.
9.  Capability to scale for more volume, velocity and size of data.
10.  Capability to interface more systems with less number of web-services.

In order to address the above features, there are various conceptual ways in which the interfaces can be implemented considering the visionary aspects of the business and related RoI (Return of Investment).  The architects have to spare some time to conduct the enterprise assessment and capability planning before they design an exhaustive interfaces.

The below are the critical aspects to decide on the 'Enterprise Integration':

• Enterprise's affordability in terms of Infrastructure, People and Process.
• Enterprise's business objectives and Roadmap.
• Future business expansion plans and timelines.

The above 3 points are important, simply because if they are not fulfilled, the money spent might not realize the vision and objectives aimed.

In the following blogs, I will be writing about steps to gradually achieve the effective integration that suits retailers and other domains. If you have any queries, please feel free to write to me @ sivakumar.th@gmail.com