Timeout for REST calls

Quite often, we need a REST API to respond within a specified time frame, say 1500 ms.
Implementing this using the excellent Spring RestTemplate is very simple. A good blog explaining how to use RestTemplate is available here - http://www.baeldung.com/rest-template

   

Spring Sleuth Tips and Tricks

Spring Sleuth coupled with Zipkin in an excellent tool to measure the performance bottlenecks in your distributed microservices architecture.

Spring Sleuth automatically creates HTTP and Messaging interceptors when calls are made to other microservices over REST or RabbitMQ. But what if you want to instrument the time taken to execute a method or time take for a third-party service to respond?

Initially we tried with the @NewSpan annotation on methods, but soon realized that this annotation DOES NOT work in the same class. A detailed discussion thread on this limitation is available here - https://github.com/spring-cloud/spring-cloud-sleuth/issues/617.

The core reason is that the proxies that Spring creates do not work for the method calls in the same object. Sleuth creates a proxy of the object so if we are calling a proxied method from the proxy itself then the method will be called directly without going through the proxy logic, hence no span will be created. Hence we need to manually add the span as follows and it would show up in the beautiful Zipkin UI.

Another great blog post around these techniques is here - http://www.baeldung.com/spring-cloud-sleuth-single-application

Parsing an arbitrary JSON string and extracting values

Developers use libraries such as Jackson to convert JSON strings to Java objects and vice-versa.
But what if you need to parse a JSON string and do not know the complete schema of it. Since you do not have the complete schema of the JSON, you cannot create the Java class.

Another situation could be when you don't want to create a Java class and just want to extract a few values from the JSON string. Given below are 2 techniques for achieving this:

Option 1:

Convert the JSON string to a HashMap. All libraries such as Jackson, Gson provide methods to convert the JSON string into a map. Then by using standard map functions, you can extract the values you are interested in. Sample code available here.

Option 2: 

Convert the JSON string into a 'tree model' object. Jackson has a default tree model object called JsonNode. Jackson also supports JSON Pointer Expression, so you can directly access a node value like this:

Ruminating on the single-threaded model of NodeJS and Node-RED

Many developers and architects have asked me questions on the single-threaded nature of NodeJS and whether we can use it effectively on multi-core machines. Since Node-RED is based on NodeJS, whatever is valid for NodeJS is also valid for Node-RED.

NodeJS has a single thread per process model. When you start NodeJS, it would start a single process with one thread in it. Due to it's non-blocking IO paradigm, it can easily handle multiple client requests concurrently, as there is no thread that is blocking for any IO operation.

Now the next question is around the optimal usage of multi-core machines. If you have 8 core or 16 core machines on your cloud and just run one NodeJS process on it, then you are obviously under-utilizing the resources you have paid for. In order to use all the cores effectively, we have the following options in NodeJS:

1) For compute-intensive tasks: NodeJS can fork out child processes for heavy-duty stuff - e.g. if you are processing images or video files, you can fork out child NodeJS processes from the parent process. Communication between parent and child processes can happen over IPC.

2) For scaling REST APIs: You can start multiple NodeJS processes on a single server - e.g. if you have 8 cores, start 8 NodeJS processes. Put a load balancer (e.g. nginx) in front of these processes. You would anyways have some kind of load-balancing setup in your production environment and the same can be leveraged.

3) Cluster support in newer versions on NodeJS: In the latest versions of NodeJS, you have support for Clusters. Cluster enables us to start multiple NodeJS processes that all share the same server port - e.g. 8 NodeJS processes all listening to port 80.  This is the best OOTB option available today in NodeJS.

Hence it is indeed possible to effectively utilize NodeJS on multi-core machines. A good discussion on the same is available on StackOverflow here.

Another interesting question that is often asked is whether Java developers should jump ship and start development in NodeJS because of its speed and simplicity. Also, NodeJS evangelists keep harping about the fact that the single-threaded nature of Node removes the complexity of multi-threading, etc.

Here are my thoughts on this:

1) Today Java has first class support for non-blocking IO, similar to NodeJ. Take a look at the superb open-source  Netty library that powers all the cloud services at Apple.

2) Most of the complexity of multithreading is abstracted away by popular frameworks such as Spring - e.g. Spring Cloud enables developers to write highly scalable and modular distributed applications that are cloud-native without dealing with the complexities of multi-threading.

3) The Servlet 3.0 specification introduced async requests and the Spring REST MVC framework also supports non-blocking REST services as described here.  

Thus today, all the goodies of NodeJS are available on the Java platform. Also plenty of  Java OSS to kick start your development with all the necessary plumbing code.

Ruminating on a nifty tool called 'ngrok'

Many a times developers want to test their APIs running on their development machine against a mobile app. To do so, developers typically do the following:

Option A) Create a hotspot Wifi on their development machine and connect the mobile phone to this Wifi. Since both the mobile phone and their development machine is on the same network, the mobile app can call the APIs running on the localhost of the developer machine.

Option B) We host the APIs on a public cloud (with a public IP), so that the mobile app can access it. For this, the developer needs to setup a automated CI/CD process that would deploy the REST APIs on the middleware.

There is a third option where-in, the developer can run the API server on his local machine and just create a proxy server on the internet that would securely tunnel the request to this local machine through the firewall. This can be done using ngrok - https://ngrok.com/

ngrok can be setup on your local machine within minutes and can serve as a great debugging tool. It has a network sniffer built in that can be assessed from http://localhost:4040. 

We would highly recommend this tool for any mobile/API developer. 

ngrok can also tunnel via SSH if you don't want to install or run the ngrok agent on the target machine - https://ngrok.com/docs/secure-tunnels/tunnels/ssh-reverse-tunnel-agent/

ngrok also has a nifty tool to setup a quick fileserver for files on your local machine. Just use the following command: ./ngrok http "file:///C:\Users\Docs" --basic-auth="user_id:password" 

Ruminating on non-blocking REST services

In a typical REST service environment, a thread is allocated to each incoming HTTP request for processing. So if you have configured your container to start 50 threads, then you can handle 50 concurrent HTTP requests and any additional HTTP request would be queued till a thread is free.

Today, using principles of non-blocking IO and reactive programming, we can break the tight coupling between a thread and a web request. The Servlet-3 specification also supports async requests as explained in this article - http://www.journaldev.com/2008/async-servlet-feature-of-servlet-3. The core idea is to  delegate the long-running or asynchronous processing to another background thread (Task Executor), so that the HTTP handler threads are not starved.

One might argue that we are just moving the 'blocking thread' bottleneck from the HTTP threads to the backend thread pool (Task Executors). But this does result in better performance, as we can serve more HTTP clients.

The Spring MVC documentation on Async REST MVC is worth a perusal to understand the main concepts - http://docs.spring.io/spring/docs/current/spring-framework-reference/htmlsingle/#mvc-ann-async

A good article that demonstrates how Sprint MVC can be used to build non-blocking REST services that call a backend service exposed via JMS - https://dzone.com/articles/non-blocking-rest-services-with-spring. All the source code for this example is available here. One must understand the basics of Spring Integration framework to work with the above example.

The following discussion thread on StackOverFlow would also give a good idea on how to implement this - http://stackoverflow.com/questions/14134086/request-queue-in-front-of-a-rest-service

The following blog-post shows another example using Spring Boot on Docker - http://spiral-architect.pl/2016/04/25/asynchronous-processing-of-restful-web-services-requests-in-spring-and-java-ee-worlds-wrapped-in-docker-part-1-spring/

Implementing a Request Response with Messaging (JMS)

Quite often, we need to implement a request/response paradigm on JMS - e.g. calling a backend function on a mainframe or a third-party interface that has an AMQP endpoint. To implement this, we have the following 3 options:

Option 1: The client creates a temporary queue and embeds the name-address of this queue in the message header (JMSReplyTo message header) before sending it to the server. The server processes the request message and writes the response message on to the temp queue mentioned in the JMSReplyTo header.
The advantage of this approach is that the server does not need to know the destination of the client response message in advance.
An important point to remember is that a temporary queue is only valid till the connection object is open. Temporary queues are generally light-weight, but it depends on the implementation of the MOM (message-oriented middleware).

 Option 2: Create a permanent queue for response messages from the server. The client sets a correlation ID on the message before it is sent to the server. The client then listens on the response queue using a JMS selector - using the correlation ID header value as the selector property. This ensures that only the appropriate message from the response queue is delivered to the appropriate client.

Option 3: Use a combination of Option 1 and Option 2.  Each client creates a temporary queue and JMS Consumer on startup. We need to use this option if the client does not block for each request. So the client can keep on sending messages and listen to multiple response messages on the temp queue and then match the req/res using the correlation ID.

Spring has implemented this design pattern in the JMS Outbound Gateway class - http://docs.spring.io/spring-integration/reference/html/jms.html#jms-outbound-gateway

The JMS spec also contains an API for basic req/res using temporary queues - QueueRequestor and TopicRequestor. So if your MOM supports JMS, then you can use this basic implementation if it suffices your needs.

The following links would serve as a good read on this topic.
http://www.enterpriseintegrationpatterns.com/patterns/messaging/RequestReplyJmsExample.html
http://activemq.apache.org/how-should-i-implement-request-response-with-jms.html

Orchestrating Microservices using Event Driven Architecture (EDA) paradigms

If you follow the microservices architecture style, you would have a bunch of services running in their own independent process space.

But how do you orchestrate services for a given business workflow? For e.g. a business transaction that spans multiple calls to microservices.
Point to point integrations would result in 'Dependency Hell'. If each microservice calls the other microservice directly over HTTP API calls, then very soon we have a spaghetti of API dependencies.

One simple design pattern to resolve this is by using the EDA (Event Driven Architecture) paradigm. Each microservices does its job and then publishes an event. Other microservices subscribe to the event and act on it as necessary.

This pub/sub model results in loose coupling between the services and makes the system much more maintainable. A good blog-post covering this paradigm in more details is present here. 

Service Discovery Mechanisms in Microservices

In a microservices based architecture, we would not know the number of instances of a server or their IP addresses beforehand. This is because microservices typically run in VMs or Docker containers that are dynamically spawned based on usage load.

So consumers would need some kind of service discovery mechanism to communicate with microservices. There are two options to design this -

a) Server-side Service Discovery - Here the consumers make a request to a load-balancer/service registry and then the request is routed to the actual service end-point. This paradigm is clearly explained on this blog here. Examples of this design pattern is the AWS Elastic Load Balancer.


b) Client-side Service Discovery - Here the consumers use a small library for making service calls. This library makes calls to the service registry and obtains the load-balanced actual service end-point. Netflix uses this approach and its service registry is called Eureka and its client library is called Ribbon.

Handling failures and improving resilience in microservices

In a microservices architecture, one has to build services that can handle failures. For e.g. If a microservice calls another dependent microservice that is down, then we need to handle this using timeouts and implement the Circuit Breaker pattern.

Netflix has open-sourced an incredibly useful library called as Hystrix to solve such problems. Anyone building large scale distributed architectures on the Java platform would find Hystrix a boon. When you make a remote service call through Hystrix libraries, it does the following:

1. If the remote service call does not return within a specified threshold, Hystrix times-out the call.
2. If a service is throwing errors and the number of errors exceed a threshold, then Hystrix would trip the circuit-breaker and all requests would fail-fast for a specified amount of time (recovery period)
3. Hystrix enables developers to implement a fall-back action when a request fails, for e,g returning a default value or a null value or from cache. 

The full operating model of Hystrix is explained in great details on Github wiki 

It was also interesting to learn that the tech guys at Flipkart have taken Hystrix and implemented a service proxy on top of it called 'Phantom'. Looks like the advantage of using Phantom is that your consumers do not have to code against the Hystrix libraries. 

Calling a secure HTTPS webservice from a Java client

Over the last decade, I have seen so many developers struggle with digital certificates when they have to call a secure webservice. A lot of confusion arises when a secure https webservice call is made from a servlet running in Tomcat. This is because the exception stack shows a SSLHandshake exception and then developers keep fiddling with the Tomcat connector configuration as stated here.

But when we make a connection to a secure server, what we need is to trust the digital certificate of the server. If the digital certificate of the server has been signed by a trusted root authority such as 'Verisign', 'eTrust', then our default Java Trust Store would automatically validate it. But if the server has a self-signed certificate, then we have to add the server's digital certificate to the trust store.

There are multiple ways of doing this. A long time ago, I had blogged about one option that entails setting the Java system properties. This can be done through code or by setting the Java properties of the JVM during startup. For e.g.

System.setProperty("javax.net.ssl.trustStore", trustFilename );
System.setProperty("javax.net.ssl.trustStorePassword", "changeit") ;

Different AppServers (WebSphere, Weblogic, etc.) may provide different ways to add certs to the trust store.

Another option is to create a cert-store (filename:jssecacerts) that contains the digital cert of the server and copy that cert-store file to the “$JAVA_HOME\jre\lib\security” folder. There is also a nifty program called InstallCert.java that downloads the certificate and creates the cert-store file. A good tutorial on the same is available here.
I have also created a mirror of InstallCert.java here. This program cam be run without any dependencies on external libraries and I have found it to be very handy.

So what is the difference between setting the TrustStore system property and adding the jssecacerts file?
Well, the documentation of JSSE should help our understanding here. The TrustManager performs the following steps to search for trusted certs:
1.  system property javax.net.ssl.trustStore
2.  $JAVA_HOME/lib/security/jssecacerts
3. $JAVA_HOME/lib/security/cacerts (shipped by default)

It's important to note that is the TrustManager finds the jssecacerts file, then it would not read cacerts file! Hence it may be a better option to add the server digital cert to the cacerts keystore file. To add a certificate to a keystore, there is a nice GUI program called portecle. Alternatively do it from the command prompt using the keytool command as stated here. 

Beautiful REST API documentation

InfoQ has published a list of tools that can be used for creating some cool documentation for our Web APIs.

I was particularly impressed with Swagger and the demo available here.
RAML from MuleSoft was also quite interesting. The list of projects available for RAML makes it a serious candidate for documenting our REST APIs. 

Ruminating on EDI

Despite being very old, EDI (Electronic Data Interchange) is still a dominant force in many industries; especially in healthcare. Organizations have invested so much in EDI, that it's not economically practical to shift to other standards such as XML.

A good primer on EDI is available at:
http://ondisplayapparel.com/whitepapers/What_is_EDI_Whitepaper-Taps.pdf

There are two popular standards for EDI: X12 and EDIFACT. X12 was designed as the standard for EDI transactions in US, and EDIFACT emerged out of X12 for international use. EDIFACT is a global EDI standard supporting multi-country and multi-industry exchange. There are a lot of differences between X12 and EDIFACT, one important difference being the fact that X12 assigns numeric values to documents whereas EDIFACT lists names or abbreviations.

In the EDI world, we have the concept of a VAN (Value Added Network). VANs are service providers that act as messaging hubs between trading partners and handling the EDI communication.

A EDI interchange message is made up of "Segments" and "Elements". Each segment begins with a two- or three-word identifier (ISA, GS, ST, N1, REF) and ends with a delimiter. The elements within each segment are also separated by a different delimiter. A good example for HIPAA X12 Claim transaction is available here. 

WS-Security Username Token Implementation using WCF

The following article on microsoft site is an excellent tutorial for beginners looking to use open standards such as WS-Security to secure their WCF services. Perusal highly recommended.

WS-Security with Windows Communication Foundation

Contracts in REST based services

Traditionally REST based services did not have formal contracts between the service consumer and service provider. There used to be a out-of-band agreement between them on the context of the message being passed.

Also the service provider (e.g. Amazon) would publish some API libraries and sample code across popular languages such as Java. C#.NET, etc. Most developers would easily understand how to use the service by looking at the examples.

Sometime back, there was a debate on InfoQ on the topic of having standards for describing contracts for REST based services. There were interesting differences of opinion on this.

There was a standard defined called WADL that was the equivalent of WSDL for REST based services. Apache CXF supports WADL, but I have not seen many enterprises embracing this. Also WADL supports only XML payloads. What about JSON payloads?

I like the DataContract abstraction in .NET WCF. Using WCF configuration, we can specify where the binding should happen as XML or JSON in a REST service. 

Troubleshooting XML namespaces binding in SOAP request using JAXB

Recently helped a team resolve an issue regarding namespace handling in JAX-WS / CXF. Jotting down the solution as it might help other folks breaking their heads on this issue :)

Our Java application needed to consume a .NET webservice and were facing challenges in SOA interoperability. We created the client stubs using the WSDL2Java tool of CXF. The WSDL had - elementFormDefault="qualified"

The problem we ran across was that the complex types were all being returned as 'null'. Enabling the network sniffer, we checked the raw SOAP reponse reaching the client. The response was OK with all fields populated. Hence the real issue was in the data-binding that was happening to the Java objects.

A quick google search revealed that the default behavior of JAXB when it encounters marshelling/unmarshalling errors is to ingore the exception and put it as a warning messages !!. This was a shock, as there was no way to debug the issue on the server.

We then wrote a sample JAXB application and tried to un-marshall the XML/SOAP message. It is here that we started getting the following error stack:
org.apache.cxf.interceptor.Fault: Unmarshalling Error: unexpected element

Hence, it was proved that the real culprit was the JAXB unmarshaller that was not generating the Java classes with the correct namespace.

First we checked the generated Java classes and found a file called as "package-info.java" that had a XML namespace 'annotated' on the package name - essentially a package annotation. So this was supposed to work, but why is the unmarshaller throwing an exception?

We tried adding the following attributes to the annotation and then the unmarshaller started working !!!
@javax.xml.bind.annotation.XmlSchema (
    namespace = "http://com.mypackage",
    elementFormDefault = javax.xml.bind.annotation.XmlNsForm.QUALIFIED,
    attributeFormDefault = javax.xml.bind.annotation.XmlNsForm.UNQUALIFIED
  ) 

Not sure why the WSDL2Java tool did not add these automatically based on the elementFormDefault="qualified" present in the WSDL.  But this trick worked and we could consume the .NET webservice. We had to modify the build script to replace the "package-info.java" file everytime it was recreated.

Another option is to manually add the "namespace" attribute to all XMLTypes in the generated Java classes; but this is very tedious.

Centralized Clustered ESB vs Distributed ESB

Often folks ask me the difference between traditional EAI (Enterprise Application Integration) products and ESB (Enterprise Service Bus). Traditionally EAI products followed the hub-spoke model and when we look at a lot of topology options of ESB's then one would see that the hub-spoke model is still followed!

Given the fact that almost all EAI product vendors have metamorphized their old products to 'ESB' adds to the confusion.  For e.g. The popular IBM Message Broker product line is being branded as 'Advanced ESB'. Microsoft has released a ESB Guidance Package that depends on the BizTalk platform, etc.

In theory, there is a difference between EAI (hub-n-spoke) architecture and ESB (distributed bus) architecture. In a hub/spoke model, the hub 'broker' becomes the single point of failure as all traffic is routed through it. This drawback can be addressed by clustering brokers for high availability.

In a distributed ESB, there is a network of brokers that colloborate to form a messaging fabric. So in essence, there is a small lightweight ESB engine that runs on every node where a SOA component is deployed. This lightweight ESB engine would do the message transformation and routing.

For e.g. in Apache ServiceMix, you can create a network of message brokers (ActiveMQ). Multiple instances of ServiceMix can be networked together using ActiveMQ brokers. The client application sees it as one logical normalized message router (NMR). But here again, the configuration info (e.g. routing information, service names, etc.) are centralized somewhere.


So then what is the fundamental difference between hub-n-spoke and ESB? IBM did a good job in clearing this confusion, by brining in 2 concepts/terms - "centralization of control config" and "distribution of infrastructure".  A good blog explaining these concepts from IBM is here. Loved the diagram flow on this post. Point made :)

Snippet from IBM site:
"Both ESB and hub-and-spoke solutions centralize control of configuration, such as the routing of service interactions, the naming of services, etc. 
Similarly, both solutions might deploy in a simple centralized infrastructure, or in a more sophisticated, distributed manner. In fact, if the Hub-and-spoke model has these features it is essentially an ESB."

As explained earlier, some opensource ESBs such as Apache Service Mix and Petals ESB are lightweight and have the core esb engine (aka service engine) deployed on each node. These folks call themselves "distributed ESB". Other vendors such as IBM, use the concept of "Federated ESBs" for distributed topologies across ESB domains.

Understanding REST

In one of my old posts, I had elaborated on the differences between REST and simple POX/HTTP.

Recently came across an interesting post by Ryan Tomayko; where-in he trys to explain REST in a simple narrative style. A must read :)    http://tomayko.com/writings/rest-to-my-wife

Another interesting discussion thread on REST is available at StackOverFlow - regarding verbs and error codes.

SOA interoperability - .NET WCF from Java

Recently, one of our teams was struggling to integrate a Java application with a .NET WCF service. The exception that was thrown on the Java side was a SOAPFault as shown below:

SOAPFaultException: The message with Action 'https://tempService:48493/VirtualTechnician/AcceptVehicleTermsOfService' cannot be processed at the receiver, due to a ContractFilter mismatch at the EndpointDispatcher. This may be because of either a contract mismatch (mismatched Actions between sender and receiver) or a binding/security mismatch between the sender and the receiver. Check that sender and receiver have the same contract and the same binding (including security requirements, e.g. Message, Transport, None). 

After a lot of debugging using SOAPUI and WireShark, we found out that the problem was not in the SOAP message, but in the HTTP header. The SOAP Action HTTP header needs to be set in the HTTP Post Request.

On JAX-WS, it can be done with the following code snippet:

BindingProvider bp = (BindingProvider) smDispatch;
            bp.getRequestContext().put(BindingProvider.SESSION_MAINTAIN_PROPERTY, Boolean.TRUE);
            bp.getRequestContext().put(BindingProvider.SOAPACTION_USE_PROPERTY, Boolean.TRUE);
            bp.getRequestContext().put(BindingProvider.SOAPACTION_URI_PROPERTY, "http://tempuri.org/IVtOwl/AcceptVehicleTermsOfService");// Correct SOAP Action

Taxonomy of Services

In one of my previous posts, I had blogged about creating a taxonomy of services using functional categorization.
For e.g. Entity Services, Task/Activity Services, Process Services and Infrastructure services.

But services can also be categorized from different perspectives such as layers or intent of use. For e.g.
Categorization based on Service Layer:

1. Business Services: Represent high level business functions that define an enterprise.
2. Application Services: Application specific and usually will be aggregated in a composite service at the business level.
3. Infrastructure Services: Utility functions that deliver cross cutting functions.

Categorization based on scope:
1. Enterprise Services: Multiple LOBs use the service.
2. Domain Services: Applicable only within a LOB.
3. Application Services: Local to the App Level.