Ruminating on non-blocking REST services

In a typical REST service environment, a thread is allocated to each incoming HTTP request for processing. So if you have configured your container to start 50 threads, then you can handle 50 concurrent HTTP requests and any additional HTTP request would be queued till a thread is free.

Today, using principles of non-blocking IO and reactive programming, we can break the tight coupling between a thread and a web request. The Servlet-3 specification also supports async requests as explained in this article - http://www.journaldev.com/2008/async-servlet-feature-of-servlet-3. The core idea is to  delegate the long-running or asynchronous processing to another background thread (Task Executor), so that the HTTP handler threads are not starved.

One might argue that we are just moving the 'blocking thread' bottleneck from the HTTP threads to the backend thread pool (Task Executors). But this does result in better performance, as we can serve more HTTP clients.

The Spring MVC documentation on Async REST MVC is worth a perusal to understand the main concepts - http://docs.spring.io/spring/docs/current/spring-framework-reference/htmlsingle/#mvc-ann-async

A good article that demonstrates how Sprint MVC can be used to build non-blocking REST services that call a backend service exposed via JMS - https://dzone.com/articles/non-blocking-rest-services-with-spring. All the source code for this example is available here. One must understand the basics of Spring Integration framework to work with the above example.

The following discussion thread on StackOverFlow would also give a good idea on how to implement this - http://stackoverflow.com/questions/14134086/request-queue-in-front-of-a-rest-service

The following blog-post shows another example using Spring Boot on Docker - http://spiral-architect.pl/2016/04/25/asynchronous-processing-of-restful-web-services-requests-in-spring-and-java-ee-worlds-wrapped-in-docker-part-1-spring/

Concurrency and scaling strategies for MDPs and MDBs

Message Driven Beans offer a lot of advantages over a standalone JMS consumer as listed in this blog-post. The Spring framework provides us with another lightweight alternative called MDP (Message Driven POJOs) that has all the goodies of MDB, but without any heavy JEE server baggage.

The strategy for implementing a scalable and load-balanced message consumption solution is a bit different in MDBs vs. MDPs.

An MDB is managed by the JEE container and is 'thread-safe' by default. The JEE container maintains a pool of MDBs and allows only one thread to execute an MDB at one time. Thus if you configure your JEE container to spawn 10 MDBs, you can have 10 JMS consumers processing messages concurrently.

Spring MDPs are typically managed by the DMLC (DefaultMessageListenerContainer). Each MDP is typically a singleton, but can have multiple threads running through it. Hence an MDP is NOT 'thread-safe' by default and we have to make sure that our MDPs are stateless - e.g. do not have instance variables, etc. The DMLC can be configured for min/max concurrent consumers to dynamically scale the number of consumers. All connection, session and consumer objects are cached by Spring to improve performance. Jotting down some important stuff to remember regarding DMLC from the Spring Java Docs.

"On startup, DMLC obtains a fixed number of JMS Sessions to invoke the listener, and optionally allows for dynamic adaptation at runtime (up to a maximum number). Actual MessageListener execution happens in asynchronous work units which are created through Spring's TaskExecutor abstraction. By default, the specified number of invoker tasks will be created on startup, according to the "concurrentConsumers" setting."

It is also possible in Spring to create a pool of MDPs and give one to each TaskExecutor, but we have never tested this and never felt the need for this. Making your MDPs stateless and hence 'thread-safe' would suffice for almost all business needs. Nevertheless, if you want to create a pool of MDP objects, then this link would help - http://forum.spring.io/forum/spring-projects/integration/jms/25794-mdb-vs-mdp-concurrency

Another good article on Spring MDP scaling that is worth perusing is here - http://bsnyderblog.blogspot.in/2010/05/tuning-jms-message-consumption-in.html

Performance benchmarks

Every development project needs a formal performance engineering process - one that emphasizes on early performance testing and benchmarking.

For performance benchmarks, it is recommended to do a shallow and wide implementation of a few critical use-cases and then run the load tests against the target hardware. These test results would help in some basic capacity planning.

But what if you have to do some initial rough capacity planning to allocate budgets and do not have the time to do a formal benchmarking exercise. It is here that standard performance benchmarks help. These standard performance benchmarks take a sample transactional use-case (e.g. Order Processing System) and run this workload on various platforms to gather statistics. There are 2 standards that are quite popular -

1. TPC (Transaction Processing Performance Council) - (TPC) is a non-profit organization founded to define transaction processing and database benchmarks and to disseminate objective, verifiable TPC performance data to the industry. TPC-C is the benchmark for OLTP workloads. 


2. SPECjEnterprise2010 - SPECjEnterprise2010 is an industry-standard benchmark designed to measure the performance of application servers conforming to the Java EE 5.0 or later specifications.

Interesting results of the performance benchmarks on various hardware can be found here:
http://www.tpc.org/tpcc/results/tpcc_perf_results.asp
http://www.spec.org/jEnterprise2010/results/jEnterprise2010.html

For the past few years, the Java Day Trader application and its .NET equivalent StockTrader application have been used by vendors to compare the performance of Java vs .NET on their respective platforms. Jotting down some links that point to some interesting debatable data :)

http://www.ibm.com/developerworks/opensource/library/os-perfbenchmk/index.html

http://blogs.msdn.com/b/wenlong/archive/2007/08/10/trade-benchmark-net-3-0-vs-ibm-websphere-6-1.aspx

http://msdn.microsoft.com/en-us/netframework/bb499684.aspx

https://cwiki.apache.org/GMOxDOC22/daytrader-a-more-complex-application.html

'volatile' keywork in Java

Found this excellent article on the web explaining the 'volatile' keyword in Java and how it can be used for concurrency. The tutorial also explains the changes to the volatile keyword functioning in Java 5.

Also found it interesting to understand what 'livelock' is? We often encounter dead-lock and thread starvation in parallel programming, but livelock is also possible :)

Difference between Concurrent Collections and Synchronized Collections in JDK

Traditionally, we have also used object locks (semaphores) and synchronized methods to make our collections thread-safe. But having an exclusive lock on an object brings in scalability issues.

Hence the latest versions of JDK have a new package called "java.util.concurrent". This package contains many new collections objects that are thread-safe, but not so because of synchronization :)

More details at this link: http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/package-summary.html

Snippet from the above link:

The "Concurrent" prefix used with some classes in this package is a shorthand indicating several differences from similar "synchronized" classes. For example java.util.Hashtable and Collections.synchronizedMap(new HashMap()) are synchronized. 

But ConcurrentHashMap is "concurrent". A concurrent collection is thread-safe, but not governed by a single exclusion lock. In the particular case of ConcurrentHashMap, it safely permits any number of concurrent reads as well as a tunable number of concurrent writes.

 "Synchronized" classes can be useful when you need to prevent all access to a collection via a single lock, at the expense of poorer scalability. In other cases in which multiple threads are expected to access a common collection, "concurrent" versions are normally preferable. And unsynchronized collections are preferable when either collections are unshared, or are accessible only when holding other locks. 

Most concurrent Collection implementations (including most Queues) also differ from the usual java.util conventions in that their Iterators provide weakly consistent rather than fast-fail traversal. A weakly consistent iterator is thread-safe, but does not necessarily freeze the collection while iterating, so it may (or may not) reflect any updates since the iterator was created. 

Also a good post on Concurrency basics is available at: http://docs.oracle.com/javase/tutorial/essential/concurrency/memconsist.html (All chapters a must read :)

Another good blog that explains how ConcurrentHashMap maintains several  locks instead of one single mutex to deliver better performance.