Ruminating on Observability

It is more critical than ever in today's complex and dispersed IT settings to have a complete grasp of how your systems are performing. This is where the concept of observability comes into play. The capacity to comprehend the condition of a system by gathering and analysing data from various sources is referred to as observability.

Observabilty has three critical pillars: 

• Distributed Logging (using ELK, Splunk)
• Metrics (performance instrumentation in code)
• Tracing (E2E visibility across the tech stack)

Distributed Logging: Logs keep track of events that happen in a system. They may be used to discover problems, performance bottlenecks, and the flow of traffic through a system. In a modern scalable distributed architecture, we need logging frameworks that support collection and ingestion of logs across the complete tech stack. Platforms such as Splunk and ELK (Elastic, Logstash, Kibana) support this and are popular frameworks for distributed logging. 

Metrics (performance instrumentation in code): Metrics are numerical measures of a system's status. They may be used to monitor CPU use, memory consumption, and request latency, among other things. Some of the most popular frameworks for metrics are Micrometer , Prometheus and DropWizard Metrics. 

Tracing (E2E visibility across the tech stack): Traces are a record of a request's route through a system. They may be utilised to determine the core cause of performance issues and to comprehend how various system components interact with one another. A unique Trace-ID is used to corelate the request across all the components of the tech stack. 

Platforms such as Dynatrace, AppDynamics and DataDog provide comprehensive features to implement all aspects of Observability. 

The three observability pillars operate together to offer a complete picture of a system's behaviour. By collecting and analysing data from all three sources, you can acquire a thorough picture of how your systems operate and discover possible issues before they affect your consumers.

There are a number of benefits to implementing the three pillars of observability. These benefits include:

• The ability to identify and troubleshoot problems faster
• The ability to improve performance and reliability
• The ability to make better decisions about system design and architecture

If you want to increase the observability of your systems, I recommend that you study more about the three pillars of observability and the many techniques to apply them. You can take your IT operations to the next level if you have a thorough grasp of observability.

Ruminating on Memory Mapped Files

 Today, both Java and .NET have full support for memory mapped files. Memory mapped files are not only very fast for read/write operations, but they also serve another important function --- they can be shared between different processes. This enables us to use memory mapped files as shared data store for IPC. 

The following excellent articles by Microsoft and MathWorks explains how to use memory mapped files. 

https://docs.microsoft.com/en-us/dotnet/standard/io/memory-mapped-files

https://www.mathworks.com/help/matlab/import_export/overview-of-memory-mapping.html

Snippets from the above articles:

A memory-mapped file contains the contents of a file in virtual memory. This mapping between a file and memory space enables an application, including multiple processes, to modify the file by reading and writing directly to the memory. 

Memory-mapped files can be shared across multiple processes. Processes can map to the same memory-mapped file by using a common name that is assigned by the process that created the file.

Accessing files via memory map is faster than using I/O functions such as fread and fwrite. Data is read and written using the virtual memory capabilities that are built in to the operating system rather than having to allocate, copy into, and then deallocate data buffers owned by the process. 

Memory-mapping works best with binary files, and in the following scenarios:

• For large files that you want to access randomly one or more times
• For small files that you want to read into memory once and access frequently
• For data that you want to share between applications
• When you want to work with data in a file as if it were an array

Due to limits set by the operating system, the maximum amount of data you can map with a single instance of a memory map is 2 gigabytes on 32-bit systems, and 256 terabytes on 64-bit systems.

Ruminating on Thread Pool sizes and names

Recently we were analyzing the thread-dumps of some JVMs in production and found a large number of threads created in certain thread-pools. Since the name of the thread-pool was generic (e.g. thread-pool-3, etc.) it was very difficult to diagnose the code that spawned the thread-pool.

The following code snippets should help developers in properly naming their thread-pools and also limiting the number of threads. In a cloud environment, the number of default threads in a pool will vary based on the CPU's available - for e.g. in the absence of a thread-pool size, the thread pool may grow to hundreds of threads.

We had seen this happen for the RabbitMQ java client. The default RabbitMQ java client uses a thread pool for callback messages (ACK) and the size of this thread-pool depends on the number of CPUs. Since the JVM is not aware of the docker config, all the processors on the host machine are considered and a large thread pool is created.

More details available here - https://github.com/logstash-plugins/logstash-input-rabbitmq/issues/93

Object pooling made simple using Apache Commons Pool2

If you are looking for a quick implementation of an object pool, then look no further than the excellent Apache Commons Pool2 implementation. Pool2 is far better and faster than the original commons pool library.

Object pool can be used to cache those objects that are expensive to setup and cannot be created for every request or thread - e.g. DB connections, MQTT broker connections, AMQP broker connections, etc. 

The following code snippets would show you how simple it is to create a pool for your 'expensive-to-create' objects. 

Any object pool typically requires 2 parameters  [GenericObjectPool.java] --- 

1) A factory object to handle creation and destruction of your objects [MyObjectFactory.java]

2) A configuration object to configure your pool. [GenericObjectPoolConfig,java]

Spring Sleuth Tips and Tricks

Spring Sleuth coupled with Zipkin in an excellent tool to measure the performance bottlenecks in your distributed microservices architecture.

Spring Sleuth automatically creates HTTP and Messaging interceptors when calls are made to other microservices over REST or RabbitMQ. But what if you want to instrument the time taken to execute a method or time take for a third-party service to respond?

Initially we tried with the @NewSpan annotation on methods, but soon realized that this annotation DOES NOT work in the same class. A detailed discussion thread on this limitation is available here - https://github.com/spring-cloud/spring-cloud-sleuth/issues/617.

The core reason is that the proxies that Spring creates do not work for the method calls in the same object. Sleuth creates a proxy of the object so if we are calling a proxied method from the proxy itself then the method will be called directly without going through the proxy logic, hence no span will be created. Hence we need to manually add the span as follows and it would show up in the beautiful Zipkin UI.

Another great blog post around these techniques is here - http://www.baeldung.com/spring-cloud-sleuth-single-application

Performance benefits of HTTP/2

The HTTP/2 protocol brings in a lot of benefits in terms of performance for modern web applications. To understand what benefits HTTP/2 brings, it is important to understand the limitations of HTTP/1.1  protocol.

The HTTP protocol only allows one 'outstanding' request per TCP connection - i.e. if a page is downloading 100 assets, only one request can be made per TCP connection. Browsers typically open 4-8 TCP connections per page to load the page faster (parallel requests), but this results in network congestion.

The HTTP/2 protocol is fully multiplexed - i.e. it allows multiple parallel requests/responses on the same TCP connection. It also compresses HTTP headers (reduce payload size) and is a binary protocol (not textual). The protocol also allows servers to proactively push responses directly to the browsers cache, thus avoiding HTTP requests.

The following links give an excellent explanation of the HTTP/2 protocol and its benefits.
https://http2.github.io/faq/
https://bagder.gitbooks.io/http2-explained/en/

The following site shows the performance of HTTP/2 compared to HTTP/1.1 when loading tons of images from the server - https://www.tunetheweb.com/performance-test-360/

There is another site - https://www.httpvshttps.com/ which states that HTTPS is faster than HTTP, but that is because HTTPS was using HTTP/2 by default in the background.

Load Testing Tools for IoT platforms over MQTT

While IoT platforms can be tested using traditional load testing tools if standard protocols such as HTTP are used, there are a suite of other tools that can be used if you need to test the MQTT throughput capacity of your IoT platforms.

Jotting down a list of tools that can be used to test the MQTT broker of IoT platforms.

• MQTT-Malaria 
• JMeter - MQTT Plugin
• MQTT-LoadTest
• Gatling-MQTT Plugin
• MQTT Bench

Monitoring TOMEE using VisualVM

A few years back, we moved to Jboss from Tomcat for our production servers, because there was no viable enterprise support for Tomcat.

Today, we have viable options such as support from Tomitribe.

The below article on Tomitribe gives a good overview of setting up VisualVM for monitoring Tomcat.

http://www.tomitribe.com/blog/2014/07/monitoring-an-apache-tomee-service-on-windows-with-visualvm/

Default tools in the JDK

Found the below article worth a perusal. We get so used to using sophisticated tools that we forget there are things we can do with a bare JDK :)

http://zeroturnaround.com/rebellabs/the-6-built-in-jdk-tools-the-average-developer-should-learn-to-use-more/

Ruminating on Rate Limiting

As architects, when we define the API strategy for any organization, we also need to design the 'Rate Limiting' features for that API. The concept of Rate Limiting is not new and the term has been used in networking world for long to represent the control of rate of traffic over the internet.

Other common examples of Rate Limiting that we see very often are as follows:

1. Limit consecutive wrong password entries to 3.
2. Maximum size of an email attachment.
3. Max number of emails one can send in a day.
4. Max number of search queries one can fire every minute.
5. Max. broadband download size per day, etc. 

Rate Limiting is also an important line of defense from a security perspective. Jeff Atwood has a good blog post on 'Rate Limiting' available at: http://blog.codinghorror.com/rate-limiting-and-velocity-checking/

For services or APIs, there are standard ways in which we can rate limit the requests. For e.g.

• Based on API key: This is how Twitter rate limits their API. Each account with a API key can only make x requests/{time period}. For e.g. 10 requests every 5 mins, 500 requests per day, etc.
• Based on IP address: This may not work behind a proxy due to NATing.

Ruminating on Netty Performance

Recently we had again used Netty for building a Event Server from grounds-up and we were amazed at the performance of this amazing library. The secret sauce of Netty is ofcourse the implementation of the Reactor Pattern using Java NIO. More information about the Reactive design paradigm can be found @ http://www.reactivemanifesto.org/

The following links bear testimony to the extreme performance of Netty.

http://netty.io/testimonials

http://www.infoq.com/news/2013/11/netty4-twitter (5x performance improvement)

http://yahooeng.tumblr.com/post/64758709722/making-storm-fly-with-netty (Netty used in Yahoo Storm)

Ruminating on Bloom Filters

Of late, I have been trying to understand "Bloom Filters" and the reason why they are used in many popular NoSQL datastores such as Cassandra, HBase, etc.  Even Google Chrome brower uses it to match URLs of malicious sites.

So what exactly is a Bloom filter? A Bloom filter is a data-structure; essentially a byte array (aka byte vector). Using a bloom filter, we can quickly test for the containment of a particular element in a given set. For e.g. Suppose you have a set of 100K URLs that are malicious. How do you check if the entered URL belongs to the list? You can store all the 100K URLs in memory and iterate through them, but that would be expensive in terms of CPU cycles and memory requirements.

A Bloom filter is a very simple and efficient way to test for containment - with a one-sided error probability. What this means is that if the Bloom filter states that an element is NOT present in the list, then it is 100% true. But if it returns true for the containment test, then it means that the element "MAY" be there; i.e. you could have the risk of a 'false positive'. You can design your Bloom filter data structure in such a way, so that you can predict the probability of 'false positives'; for e.g. 3% probability in a set of 100 million entries. Hence a bloom filter can return 'false positives' but cannot return 'false negatives'.

With the above understanding, you can guess why Bloom filters are popular with NoSQL data-stores. It is used to quickly check whether a row exists in the database or not, before going for disk-access.

The following articles would give you a quick understading of how Bloom Filters work:
http://billmill.org/bloomfilter-tutorial/
http://www.michaelnielsen.org/ddi/why-bloom-filters-work-the-way-they-do/

In a nutshell, in simple terms the following steps are taken:

1. Take a byte array (all elements of the array set to 0)
2. For each element in your set, do the following:
• Hash the element k times (with different hash algorithms). Let's say you hashed it 2 times and the values were 7 and 10. Set the bits at these indexes in the byte array to 1. 
• Do this for all the elements in the set. Thus you would end up with the byte array (Bloom filter) having 0s and 1s scattered around. 
3. Now to test whether an element is present in the set of not, we hash the element using the same hash functions. Now check if there is a 1 or 0 in the Bloom filter at those index positions. If there is a 0, that means the element is definitely not in the list. If there is a 1, then the element may be there and you can proceed with the costly disk-access or service call. 

This stackoverflow discussion also has a good explanation of how bloom filters work.
Google's popular Guava library aslo has a Bloom Filter class that be used in Java projects. The trick is to select the right size of the byte array and the hash functions to use for excellent performance.
.NET implementations are available here and here. 

Ruminating on multicore CPUs and hyperthreading

My laptop has an i3 processor and I knew that i3 processors have 2 physical cores. But when I open Task Manager, I can see 4 CPUs. Even opening "device manager" shows 4 CPUs. So I was a bit confused on this.

A bit of research on the internet showed that the i3 processors have Hyper-Threading (HT) enabled by default. What that means is that the OS sees each 'physical' core as two 'logical cores'. This enables the operating system to schedule tasks to both the logical CPUs simultaneously.

The performance increases because whenever there is a cache miss or the CPU enters wait state due to a dependency, the other waiting thread can be put on the CPU core immendiately. This ensures optimal utilization of our CPU core resources.

So how is this different from traditional multithreading? In multi-threading the OS does the time-division multiplexing between multiple threads, whereas in HT the OS sees the core as two logical CPUs. Also in HT, when we talk about threads, it is hardware level threads. Each OS maps its OS level threads to the hardware threads.

Web Application Performance Optimization Tips

The following link on Yahoo Developer network on Web App Performance is timeless ! I remember having used these techniques around 8 yrs ago and all of them are still valid. A must read for any web application architect.
http://developer.yahoo.com/performance/rules.html

Another cool utility provided by Yahoo for optimizing image file size on a web page is "Smush.it".
Just upload any image to this site and it would optimize the image size and allow the new image file to be downloaded for your use. 

Ruminating on Availability and Reliability

High availability is a function of both hardware + software combined. In order to design a highly available infrastructure, we have to ensure that all the components are made highly available and not just the database or app servers. This includes the network switches, SSO servers, power supply, etc.

The availability of each component is calculated and then we typically multiply the availabilities of all components together to get the overall availability, usually expressed as a percentage.

Common patterns for high availability are: Clustering & load-balancing, data replication (near real time), warm standby servers, effective DR strategy, etc. From an application architecture perspective availability would depend on effective caching, memory management, hardened security mechanisms, etc.

Application downtime occurs not just because of hardware failures, but could be due to lack of adequate testing (including unit testing, integration testing, performance testing, etc.) It's also very important to have proper monitoring mechanisms in place to proactively detect failures, performance issues, etc.

So how is availability typically measured? It is expressed as a percentage; for e.g. 99.9% availability.
To calculate the availability of a component, we need to understand the following 2 concepts:

Mean Time Between Failure (MTBF): It is defined as the average length of time the application runs before failing. Formula: Total Hours Ran / No. of failures (count)

Mean Time To Recovery (MTTR): It is defined as the average length of time needed to repair and restore service after a failure. Formula: Hours spend on repair / Failure Count

Formula: Availability = (MTBF / (MTBF + MTTR)) X 100

Using the above formula, we get the following percentages:

3 nines (99.9% availability) represents about ~ 9 hours of service outage in a single year. 
4 nines (99.99% availability) come to ~ 1 hour of outage in a year. 
5 nines (99.999% availability) represents only about 5 minutes of outage per year.

Ruminating on Server Side Push Techniques

The ability to push data from the server to the web browser has always been a pipe-dream for many web architects. Jotting down the various techniques that I used in the past and the new technologies on the horizon that would enable server side push.

• Long Polling (Comet): For the last few years, this technique has been most popular and is used behind the scenes by multiple ajax frameworks, such as DoJo, WebSphere Ajax toolkit, etc. The fundamental concept behind this technique is for the server to hold on to the request and not respond till there is some data. Once the data is ready, push the data to the browser as the HTTP Response. After getting the response, the client would again make a new poll request and wait for the response. Hence the term - "long polling". 
• Persistent Connections / Incomplete Response: Another technique in which the server never ends the response stream, but always keeps it open. There is a  special MIME type called multipart/x-mixed-replace, that is supported by most browsers expect IE :) This MIME type enables the server to keep the response stream open and send data in deltas to the browser.
• HTML 5 WebSockets: The new HTML 5 specification brings to us the power of WebSockets that enable full-duplex bidirectional data flow between browsers and servers. Very soon, we would have all browsers/servers supporting this. 

Peformance impact of TDE at database level

I was always under the opinion that column-level encryption is better and more performant than database or tablespace level encryption. But after much research and understanding the internal working on TDE (Transparent Data Encryption) on SQLServer and Oracle, it does not look to be a bad deal !

In fact, if we have a lot of columns that need to be encrypted and also need to fire queries against the encrypted columns, then a full database (tablespace) level encryption using TDE seems to be the best option.

I was a bit skeptical on the issue of performance degradation in using full database TDE, but it may not be so. First and foremost, column-level (cell) encryption can severely affect the database query optimization functions and result in significantly worse performance than encrypting the entire database.
When we use TDE at the database (tablespace) level, then the DB engine can use bulk encryption for entire blocks of data as they are written to or read from the disk.

It is important to note that full database TDE actually works at the data-file level and not at each table/column level. To put it in other words, the data is not encrypted but rather entire data files (index files, log files, etc.) are encrypted.

Microsoft states that the performance degradation of using database level TDE is a mere 3-6%.
Oracle states that in 11g, if we use Intel XEON processesor with AES instruction set, then there is a "near-zero" impact on database performance.

It is important to note the terminology differences regarding TDE used by Microsoft and Oracle. Microsoft refers to full database encryption as TDE (not column-level). Oracle calls it TDE-tablespace and TDE-column level.

Also TDE is a proven solution from a regulatory perspective - e.g. PCI. Auditors are more comfortable approving a proven industry solution that any custom logic that is implemented in application code.

Ruminating on "Trickle Batch Load".

Traditionally most of the batch jobs were run at end-of-day; where the entire day transaction log was pulled out and processed in some way. During the good old days, when the volume of data was low, these batch processes could easily meet their business SLAs. Even if there was a failure, there was sufficient time to correct the data and rerun the process and still meet the SLA's.

But today, the volume of data has grown exponentially. Across many of our customers, we have seen challenges around meeting SLA's due to large data volumes. Also business stakeholders have become more demanding and want to see sales reports and spot trends early - many times during a day. To tackle these challenges, we have to resort to the 'trickle batch' load design pattern.

In trickle batch, small delta changes from source systems are sent for processing mutiple times a day. There are various advantages of such a design -

• The business gets real-time access to critical information. For e.g. sales per hour.
• Business SLA's can be easily met, as EOD processes are no longer a bottleneck. 
• Operational benefits include low network latency and reduced CPU/resource utilization.
• Early detection of issues - network issues, file corruption, etc. 

The typically design strategies used to implement "trickle batch" are using the CDC (Change Data Capture) capabilities of Databases and ETL tools.
Today almost all ETL tools such as IBM InfoSphere DataStage, Informatica, Oracle Data Integrator, etc. have in-built CDC capabilities. 

Trickle batches typically feed a ODS (Operational Data Store) that can run transactional reports. Data from the ODS is then fed to a DW appliance for MIS reporting.

Ruminating on Session Replication

In any web application cluster, we need to configure Session Replication for failover and high availability. While configuring session replication, the following common queries often come across our minds. I have attempted to answer these questions in a product agnostic way.

Q) Is the session replicated sychronously or asynchronously?
A) Session replication can occur either synchronously or asynchronously. In sychronous replication, the request does not return until the session has been replicated across all members of the cluster. This obviously has performance implications. In asynchronous replication, the response is returned and the session data is queued to be replicated across the cluster nodes. Typically asynchronous replication is the default mode on most app servers and this is configured with "Session Affinity".

Q) Is the entire session replicated or only the delta of what has changed?
A) It could be quite difficult to keep track of all session data modifications accurately. For e.g. someone would just refer a object in session and change its properties, without calling setAttribute() again. Hence typically app servers would replicate the entire session object each time for every request.

Q) What are the different topology options for configuring the memory-to-memory session replication?
A) For small clusters, we can set up all-to-all peer replication - i.e. the session is replicated across all the nodes of the cluster. For large clusters, we can set up "replication groups" or "buddy groups" that are essentially a group of nodes that would replicate session data between themselves. In some environments such as WebSphere Extreme Scale, one can configure dedicated JVMs in a grid for storing sessions.

Byte code instrumentation and the ORM magic

All ORM tools use some kind of byte-code instumentation to do the persistance magic behind the scenes. But as an architect, it is important to understand what Hibernate or any JPA tool does to the entity classes?

Hibernate 'enhances' entity classes at runtime using a byte-code library called Javaassist. For e.g. it adds a '_dirty' flag to each field. It also adds a '_loaded' flag for each field to support lazy loading. A good blog explaining these concepts is here. So Hibernate reads the XML configuration or obtains annotations at runtime using reflection to apply byte-code instumentation.

There are various ways of doing byte-code instumentation using libraries such as CGLib, ASM, Javaassist, etc.
This byte-code enhancement can be done at compile-time or run-time. For Hibernate, besides a few special cases which require compile time 'enhancement' to byte-code; all common scenarios can be satisfied with runtime instrumentation.

The following link gives a good overview of all the enhancement options available in JPA.
http://openjpa.apache.org/builds/1.2.1/apache-openjpa-1.2.1/docs/manual/ref_guide_pc_enhance.html

In the .NET world, NHibernate uses the Linfu or Castle Dynamic Proxies byte-code enhancement providers. 
http://nhforge.org/blogs/nhibernate/archive/2008/11/09/nh2-1-0-bytecode-providers.aspx