Byte code instrumentation and the ORM magic

All ORM tools use some kind of byte-code instumentation to do the persistance magic behind the scenes. But as an architect, it is important to understand what Hibernate or any JPA tool does to the entity classes?

Hibernate 'enhances' entity classes at runtime using a byte-code library called Javaassist. For e.g. it adds a '_dirty' flag to each field. It also adds a '_loaded' flag for each field to support lazy loading. A good blog explaining these concepts is here. So Hibernate reads the XML configuration or obtains annotations at runtime using reflection to apply byte-code instumentation.

There are various ways of doing byte-code instumentation using libraries such as CGLib, ASM, Javaassist, etc.
This byte-code enhancement can be done at compile-time or run-time. For Hibernate, besides a few special cases which require compile time 'enhancement' to byte-code; all common scenarios can be satisfied with runtime instrumentation.

The following link gives a good overview of all the enhancement options available in JPA.
http://openjpa.apache.org/builds/1.2.1/apache-openjpa-1.2.1/docs/manual/ref_guide_pc_enhance.html

In the .NET world, NHibernate uses the Linfu or Castle Dynamic Proxies byte-code enhancement providers. 
http://nhforge.org/blogs/nhibernate/archive/2008/11/09/nh2-1-0-bytecode-providers.aspx

Mapping between Entity Objects and DTOs

Very often, we need to map between our Entity objects and DTO's. This mapping code can be quite tedious to write.
There is a lot of hot debate on whether to use DTO's or just pass the entity objects directly to the view or webservices. There are pros and cons of each approach. Some good links on this debate are listed here:

Data Transfer Object - MSDN

http://stackoverflow.com/questions/5216633/jpa-entities-and-vs-dtos

Pros and Cons of Data Transfer Objects 

If you are using popular ORM tools such as Hibernate, iBatis or any other JPA complaint tool, then it may not even be possible to use the Enrity objects directly in your service or presentation tier. This is because these ORM toolkits typically use some kind of byte-code instrumentation to do the persistance magic behind the scenes. A good link explaining this is available here.

To avoid the drudgery of writing the 'Adapter/Mapping' code for each Entity object and DTO object, we can use some cool AutoMapper tools. These AutoMapper tools work on Reflection techniques and automatically map the source and target object properties. Custom mapping is supported using XML configuration or through code.

In the .NET world, there is a popular AutoMapper tool that has become the de-facto standard for a lot of .NET projects. In the Java world, there are 2 popular alternatives - Dozer and ModelMapper.
I found Dozer to be more comprehensive with some pretty good features. The usage is super-simple if you use the Singleton Wrapper and place the custom mapping file in the classpath.

If you are using the Spring Framework, then the 'BeanUtils' class has some simple static methods to copy properties from one object to the other. 

Performance benchmarks

Every development project needs a formal performance engineering process - one that emphasizes on early performance testing and benchmarking.

For performance benchmarks, it is recommended to do a shallow and wide implementation of a few critical use-cases and then run the load tests against the target hardware. These test results would help in some basic capacity planning.

But what if you have to do some initial rough capacity planning to allocate budgets and do not have the time to do a formal benchmarking exercise. It is here that standard performance benchmarks help. These standard performance benchmarks take a sample transactional use-case (e.g. Order Processing System) and run this workload on various platforms to gather statistics. There are 2 standards that are quite popular -

1. TPC (Transaction Processing Performance Council) - (TPC) is a non-profit organization founded to define transaction processing and database benchmarks and to disseminate objective, verifiable TPC performance data to the industry. TPC-C is the benchmark for OLTP workloads. 


2. SPECjEnterprise2010 - SPECjEnterprise2010 is an industry-standard benchmark designed to measure the performance of application servers conforming to the Java EE 5.0 or later specifications.

Interesting results of the performance benchmarks on various hardware can be found here:
http://www.tpc.org/tpcc/results/tpcc_perf_results.asp
http://www.spec.org/jEnterprise2010/results/jEnterprise2010.html

For the past few years, the Java Day Trader application and its .NET equivalent StockTrader application have been used by vendors to compare the performance of Java vs .NET on their respective platforms. Jotting down some links that point to some interesting debatable data :)

http://www.ibm.com/developerworks/opensource/library/os-perfbenchmk/index.html

http://blogs.msdn.com/b/wenlong/archive/2007/08/10/trade-benchmark-net-3-0-vs-ibm-websphere-6-1.aspx

http://msdn.microsoft.com/en-us/netframework/bb499684.aspx

https://cwiki.apache.org/GMOxDOC22/daytrader-a-more-complex-application.html

JavaDB (Derby) in JDK 1.6 and above

JDK 1.6 and above ship with a default pure Java database called as "JavaDB". Is is based on the open source Apache Derby project.

By default, on a Windows platform JavaDB gets installed at "C:\Program Files\Sun\JavaDB".
Set the 'DERBY_HOME' system property to this path. Also put 'DERBY_HOME/bin' in the PATH property.

There is a good tutorial here that should get you up and running with JavaDB in 10-15 mins :)

Derby does not have a default GUI admin tool, but one can use many third-party tools such as SQuirrel and others. I think JavaDB provides a good alternative to MySQL for some scenarios.

What is a framework?

When someone says they have defined a "framework", what does it mean? Is a framework just a library of resuable components? Or is it something more?

There is a good article on CodeProject on the same topic - http://www.codeproject.com/Articles/5381/What-Is-A-Framework

The book "Applying UML and Patterns" by Craig Larman also gives a very good understanding of the concept. Jotting down snippets from both these resources, in my own words.One may consider them the 10 guiding principles while designing a framework.

1. At the risk of oversimplification, a framework can be defined as a cohesive set of classes/interfaces that provide services for the core part of a logical subsystem. 
2. A framework contains both concrete and abstract classes that define interfaces to conform to, and other object interactions.
3. Frameworks usually allow the end-users to define sub-classes of existing framework classes for customization and extension of the framework services.
4. A framework enforces adherence to a consistent design approach.
5. Relies on the "Hollywood Principle" - "Don't call us, we will call you". This pattern is also called as IoC (Inversion of Control). 
6. A framework makes it easier to work with complex technologies.
7. A framework reduces/eliminates repetitive tasks.
8. A framework is often re-usable across multiple scenarios -  regardless of high level design considerations. Frameworks offer a higher degree of reuse - much more than individual classes.
9. A framework forces the team to implement code in a way that promotes consistent coding, fewer bugs, and more flexible applications.
10.  A framework can be used as a software building block in the system architecture definition. 

Eclipse Memory Analyser

Read the following good reviews on Eclipse Memory Analyser. Looks like it can read both SUN JVM HPROF memory dumps as well as IBM JDK dumps.

http://memoryanalyzer.blogspot.in/2010/01/heap-dump-analysis-with-memory-analyzer.html

http://memoryanalyzer.blogspot.in/2010/02/heap-dump-analysis-with-memory-analyzer.html#more

http://www.eclipse.org/mat/

Some other interesting blogs that would help us resolve OOM errors :)

http://www.rallydev.com/engblog/2011/09/20/outofmemoryerror-fun-with-heap-dump-analysis/

http://www.rallydev.com/engblog/2012/03/16/java-memory-problems-why-is-my-heap-exhausted/

There is also a good article that contains sample code to simulate a Java OOM error and uses the Memory Analyser tool to identify the root cause of the error - http://www.javacodegeeks.com/2012/05/gc-overhead-limit-exceeded-java-heap.html

C heap vs Java heap

Found this interesting discussion on StackOverFlow around C Heap and Java Heap.
A good read and its important to understand that the JVM is also ultimately a C program :)
 
http://stackoverflow.com/questions/78352/what-runs-in-a-c-heap-vs-a-java-heap-in-hp-ux-environment-jvms

RAID basics

Found this good blog that explains in simple terms, the various levels of RAID (Redundant Array of Independent Disks).

RAID 10 has become the defacto standard for relational databases, due to the excellent redundancy and performance given by them.  In RAID 10 (also known as 1+0), blocks are mirrored and also striped. 

'volatile' keywork in Java

Found this excellent article on the web explaining the 'volatile' keyword in Java and how it can be used for concurrency. The tutorial also explains the changes to the volatile keyword functioning in Java 5.

Also found it interesting to understand what 'livelock' is? We often encounter dead-lock and thread starvation in parallel programming, but livelock is also possible :)

Difference between Concurrent Collections and Synchronized Collections in JDK

Traditionally, we have also used object locks (semaphores) and synchronized methods to make our collections thread-safe. But having an exclusive lock on an object brings in scalability issues.

Hence the latest versions of JDK have a new package called "java.util.concurrent". This package contains many new collections objects that are thread-safe, but not so because of synchronization :)

More details at this link: http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/package-summary.html

Snippet from the above link:

The "Concurrent" prefix used with some classes in this package is a shorthand indicating several differences from similar "synchronized" classes. For example java.util.Hashtable and Collections.synchronizedMap(new HashMap()) are synchronized. 

But ConcurrentHashMap is "concurrent". A concurrent collection is thread-safe, but not governed by a single exclusion lock. In the particular case of ConcurrentHashMap, it safely permits any number of concurrent reads as well as a tunable number of concurrent writes.

 "Synchronized" classes can be useful when you need to prevent all access to a collection via a single lock, at the expense of poorer scalability. In other cases in which multiple threads are expected to access a common collection, "concurrent" versions are normally preferable. And unsynchronized collections are preferable when either collections are unshared, or are accessible only when holding other locks. 

Most concurrent Collection implementations (including most Queues) also differ from the usual java.util conventions in that their Iterators provide weakly consistent rather than fast-fail traversal. A weakly consistent iterator is thread-safe, but does not necessarily freeze the collection while iterating, so it may (or may not) reflect any updates since the iterator was created. 

Also a good post on Concurrency basics is available at: http://docs.oracle.com/javase/tutorial/essential/concurrency/memconsist.html (All chapters a must read :)

Another good blog that explains how ConcurrentHashMap maintains several  locks instead of one single mutex to deliver better performance.

Google Chart APIs

The last time (around 1 year ago), when I had evaluated Google Charts API, I was a bit disappointed. The Chart API only enabled you to embed an image (chart) that would be created on Google servers.

But looks like Google has completely revamped the concept and branding and have a brand new Chart API.

The new charting API looks cool and very easy to use. There are also samples and libraries that will help you write server side code to pass chart data to the client. I was particularly impressed with the Java DataSource library and Oracle PL/SQL library.

It is important to note that Google right now, does not allow these JS API to be downloaded offline and used in a web app that does not have internet connection (e.g. intranet applications). Please consider this constraint before you decide to use Google Chart Tools.

The older version of the image charts is still available here.There is also an online tool for quicking creating an image of a chart. Could be useful if you quickly want to create some stuff for your PPTs :)

JavaScript coding guidelines

With RIA applications becoming the norm, developers have to deal with a lot of JavaScript code. It is important to have proper coding conventions for JS too. Was glad to see a good document posted by Google on JavaScript coding conventions.

http://google-styleguide.googlecode.com/svn/trunk/javascriptguide.xml

Also, there are tools available to check the code quality of JavaScript code. Here are a few examples available:

http://docs.codehaus.org/display/SONAR/JavaScript+Plugin

http://jslint.com/


 

JAMon lives on !

I have been a big fan of JAMon tool for monitoring Java applications.It is lightweight and has a cool UI that gives you the stats you want. Also it is very clean and simple to use.

The last time I used JAMon was around 5 years ago. I was suprised to see that the project is still alive and kicking and has a few updates that make it even more interesting. I liked the concept of Listeners, which makes it easy to customoize JAMon.

The one thing that was missing in JAMon in the yesteryears was a consolidation app to read stats from multiple nodes in a cluster and present in on a unified dashboard. Luckily there seems to be a project called JARep that just does that. There is no much documentation available on JARep, but there is a good case study on DZone that explains how JARep works.

The case study is available at: http://architects.dzone.com/articles/case-study-performance-tuning--0

Though I have experiemented with a lot of other tools, I found JAMon to be the best and simplest. One can get it up and running in a project with a few minutes.

Passing wilcard * in Runtime.exec() command

One of the projects I was consulting on approached me with a peculiar problem.
The application was executing unix commands from a Java program using RunTime.exec() APIs. But strangely for some reason, the "rm" command was not working. The user account running the Java process had the necessary rights for deletion, so the problem was somewhere else.

A quick googling around answered the problem :)
http://www.coderanch.com/t/423573/java/java/Passing-wilcard-Runtime-exec-command

Its important to remember that when we pass wildcards to the RunTime.exec() API, it will treat it as a string only. It is the Unix shell that understands the "*" syntax.
Hence something like this would work: Runtime.exec(new String[] { "sh", "-c", "rm /tmp/ABC*" });

Cool library for Java NIO

I often felt the need to write a wrapper class libaray around Java NIO packages to reduce the complexity for developers.
I was pleased to find one opensource project that does just that :)

The Netty project provides a very clean and easy API for writing network applications. Very useful if we need to implement a customized protocol for special purpose.

Closed Loop vs Open Loop Models in Card Processing

Found this good article on the internet that describes the differences between closed loop and open loop models.
Excerpts from the article:

"Open-loop payments networks, such as Visa and MasterCard, are multi-party and operate through a system that connects two financial institutions—one that issues the card to the cardholder, known as the issuing financial institution or issuer, and one that has the banking relationship with the merchant, known as the acquiring financial institution or acquirer—and manages information and the flow of value between them.
In a typical closed-loop payments network, the payment services are provided directly to merchants and cardholders by the owner of the network without involving third-party financial institution intermediaries. Closed-loop networks can range in size from networks such as American Express and Discover, which issue cards directly to consumers and serve merchants directly."


The site also has another interesting link on how companies such as American Express make money and the competitive advantage they gain because of the closed loop model.

The Architecture of Open Source Applications

Found this cool book on the web, that explains the history behind many successful open source projects.

The chapters also contain a good amount of technical information - which was a pleasure to read :)

http://www.aosabook.org/en/index.html

Techniques for website design on all devices

Today, there is a growing demand to create websites that can render across multiple devices such as desktop browsers, tables and smart phones. New standards such as HTML5 help in this regard.

But it's important to apply proper design principles when we develop web pages that can reder across a wide array of devices. Found this cool article that describes a few techniques that can be used. The author of this blog 'Ethan Marcotte' has also written a book on this called "Responsive Web Design".

The core philosophy is how can we use CSS3/HTML5 to enable our application to detect the capabilities (size, resolution, JS support, etc.) of the browser and adapt the layout of the page accordingly. 

Lightweight UML sketching tool - UMLet

For years, I have been searching for a lightweight tool to quickly draw UML diagrams - in order to brainstorm an idea with my team. The traditional tools I have been using for UML were Rational Software Architect, IBM System Architect, Visual Studio 2010, Visio, ArgoUML, etc.

All the above tools are good, but are quite heavy to use and require installation. If you need a robust and quick UML sketching tool, then UMLet will blow your mind :)

I was particularly impressed with the simplicity of the tool. Drawing class diagrams, sequence/activity diagrams, package structures, deployment diagrams are a breeze....And it can run standalone as a JAR file or as an eclipse plugin. Loved the 'geeky' properties editor. U need to use it to appreciate it :)
It does not support code generation, as it was not the design intention of the tool.
Highly recommended for all agile architects!

How to ensure that IOCP is used for async operations in .NET?

In my last post, I had blogged about IO Completion Ports and how they work at the OS kernel level to provide for non-blocking IO.

But how can the 'average Joe' developer ensure that IOCP is being used when he uses async operations in .NET?

Well, the good news is that a developer need not worry about the complexities of IOCP as long as he is using the BeginXXX and EndXXX methods of all objects that support async operations. For e.g. SQLCommand has BeginExecuteReader/EndExecuteReader that you can use for asynchronously reading data from a database. FileStream, Socket class all have BeginXXX/EndXXX methods that use IOCP in the background. Under the bonnet,  these methods use IO completion ports which means that the thread handling the request can be returned to the threadpool while the IO operation completes. 

Some versions of Windows OS may not support IOCP on all devices, but the developer need not worry about this. Depending on the target platform, the .NET Framework will decide to use the IOCompletionPorts API or not, maximizing the performance and minimizing the resources.

An important caveat is to avoid using normal async operations for non-blocking IO - such as "ThreadPool.QueueUserWorkItem, Delegate.BeginInvoke", etc. because these do not use IOCP, but just pick up another thread from the managed thread pool. This defeats the very purpose of non-blocking IO, because then the async thread is drawn from the same process-wide CLR thread pool.

Non blocking IO in .NET (Completion Ports)

Non blocking IO is implemented in Windows by a concept called 'IO Completion Ports' (IOCP).
Using IOCP, we can build highly scalable server side applications that can perform asynchronous IO to deliver maximum throughput for large workloads.

Traditionally server side applications were written by assigning one thread to a socket connection. But this approach seriously limited the number of concurrent connections that a server can handle. By using IOCP, we can overcome the "one-thread-per-client" problem, because 'worker' threads are not blocked for IO. Rather there is a separate pool of IO threads called 'Completion Port Threads' that wait on a special kernel level object called 'Completion Port'.

A completion port is a kernel level object that you can bind with a file handle - either a file stream, database connection or a socket stream. Multiple file handles can be bound to a single completion port. The .NET CLR maintains its own completion port and can bind any file handle to it. Each completion port has a queue associate with it. Once a IO operation completes, a message (completion packet) is posted to the queue. IO threads block or 'wait' on this completion port queue, till a message is posted. The waiting IO threads (a.k.a completion port thread) pick up the messages in the queue in FIFO order. Hence any thread may handle any completion message packet. It is important to note that threads are 'woken' in a LIFO order, so chances are that caches are still warm.

The following links throw more light on this:
http://blog.stevensanderson.com/2008/04/05/improve-scalability-in-aspnet-mvc-using-asynchronous-requests/
http://www.codeproject.com/Articles/1052/Developing-a-Truly-Scalable-Winsock-Server-using-I

Why does the .NET Thread Pool have a separate worker thread pool and a Completion Port pool?
I believe that technically there is no fundamental difference in the nature of the threads associated with each pool. Worker threads are meant to do active work, where as Completion Port threads are meant to wait on completion ports. Since IO threads wait on CPs, they may block for longer periods of time. Hence the .NET framework has created separate categories for them. If there was a single pool, then there could be a situation where a high demand on worker threads exhausts all the threads available to dispatch native I/O callbacks,
potentially leading to deadlock.

Looks like in IIS 7, the threading model has undergone drastic changes. More info available here. 

Why no delegates in Java? And will Closures come to Java?

Having worked across Java and .NET platforms, I often compare the features of one over the other. One of the interesting features of the .NET platform is the concept of 'delegates'.

At first, a Java guy may take some time to understand the concept of delegates, but once you are hooked on to it, you tend to use it everywhere...because it is so convienient. The .NET framework uses delegates extensively throughtout its event framework.Java folks have traditionally used the 'Listener' interface pattern for eventing. Even concurrent/parallel libraries in .NET heavily use delegates, whereas Java folks have to still stick with interfaces :(  The closest equivalent to delegates in Java is the anonymous inner class - which IMHO is messy to read and write.

StackOverFlow has a series of interesting discussion threads on this topic:
http://stackoverflow.com/questions/44912/java-delegates
http://stackoverflow.com/questions/2635013/why-not-net-style-delegates-rather-than-closures-in-java
http://stackoverflow.com/questions/1340231/is-there-an-equivilent-of-c-sharp-anonymous-delegates-in-java
http://stackoverflow.com/questions/1973579/why-doesnt-java-have-method-delegates


Another interesting feature that many dynamic languages have is 'closures'. A closure is similar to the concept of delegate, but they are not quite the same. Martin Fowler has a good bliki post explaining the concept of Closures and the difference compared to delegates.

.NET supports both closures and delegates. Found this good article explaning closures in .NET.

Connection timeouts in a mirrored SQLServer

Recently, one of my teams was facing a connection timeout issue when we tried to implement 'parallelism' in a data-driven application.
A colleague of my mine pointed out that there was a bug in ADO.NET (with a mirrored SQL Server) that could result in this wierd behavior. More details available at this link.

Quick resolution is to try increasing the connection timeout and allocate a greater no of connections at start-up in the pool.

Business Intelligence vs Analytics

My collegue Sandeep Raut has a very simple blog-post explaining the differences between traditional BI and Analytics. Summarizing a few key points from the blog below.

"BI traditionally is concerned with creating reports on past data or even current live data. We create OLAP cubes using which we can slice & dice the data, even do a drill down. Analytics is about analyzing the data using mathematics/statistics to identify patterns. These patterns can then be used to predict what may happen in the future. Analytics is about identifying relationships between key data variables that were unknown before. It is about surfacing unknown patterns."

But in my humble opinion, should Analytics not be a subset of BI? I can understand the hype that product vendors create to differentiate their products in the market, but can Analytics exist in isolation to BI? Even predictive data analysis using "realt-time" data/text mining techniques would logically fall under BI....
After all BI is all about meeting business needs through actionable information !
Maybe it is just a game of words and semantics. I remember a few years back, the term DSS (Decision Support Systems) was more widely used than BI :)

Using Parallelism in .NET WinForm applications

We all have gone through the travials of multi-threaded programming in WinForm applications. The challenge in WinForm applications is that the UI controls are bound to the thread that created/rendered them; i.e. the UI control can only by updated by the main thread or the GUI thread that created it.

But to keep the UI responsive, we cannot execute any long running task (>0.5 sec) on the UI thread, else the GUI would hang or freeze. If we run the business logic asynchronously on another thread, then how do we pass the results back to the main GUI thread to update the UI?

Traditionally this has been done using the Control.Invoke() methods. More details on this approach is available on this link: http://msdn.microsoft.com/en-gb/magazine/cc300429.aspx

But with the introduction of TPL, there is another alternative way of doing this. We can use the TaskScheduler and SynchronizationContext classes to call heavy lifting work and then pass the results to the main GUI thread.

For e.g.

TaskScheduler uiScheduler = 
           TaskScheduler.FromCurrentSynchronizationContext();
new Task({Your code here}).start(uiScheduler);

Given below are 2 excellent articles eloborating this in detail:
http://www.codeproject.com/Articles/152765/Task-Parallel-Library-1-of-n

http://reedcopsey.com/2010/03/18/parallelism-in-net-part-15-making-tasks-run-the-taskscheduler/

Sacha Barber has an excellent 6 series article on the intricacies of TPL, which I loved reading.

Parallelism in .NET

In one of my previous blogs, I had pointed out to an interesting article that shows how TPL controls the number of threads in the Thread Pool using hill-climbing heuristics.

In order to understand why TPL (Task Parallel Library) is far superior to simple muli-threading, we need to understand the concepts of global queue, local queue on each thread, work-stealing algorithms, etc.
Given below are some interesting links that explain these concepts with good illustrations.

http://www.danielmoth.com/Blog/New-And-Improved-CLR-4-Thread-Pool-Engine.aspx

http://blogs.msdn.com/b/jennifer/archive/2009/06/26/work-stealing-in-net-4-0.aspx

http://udooz.net/blog/2009/08/net-4-0-work-stealing-queue-plinq/

A few important points to remember:

• There is one global queue for the default Thread Pool in .NET 4.0
• There is also a local queue for each Thread. The Task Scheduler distributes the tasks from the global queue to the local queues on each Thread. Even sub-tasks created by each Thread get queued on the local queue. This improves the performance, as there is no contention to pick up work items (tasks) from the global queue; especially in a multi-core scenario.
• If a thread is free and there are no tasks in its local queue and also global queue, then it will steal work from other threads. This ensures that all cores are optimally utilized. This concept is called 'work stealing'.
• Tasks from the global queue are picked up in 'FIFO' order. Tasks from the local queue are picked up in 'LIFO' order based on the assumption that the last-in is still hot in the cache. Work stealing again happens in 'FIFO' order.

There is a wonderful book on parallel computing available on MSDN that is a must read for everyone.

Data Services in the Microsoft world

In my previous blog, I ranted on the concept of Data Services in creating a data virtualization layer. In the .NET world, data services equate to WCF data services (formerly a.k.a ADO.NET data services)

Microsoft is propogating the use of an open standard called OData for building REST style data services. A good article describing OData is available on MSDN. OData essentially leverages JSON/ATOM and HTTP semantics to build a simple data services layer across disparate data sources.
But looks like besides M$, there are no big vendors jumping on the OData bandwagon. Its interesting to note that WebSphere eXtreme Scale Servers also expose a OData service.

Ruminating of Data Virtualization

The industry is flooded with confusing terms when it comes to understanding 'Data Virtualization'. We have IaaS (Information as a service), Data Services, EII (Enterprise Information Integration), Data Federation, etc. and so on! The point is that there are no industry standard definitions for these analyst-coined terms and there is a lot of overlap between them.

Rick Lans tries to clear the cloud with some simple definitions here. Another interesting post by Barry Devlin throws more light on the concept of data virtualization.

The core concept behind data virtualization is to create an abstraction layer (Data Access Layer) that hides the complexities of the underlying disparate data sources and provides a unified view of the enterprise data to the applications. This can be implemented using "SOA style" Data Services or creating a virtual data layer that can be queried using SQL-like semantics. More info can be found at these links: Link1 & Link2

RedHat has a nice whitepaper explaining the concept of Data Services in a SOA environment. This post explains the benefits of data virtualization. Composite Software is a leader in data virtualization techniques and has shared a couple of interesting case studies that demonstrate the use of their data virtualization platform.

One thought that came to my mind was regarding the challenges in accessing NoSQL data from the data virtualization layer. While some type of NoSQL datastores such as XML documents, Key/Value pairs can be exposed as a relational SQL view, it may not be possible to have a uniform query interface for unstructured data. All NoSQL data stores will expose some kind of Java API that can be used for querying. Would it be possible to create a common set of meta-data for both structured and unstructured data?
In such scenarios, IMHO, the only strategy for data virtualization is to use Data Services.

Google Protocol Buffers

Just found a good post by the Google Engineering team ranting about the historical context of Google Protocol Buffers.
My first reaction to GPB was - "Why on earth another binary serialization format"?
I think the reason behind the popularity of GPB has been its simplicity and ease of use. 

This site has an interesting discussion on comparing GPB to XML/JSON.  A few snippets from the site comments/discussions -

• A major difference between protocol buffers and JSON is that protocol buffers use a binary format, while JSON is plain text.  Because it's binary, the format is more compact and easier to interpret by a computer - which makes protocol buffers faster than JSON.
• Another reason GPB is so fast is that it uses positional binding. JSON is less bloated compared to XML (which is over bloated), it still sends the name of the attribute with each record. That creates an enormous amount of overhead. PB, on the other hand, uses positional binding and doesn't send the attribute names at all.
•  Binary protocols have to deal with portability issues like byte-order (little/big-endian) etc., there are advantages when it comes to parsing dates, timestamps, etc.

Alternatives to XML Serialization

Today, there are a lot of alternatives for XML serialization of data structures. These data interchange formats are smaller and faster than processing XML.
Most popular are Google Protocol Buffers, Thrift (from FaceBook),  Avro and MessagePack. A good article comparing these alternatives is available here -
http://www.igvita.com/2011/08/01/protocol-buffers-avro-thrift-messagepack/

Wikipedia also has an interesting article comparing various data serialization.