Ruminating on Transactions

Most of us have worked on transactions one time or the other. Transactions can be broadly classified as local or distributed.
Local transactions are of the simplest form in which the application peforms CRUD operations with a single datasource/database. The simplest form of relational database access involves only the application, a resource manager, and a resource adapter. The resource manager can be a relational database management system (RDBMS), such as Oracle or SQL Server. All of the actual database management is handled by this component.
The resource adapter is the component that is the communications channel, or request translator, between the "outside world," in this case the application, and the resource manager. In Java applications, this is a JDBC driver.

In a distributed transaction, the transaction accesses and updates data on two or more networked resources, and therefore must be coordinated among those resources.
These resources could consist of several different RDBMSs housed on a single sever, for example, Oracle, SQL Server, and Sybase; or they could include several instances of a single type of database residing on a number of different servers. In any case, a distributed transaction involves coordination among the various resource managers. This coordination is the function of the transaction manager. The transaction manager is responsible for making the final decision either to commit or rollback any distributed transaction. A commit decision should lead to a successful transaction; rollback leaves the data in the database unaltered

The first step of the distributed transaction process is for the application to send a request for the transaction to the transaction manager. Although the final commit/rollback decision treats the transaction as a single logical unit, there can be many transaction branches involved. A transaction branch is associated with a request to each resource manager involved in the distributed transaction. Requests to three different RDBMSs, therefore, require three transaction branches. Each transaction branch must be committed or rolled back by the local resource manager. The transaction manager controls the boundaries of the transaction and is responsible for the final decision as to whether or not the total transaction should commit or rollback. This decision is made in two phases, called the Two-Phase Commit Protocol.

In the first phase, the transaction manager polls all of the resource managers (RDBMSs) involved in the distributed transaction to see if each one is ready to commit. If a resource manager cannot commit, it responds negatively and rolls back its particular part of the transaction so that data is not altered.

In the second phase, the transaction manager determines if any of the resource managers have responded negatively, and, if so, rolls back the whole transaction. If there are no negative responses, the translation manager commits the whole transaction, and returns the results to the application.

Making web service proxy URL dynamic in .NET

In VS.NET, whenever we add a web-reference to a webservice, what happens behind the scenes is that a proxy is created which contains the URL to the webservice.

Now if we need to shift the webservice to a production server, do we have to create the proxy again?. (The IP address of the webservice server would have changed)

Fortunately there is a easy way out. Just right-click on the proxy in VS.NET and change its URL property from 'static' to 'dynamic'. And Voila !!!, VS.NET automatically adds code to the proxy class and web.config. The newly added code in the proxy will first check for a URL key in the web.config and use it to bind to the webservice.
So when we move the webservice to a different server, we just have to change the URL in the web.config file.

Object pooling in Java

Quite often, we may have to use object pooling to conserve resources and increase performance.
I came across some pretty good abstract components that can be used to develop a object pooling mechanism.

Check out the followling link to see all the classes that are available for use:
http://jakarta.apache.org/commons/pool/

An interesting aspect of the package was the clear separation of object creation from the way the objects are pooled. There is a PoolableObjectFactory interface that provides a generic interface for managing the lifecycle of a pooled instance.

By contract, when an ObjectPool delegates to a PoolableObjectFactory : -

makeObject is called whenever a new instance is needed.
activateObject is invoked on every instance before it is returned from the pool.
passivateObject is invoked on every instance when it is returned to the pool.
destroyObject is invoked on every instance when it is being "dropped" from the pool (whether due to the response from validateObject, or for reasons specific to the pool implementation.)
validateObject is invoked in an implementation-specific fashion to determine if an instance is still valid to be returned by the pool. It will only be invoked on an "activated" instance.

Doug Lea's Home Page

I have been impressed with the genius of Doug Lea when I saw his concurrent package in Java. His home page contains a lot of interesting links : http://gee.cs.oswego.edu/dl/

Chief among them are :

• A online OO book : http://gee.cs.oswego.edu/dl/oosdw3/index.html
• Roles Before Objects : http://gee.cs.oswego.edu/dl/rp/roles.html

This page is worth a perusal.

Thread pools in Java

I have often used thread pools in my applications. My favourite opensource thread pool was the PooledExecutor class written by Doug Lea (available here )

But it's also important to understand when to use thread pools and when not to use. I recently came across a good article on the web - http://www-128.ibm.com/developerworks/java/library/j-jtp0730.html

In this article the author argues that using thread pools makes sense whenever U have a large number of tasks that need to be processed and each task is short-lived. For e.g. Webservers, FTP servers etc.
But when the tasks are long running and few in number it may make sense to actually spawn a thread for each task. Another common threading model is to have a single background thread and task queue for tasks of a certain type. AWT and Swing use this model, in which there is a GUI event thread, and all work that causes changes in the user interface must execute in that thread.

Java 5.0 has come up with a new package java.util.concurrent that contains a lot of cool classes if U need to implement threading in Ur applications.

StringBuilder in Java 5.0

There is a new class in Java5.0 - StringBuilder class that can be used whenever we are doing some heavy-duty string manipulation and we do not wish to get bogged down by the immutable property of strings.
But then what happened to the old good StringBuffer class. Well, it's still there.
Here's what the JavaDoc says:

This StringBuilder class provides an API compatible with StringBuffer, but with no guarantee of synchronization. This class is designed for use as a drop-in replacement for StringBuffer in places where the string buffer was being used by a single thread (as is generally the case). Where possible, it is recommended that this class be used in preference to StringBuffer as it will be faster under most implementations.

Instances of StringBuilder are not safe for use by multiple threads. If such synchronization is required then it is recommended that StringBuffer be used.

Understanding Unicode

I still see many people with a lot of myths about Unicode. I guess the reason for this is that a lot of people still feel that Unicode is a encoding format that uses 16 bits to represent a character.
Let's put a few things in perspective here:

Unicode is a standard which has defined a character code for every character in most of the speaking languages in the world. Also it has defined a character code for items such as scientific, mathematical, and technical symbols, and even musical notation. These character codes are also known as code points.

Unicode characters may be encoded at any code point from U+0000 to U+10FFFF, i.e. Unicode reserves 1,114,112 (= 220 + 216) code points, and currently assigns characters to more than 96,000 of those code points. The first 256 codes precisely match those of ISO 8859-1, the most popular 8-bit character encoding in the "Western world"; as a result, the first 128 characters are also identical to ASCII.

The number of bits used to represent each code point may differ - e.g. 8 bits, 16 bits.
The size of the code unit used for expressing those code points may be 8 bits (for UTF-8), 16 bits (for UTF-16), or 32 bits (for UTF-32).
So what this means is that there are several formats for storing Unicode code points. When combined with the byte order of the hardware (BE or LE), they are known officially as "character encoding schemes." They are also known by their UTF acronyms, which stand for "Unicode Transformation Format"

UTF-8 is widely used because the first 128 bits in the byte are ASCII, and although up to four bytes can be used, only one byte is required for use in the English speaking world. UTF-16 and UTF-32 use a fixed number of bytes.

So to put in other words, Unicode text can be represented in more than one way, including UTF-8, UTF-16 and UTF-32. So, hey...what's this UTF ?

A Unicode transformation format (UTF) is an algorithmic mapping from every Unicode code point to a unique byte sequence. UTF-8 is most common on the web. UTF-16 is used by Java and Windows. UTF-32 is used by various Unix systems. The conversions between all of them are algorithmically based, fast and lossless. This makes it easy to support data input or output in multiple formats, while using a particular UTF for internal storage or processing.

For more information visit http://www.unicode.org/faq/

On Bytes and Chars...

During i8n and localization, we often come across basic fundamental issues such as:
- How many bytes make a character?
- How many characters/bytes are present in a string?

Each character gets encoded into bytes according to a specific charset. For e.g. ASCII uses 7 bit encoding, i.e. each char is represented by 7 bits. ANSI/Cp1521 uses 8-bit encoding, Unicode uses 16 bit encoding. UTF-8, which is a popular encoding set on the internet is a multibyte Unicode charset. So if someone asks - how many bytes make a character - the answer is - it depends on the charset used to encode the character.

Another interesting point in Java is the difference btw a 'char' and a character.
When we do "String.length()" in Java, we get the number of chars in the string. But a Unicode character may be made up of more than one 'char'.
This blog throws light on this concept: http://forum.java.sun.com/thread.jspa?threadID=671720

Snippet from the above blog:
---------------------------------
A char is not necessarily a complete character. Why? Supplementary characters exist in the Unicode charset. These are characters that have code points above the base set, and they have values greater than 0xFFFF. They extend all the way up to 0x10FFFF. That's a lot of characters. In Java, these supplementary characters are represented as surrogate pairs, pairs of char units that fall in a specific range. The leading or high surrogate value is in the 0xD800 through 0xDBFF range. The trailing or low surrogate value is in the 0xDC00 through 0xDFFF range. What kinds of characters are supplementary? You can find out more from the Unicode site itself.

So, if length won't tell me home many characters are in a String, what will? Fortunately, the J2SE 5.0 API has a new String method: codePointCount(int beginIndex, int endIndex). This method will tell you how many Unicode code points are between the two indices. The index values refer to code unit or char locations, so endIndex - beginIndex for the entire String is equivalent to the String's length. Anyway, here's how you might use the method:

int charLen = myString.length();
int characterLen = myString.codePointCount(0, charLen);

Preventing SQL injection attacks...

I still see a lot of applications that are vulnerable to SQL injection attacks bcoz of usage of dynamic SQL using string concatenation. It is a common myth that to circumvent SQL injection attacks we 'have' to use stored procedures.
Even if we are using dynamic SQL, it is pretty simple to avoid these attacks.

In .NET the following techniques can be used:
Step 1. Constrain input - Validate input using client side validation and server side validation (e.g. using regular expressions)
Step 2. Use parameters with stored procedures. There is one caveat with Stored procedures. If Ur stored procedure uses the 'EXEC' command which takes a string, then the same vulnerability exists there too.
Step 3. Use parameters with dynamic SQL. (Yepppiee...in .NET it's so simple to have named parameters even for dynamic SQL)

Code snippet :
-----------------------------------
SqlDataAdapter myDataAdapter = new SqlDataAdapter( "SELECT au_lname, au_fname FROM Authors WHERE au_id = @au_id", connection);
myCommand.SelectCommand.Parameters.Add("@au_id", SqlDbType.VarChar, 11); myCommand.SelectCommand.Parameters["@au_id"].Value = SSN.Text;
-----------------------------------

Other important points to be considred are using a least-privileged database account and avoiding disclosing error information to the user.
A good article discussing this is at: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnpag2/html/paght000002.asp

In Java, one can use 'Prepared Statements' and 'Stored Procedures' to prevent SQL injection attacks.

Common terms in Datawarehousing.

I have been interested in datawarehousing concepts for a long time, but unfortunately never got a chance to work on them. Here are some common concepts U need to know to understand any datawarehousing glibber.

Data Warehousing: An enterprise-wide implementation that replicates data from the same publication table on different servers/platforms to a single subscription table. This implementation effectively consolidates data from multiple sources.

Data Warehouse: A data warehouse is a subject oriented, integrated, non volatile, time variant collection of data. The data warehouse contains atomic level data and summarized data specifically structured for querying and reporting.

Data Mining: The process of finding hidden patterns and relationships in data. For instance, a consumer goods company may track 200 variables about each consumer. There are scores of possible relationships among the 200 variables. Data mining tools will identify the significant relationships.

OLAP (On-Line Analytical Processing)
Describes the systems used not for application delivery, but for analyzing the business, e.g., sales forecasting, market trends analysis, etc. These systems are also more conducive to heuristic reporting and often involves multidimensional data analysis capabilities.

OLTP (OnLine Transaction Processing)
Describes the activities and systems associated with a company's day-to-day operational processing and data (order entry, invoicing, general ledger, etc.).

Data Dictionary:
a software tool for recording the definition of data, the relationship of one category of data to another, the attributes and keys of groups of data, and so forth.

Reusable Application blocks for .NET

I often used to write some utility classes that would simplify the coding of most used tasks.
For e.g. accessing a DataReader for binding to a grid, we have to write all the database plumbing code again and again.
Hence MS has released some utility wrapper classes that can be used to simplify such routine tasks. A good article explaining this is at http://aspnet.4guysfromrolla.com/articles/070203-1.aspx

The application blocks can be downloaded from http://www.microsoft.com/downloads

How to implement Role authorization after Forms authentication

After authenticating a user using forms authentication, we may want to restict access to certain parts of the website to certain users - i.e. Authorize users.

To implement Role based authorization we would need to set up a database containing info about which role a user belongs to. Then we need to construct a Principal object specifying which role the user belongs to and assign it to the HttpContext user property.

An excellent article discussing this concept is at:
http://aspnet.4guysfromrolla.com/articles/082703-1.aspx

Advantages of using netmodules in .NET

I often wondered what was the advantage of compiling files into .net modules and linking them together using the Al.exe tool. I also noticed that al.exe actually only creates a stub dll, the netmodule files really have to be physically deployed too.

I guess the advantage of compiling to netmodules are:

• Each module may be independently developed by a separate set of developers.
• Each module may be written in any .net language.
• A multimodule assembly does have memory advantages: if you put rarely-usedtypes in one module, that module will only be loaded when one of its types is referenced, when the type is first used. If the rarely-used types are never used in an execution of the assembly, the module is not loaded.

GAC folder in Windows

When we go the default GAC folder in Windows located at : "C:\WINNT\assembly" we see the dlls that are registered in the GAC. But what if we have different versions of the same DLL registered in the GAC? How can a single Windows folder allow 2 files with the same name to exist?

To understand this, go to the DOS prompt and navigate to "C:\WINNT\assembly". Do a 'dir' command and see the contents. There would be a 'GAC' folder and inside the folder, there is a folder for each Assembly. Each assembly folder has a folder for each version. Inside this version folder there is the actual DLL. Try this out :)

XML databinding in Java using Castor

Often in my applications, I needed to persist some Java data objects as XML. For this I used to use JDOM which was quite cool to use bcoz of its Java-like API.
But there is one more XML data binding framework known as "Castor" which can handle all the marshalling and unmarshalling of object to XML for you automatically or with the help of a mapping file. Castor is indeed a cool XML data binding framework.

Furthur reading:
OnJava tutorial
http://www.castor.org/

XML parsing APIs in .NET

The .NET API has quite a few classes to help us manipulate XML data.
We have the 'XMLReader' class and its subclasses like 'XMLTextReader', 'XMLValidatingReader' ect that provide a forward-only, non-cached, read-only cursor to XML data. An interesting point is that XMLReader uses a 'pull model' unlike SAX push event model.

Then we have XPathDocument that provides a fast, read-only cache for XML document processing using XSLT. Quite good if U need to use XPath to query the XML data.

We have a special class for loading a DataSet directly into a XML form for manipultion. This class is the XMLDataDocument class that can load either relational data or XML data and manipulate that data using the W3C Document Object Model (DOM).

Diff btw Java and C#.NET

I came across this comprehensive document stating all the minute differences between Java and C#.NET. A good read :)

http://www.25hoursaday.com/CsharpVsJava.html

I still love Struts...

I have been a strong advocate of Struts right from the start. I often see developers getting confused over making a choice between struts, velocity, tiles etc.

The point is that "Struts" is a "controller-centric" MVC solution. The "View" and the "Model" parts are pluggable thanks to the plug-in architecture of Struts1.1
Hence is possible to use "Tiles" or Velocity templates inplace of plain JSP for the View in Struts.
Similarly the Model can be used to interact with EJB, JDO, Hibernate...etc.

The various choices available in Server-side Java may seem daunting to a new developer,but with experience we come to realize and appreciate the value of each solution and where it fits best :)

Diff btw 'package' access specifier in Java and 'internal' access specifier in .NET

At first I thought that the "package" access-specifier in Java is equivalent to the "internal" access-specifier in .NET. But actually there is a subtle difference.

The 'package' access-specifier in Java limits the access to a particular package/namespace, where as the 'internal' access-specifier in .NET limits the access to the containing assembly. Now a .NET assembly can contain more than one 'namespace' or package.
So is there any way to restrict the scope of a member of a class to the same namespace?

Suprisingly NO, there is none in .NET..So if we were designing by 'separation of concerns', maybe it would make sense to map one namespace to one physical assembly.

File Upload in ASP.NET

There are a couple of things we need to keep in mind while doing a file-upload in ASP.NET.

If the upload fails it could be bcoz of the following reasons:

ASP.NET limits the size of file uploads for security purposes. The default size is 4 MB. This can be changed by modifying the maxRequestLength attribute of Machine.config's element.

If we are using the HtmlInputFile html control, then we need to set the the Enctype property of the HtmlForm to "multipart/form-data" for this control to work properly, or we might get 'null' for the PostedFile property.

How does the Page's IsPostBack property work?

IsPostBack checks to see whether the HTTP request is accompanied by postback data containing a __VIEWSTATE or __EVENTTARGET parameter. If there are none, then it is not a postback.

.NET Remoting fundas

Developers often get confused when they see the following methods (all sound the same).
RegisterWellknownServiceType()
RegisterWellknownClientType()
RegisterActivatedServiceType()
RegisterActivatedClientType()

Finally I got hold of a image that explains the above in a lucid manner...

Diff btw "protected" access specifier in Java and .NET

In Java, when a class is declared as 'protected', it is also given 'package' access automatically. i.e. a protected member can be accessed within the same package by other members.

But in .NET, 'protected' access specifier means that only subclasses can see it. If we want other members of a assembly to see the 'protected' class, then we need to put the access specifier as 'protected internal'.

Thus 'protected' in Java is equivalent to 'protected internal' in .NET

Improving the performance of Eclipse

I often used to get frustrated with the speed of eclipse on my machine. Then I heard from my friend how I can increase the initial memory allocated to Eclipse. U just have to pass values from the command prompt using the vmargs argument.

eclipse -vmargs -Xms256m -Xmx256m

This single change, gave me a very good "perceived" performance benifit while using the IDE.

Other tips to increase the performance of Eclipse can be found at:
http://eclipsewiki.editme.com/GeneralFaq

Eating Ur own dog food :)

I came accross this slang many a times over the last few years. So what does "eating Ur own dog food " mean?

“A company that eats its own dog food sends the message that it
considers its own products the best on the market.”

“This slang was popularized during the dotcom craze when some
companies did not use their own products and thus could "not even eat
their own dog food". An example would've been a software company that
created operating systems but used its competitor's software on its
corporate computers.”

How do download managers work?

I have often wondered how download managers worked? I wanted to know the internal working that makes it possible for these components to download faster?

The main functions of a download manager are:
• Resuming interrupted downloads (i.e., downloading only the rest of the file instead of restarting theprocess from the very beginning);
• Scheduled operation: connecting to the Internet, downloading a list of specific files and disconnectingaccording to a user-defined schedule (e.g. at night when the connection quality is usually higher, while the connection rates are lower);
• some download managers have additional functions: searching for files on WWW and FTP servers byname, downloading files in several "streams" from one or from different mirror servers, etc.

Most download managers use the concept of "Multi-connection downloading" - the file is downloaded in several segments through multiple connections and reassembled at the user's PC.
To understand how this would work, we first need to understand a feature of web-servers.
A lot of webservers (http and ftp) today support the "resume download" function - what this means is that if your download is interrupted or stopped, U can resume downloading the file from where U left it. But the question now arises, how does the client(web-browser) tell the server what part of the file it wants or where to resume download? Is this a standard or server proprietary? I was suprised when I found out that it is the HTTP protocol itself that has support for "range downloads", i.e. when U request for a resource, U can also request what portion/segment of the resource U want to download. This information is passed from the client as a HTTP header:

See http header snipper below:

GET http://lrc.aiha.com/English/Training/Dldmgrs-Eng.pdf?Cache HTTP/1.1
Host: lrc.aiha.com
Accept: */*
User-Agent: DA 7.0
Proxy-Authorization: Basic bmFyZW5kcjpuYXJlbjEyNDM=
Connection: Close
Range: bytes=0-96143

Now what download managers do is that they start a number of threads that download different portions of the resource. So the download manager will make another request with the header:

GET http://lrc.aiha.com/English/Training/Dldmgrs-Eng.pdf?Cache HTTP/1.1
Host: lrc.aiha.com
Accept: */*
User-Agent: DA 7.0
Proxy-Authorization: Basic bmFyZW5kcjpuYXJlbjEyNDM=
Connection: Close
Range: bytes=96143-192286

This solves the mystery of how the download managers are able to simultaneously download different portions of the resource.

Imp Note: To resume interrupted downloads, it is not enough to use a download manager: the server from which the file is being downloaded should support download resumption. Unfortunately, some servers do not support thisfunction, and are called “non-resumable.”
So Ur download managers won't work (no increase in speed either), as the servers would ignore the HTTP "range" header.

But I was still confused how exactly does this increase the speed. After all if the current bandwidth is "fully utilized" with one connection, how does making more connections help? The answers I found on the net are as below:

Normal TCP connections, as used by HTTP, encounter a maximum connection throughput well below that of the available bandwidth in circumstances with even moderate amounts of packet loss and signal latency. (so bcoz of packet loss and latency, the client will have to re-request some packets ) Multiple TCP connections can help to alleviate these effects and, in doing so, provide faster downloads and better utilization of available bandwidth.

Opening more connections means less sharing with others. Web servers are set up to split their bandwidth into several streams to support as many users downloading as possible. As an example, if the download manager created eight connections to the server, the server thinks it is transmitting to eight different users and delivers all eight streams to the same user. Each of the eight requests asks for data starting at a different location in the file.

Here are the links to some good download managers:

www.getright.com
www.internetdownloadmanager.com/
www.netants.com
www.alwaysfreeware.co.uk/dload.html

Accessing Windows Registry using Java

I always wished someone could provide me with a neat API for accessing the windows registry using Java. I did not want to get into the nitty-gritty of JNI calls..
Fortunately there is an opensource library available at:
http://www.trustice.com/java/jnireg/

Check out the source code to understand it better.

Recursively adding files to Clearcase.

Recently I wanted to add a whole directory struture to Clearcase VOB and I was suprised to see that the Graphical Explorer did not have a option to recursively add an entire directory structure to source-control. Thankfully I found out a command line tool ("clearfsimport") using
which I could import/add all directories and files to the VOB.
The general usage of the command is :

clearfsimport [ –preview ] [ –follow ] [ –recurse ] [ –rmname ]
[–comment comment ] [ –mklabel label ] [ –nsetevent ] [ –identical ][ –master ] [ –unco ] source-name [ . . . ] target-VOB-directory

For more details RTFM or check out these links:

http://www.cmcrossroads.com/ubbthreads/showflat.php?Cat=&Number=33221
http://www.cmcrossroads.com/ubbthreads/showflat.php?Number=33343

Double checked Locking idiom and the Singleton pattern.

I have seen a lot of programs using the double-checked locking pattern in their singleton classes to avoid the overhead of synchronization for each method call.

But it is now known that the double-check locking idiom does not work. It is bcoz of the memory model of Java, which allows out-of-order writes to memory. i.e. There exists a window of time, when an instance is not null, but still not fully initiliazed(constructor has not returned) !!!

For more explanation, check out this simple and lucid article:
http://www-128.ibm.com/developerworks/java/library/j-dcl.html

Immutable Collections in Java

Quite often, we may feel the need for an immutable collection, i.e. users cannot modify the collection and bring it to an invalid state, but can only read it.
The Collection API has methods to help us with this. For e.g. to get an immutable list, use the following code:

List ul = Collections.unmodifiableList(list);

Check out the other methods of Collections which give you other helper methods to make collections immutable. Any attempt to modify the returned list, whether direct or via its iterator, result in an UnsupportedOperationException.

The 'immutablity' aspect can also be used as a design pattern for concurrent read-and-write access to a collection. (Think mutiple users or threads). Anyone who needs to modify a collection will make a copy of the collection object and modify it.