First steps for diagnosis of memory leaks in Websphere Application Server v6.1

- Enable verbose GC.

- If using Sun JVM, use the -XX:+HeapDumpOnOutOfMemoryError option to tell the VM to generate a heap dump if OutOfMemoryError is thrown. If using IBM JVM, then enable automatic heap dump generation.

- Start the lightweight memory leak detection

- Generate heap dumps manually if required using the wsadmin tool.

- Use the MDD4J tool (Memory Dump Diagnostic for Java) for diagnosing root causes behind memory leaks in the Java heap. The heap dump collected in the above steps will be the input to this tool. More information about this tool can be found here , here and here.
The MDD4J tool supports the following heap dump formats:
1.IBM Portable Heap Dump (.phd) format (for WebSphere Application Server Versions 6.x on most platforms)
2.IBM Text heap dump format (for WebSphere Application Server Versions 5.0 and 4.0 on most platforms)
3.HPROF heap dump format (for WebSphere Application Server on the Solaris® and HP-UX platforms)
4.SVC Dumps (WebSphere on the IBM zSeries)

On Solaris platform, starting with Java 1.5, Sun has been shipping a cool tool called jmap which allows you to attach to any 1.5 JVM and obtain heap layout information, class histograms and complete heap snapshots. The cool thing is that you don’t have to configure the JVM with any special options, and that it therefore runs exactly as during normal operation.

jmap -dump:format=b,file=snapshot2.jmap PID_OF_PROCESS
jhat snapshot2.jmap

Websphere SOA Stack

A lot of people get confused between the different product offering by IBM in the Websphere space.The major products in the Websphere family are Websphere Application Server, Websphere ESB, Websphere Process Server and Websphere Message Broker.

Process Server is based on ESB and ESB is based on Application Server. Message Broker is termed as "Advanced ESB" by IBM and should be used when we have extensive heterogeneous infrastructures, including both standard and non-standards-based applications, protocols, and data formats.

So when to use what? Use Websphere ESB whenever most of the services you want to integrate are based on SOAP, XML standards. The Websphere ESB product can also be used as Websphere Application Server as it is built on the same core. Though Websphere Application Server can host simple services, Websphere ESB gives out of the box support for sophisticated mediation flows found in ESBs. We would need Websphere Process Server whenever we need to orchestrate simple services to create business services and host them.

Another technical difference is that Websphere ESB uses the JMS provider in WAS for messaging, whereas Message Broker uses Websphere MQ as the messaging engine. This is due to legacy reasons, I believe.

More information can be found here.The FAQ available at IBMs site also contains some good info.

Webpshere v6.1 deployment hack

Whenever I wanted to change a single file on WAS server, I often used to copy that file to the 'installedApps' EAR directory of the application.

This used to work and though not mentioned in any IBM redbook; was considered as a hack by our team. But one day we ran into a problem which took us ages to decipher.

Whenever we changed the web.xml in WEB-INF, the changes were never reflected. For e.g. if we added a new struts-config entry to web.xml or a new filter, the configuration was never picked up.
We never really noticed this and thought the problem was with our struts-config file.
But the core issue was that WAS always picks up the web.xml file of an application from a different directory

-/opt/IBM/WebSphere/AppServer/profiles/
{profileName}/config/cells/{CellName}/applications/
{Application_EAR}/deployments/{Application_WAR}/WEB-INF

I changed the web.xml file here and all issues with struts-config were solved :)

Adding trusted root certificates in Websphere 6.1

We had deployed an application to WAS 6.1 ND. This application contained a JAX-WS 2.0 webservices client that used to call a third-party service using SSL. The digital certificate used by the third-party was self-signed and hence we needed to accept it as a trusted party in our Trust Store.

We imported the certs into a trust-store (JKS format) using the keytool command of the JDK and wrote the following code:

System.setProperty
("javax.net.ssl.trustStore", trustFilename );
System.setProperty
("javax.net.ssl.trustStorePassword", "changeit") ;
System.setProperty
("javax.net.ssl.keyStore", trustFilename );
System.setProperty
("javax.net.ssl.keyStorePassword", "changeit");

But unfortunately this was not working on WAS6.1. (This works fine on Tomcat).
In earlier versions of WAS, the iKeyman tool provided an interface to manipulate digital certs, keys, Trust stores and Key stores. But in WAS 6.1, all these tasks can be done from the web-based admin console.

So to add the certs to the Trust store, I went to "Security (on the left panel) --> SSL certificate and key management > Key stores and certificates > NodeDefaultTrustStore > Signer certificates"
Add the new root digital certs that need to be trusted to this store. The JAX-WS client should now be able to connect to the HTTPS resource. Remove any system properties that have been set before.

A good article (for WAS 6.0) describing how SSL and Digital certs work in Websphere can be found here.

Hack to deploy JAX-WS 2.0 webservice on Websphere 6.1

Our team had developed a webservice using JAX-WS 2.0 and using the Reference implementation of Sun as the JAX-WS provider. The application was developed using Netbeans and the embedded Tomcat server.

But when the application was deployed on WAS 6.1, then it failed and resulted in ClassCast exceptions. The base installation of WAS 6.1 only supports JAX-RPC 1.1.
IBM did provide an option - installing the webservices feature pack for WAS 6.1
The installation was a bit of a pain, especially on a cluster. You essentially had to create new AppServer profiles, create a new cluster and move all applications to the new cluster. This would also required considerable downtime on a live production server.

I looked for an other option. I tried to understand what was happening under the hoods when Netbeans was creating a webservice. It was clear that the webservice created was using a Servlet provided by the RI of JAX-WS 2.0. Even all the annotations and JAXB libraries were included in the lib directory. Hence there was no dependancy on any server-specific library as such.

I realised that the ClassCast exceptions on WAS 6.1 were happening due to the ClassLoader heirarchy - which was set to 'Parent First'. I decided to try to change this configuration to 'Child First' - so that JAX-WS RI libraries are picked up from WEB-INF/lib first.

There were 2 changes that I made using the web admin console. Go to "Enterprise Applications --> {Application name} --> Class Loader" and changed the following items:
Select "Classes loaded with application class loader first" and "Single class loader for application". Save these changes to the master configuration and restart the AppServer.

This hack worked and we could deploy JAX-WS 2.0 webservices on WAS 6.1 :)

Tip: We ran into a few Linkage errors, which we could resolve by removing all duplicate jar files.
A linkage error would typically occur when a class tries to call another class loaded by a different class loader.