A lot of people get confused between the different product offering by IBM in the Websphere space.The major products in the Websphere family are Websphere Application Server, Websphere ESB, Websphere Process Server and Websphere Message Broker.
Process Server is based on ESB and ESB is based on Application Server. Message Broker is termed as "Advanced ESB" by IBM and should be used when we have extensive heterogeneous infrastructures, including both standard and non-standards-based applications, protocols, and data formats.
So when to use what? Use Websphere ESB whenever most of the services you want to integrate are based on SOAP, XML standards. The Websphere ESB product can also be used as Websphere Application Server as it is built on the same core. Though Websphere Application Server can host simple services, Websphere ESB gives out of the box support for sophisticated mediation flows found in ESBs. We would need Websphere Process Server whenever we need to orchestrate simple services to create business services and host them.
Another technical difference is that Websphere ESB uses the JMS provider in WAS for messaging, whereas Message Broker uses Websphere MQ as the messaging engine. This is due to legacy reasons, I believe.
More information can be found here.The FAQ available at IBMs site also contains some good info.