Monday, September 22, 2014

Exploring Apache Kafka..

We had successfully used ActiveMQ and RabbitMQ in many projects and never felt the need to explore any other message broker. Today, my colleague introduced me to 'Apache Kafka' and was drooling over the high performance and reliability it provided. Kafka is extensively used within LinkedIn and can be used in many use-cases.

The following blog post gives a good performance benchmark of Kafka.
http://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines

Another good blog post worth reading is: http://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying

Another good tutorial on using Kafka to push messages to Hadoop is available here - http://hortonworks.com/hadoop-tutorial/simulating-transporting-realtime-events-stream-apache-kafka/