My team was working on implementing CEP (Complex Event Processing) capabilities using Apache Storm. We evaluated multiple options for doing so - one option was using a lightweight in-process CEP engine like Esper within a Storm Bolt.
But there was another option of manually implementing CEP-like aggregations (over a sliding window) using Java code. The following links show us how to do so.
http://www.michael-noll.com/blog/2013/01/18/implementing-real-time-trending-topics-in-storm/
Rolling Count Bolt on Github
While the above code would help in satisfying certain scenarios, it would not provide the flexibility of a CEP engine. We need to understand that CEP engines like (Tibco BE, Esper, StreamInsights) are fundamentally different from Apache Storm; which is more of a highly distributed stream computing platform.
A CEP engine would provide you with SQL like declarative queries and OOTB high level operators like time window, temporal patterns, etc. This brings down the complexity of writing temporal queries and aggregates. CEP engines can also detect patterns in events. But most CEP engines do not support a distributed architecture.
Hence it makes sense to combine CEP with Apache Storm - for e.g. embedding Esper within a Storm Bolt. The following links would serve as good reference -
http://stackoverflow.com/questions/29025886/esper-ha-v-s-esper-storm
http://stackoverflow.com/questions/9164785/how-to-scale-out-with-esper/9776881#9776881
But there was another option of manually implementing CEP-like aggregations (over a sliding window) using Java code. The following links show us how to do so.
http://www.michael-noll.com/blog/2013/01/18/implementing-real-time-trending-topics-in-storm/
Rolling Count Bolt on Github
While the above code would help in satisfying certain scenarios, it would not provide the flexibility of a CEP engine. We need to understand that CEP engines like (Tibco BE, Esper, StreamInsights) are fundamentally different from Apache Storm; which is more of a highly distributed stream computing platform.
A CEP engine would provide you with SQL like declarative queries and OOTB high level operators like time window, temporal patterns, etc. This brings down the complexity of writing temporal queries and aggregates. CEP engines can also detect patterns in events. But most CEP engines do not support a distributed architecture.
Hence it makes sense to combine CEP with Apache Storm - for e.g. embedding Esper within a Storm Bolt. The following links would serve as good reference -
http://stackoverflow.com/questions/29025886/esper-ha-v-s-esper-storm
http://stackoverflow.com/questions/9164785/how-to-scale-out-with-esper/9776881#9776881
No comments:
Post a Comment