Wednesday, August 1, 2012
Example Distributed Queue using Zookeeper (via Curator)
We use Storm for distributed processing. We've been using JMS as a means of driving work into Storm via the spout we developed:
But since Storm uses Zookeeper under the hood, we thought we could perhaps use Zookeeper instead of JMS for certain low-throughput distributed messaging needs, thereby decreasing the number of technologies we needed to maintain in our stack for certain applications.
Fortunately, there is a Distributed Queue recipe for Zookeeper.
Even better, Netflix has bundled an implementation of this recipe into Curator.
The docs for the Curator recipe were adequate, but I didn't see a complete example. After some muddling around, I was able to get it working. You can find the code here:
In the end, the throughput was just too slow. (You could see the messages scroll by on the screen as they were being consumed)
Eventually, I punted and decided to take a look at Kafka. That experiment is going *extremely* well. Kafka is very impressive. More on that in a future post. =)