One of the great appeals of Cassandra is its linear scalability. Need more speed? Just add water, er... nodes to your ring. Proving this out, Netflix performed one of the most famous Cassandra benchmarks.
Cassandra's numbers are most impressive, but Netflix didn't perform a side-by-side comparison of the available NoSQL platforms. Admirably, Yahoo! Cloud Serving Benchmark (YCSB) endeavored to perform just such a comparison. The results of that effort were recently published by Network World.
It's not surprising that Cassandra and HBase lead the pack in many of the metrics since both are based on Google's BigTable. It was surprising however to see HBase's latency near zero. This warranted some digging.
Now, side-by-side comparisons are always tough because they often depend highly on system configuration and the specifics of the use case / data model used. And in NetworkWorld's article, there is a key paragraph:
"During updates, HBase and Cassandra went far ahead from the main group with the average response latency time not exceeding two milliseconds. HBase was even faster. HBase client was configured with AutoFlush turned off. The updates aggregated in the client buffer and pending writes flushed asynchronously, as soon as the buffer became full. To accelerate updates processing on the server, the deferred log flush was enabled and WAL edits were kept in memory during the flush period"
With Autoflush disabled, writes are buffered until flush is called. See:
http://hbase.apache.org/book/perf.writing.html
If I'm reading that correctly, with autoflush disabled, the durability of a put operation is not guaranteed until the flush occurs. This really sets up an unfair comparison with the other systems where durability is guaranteed on each write. When buffering the data locally, nothing is sent over the wire, which naturally results in near-zero latency!
The change required to YCSB to level the playing field can be seen here:
I think its certainly worth including metrics when the autoflush is disabled because that is a valid use case, but YCSB should also include the results when autoflush is enabled. Similarly, it would be great to see Cassandra's numbers when using different consistency levels and replication factors. Durability is something we all need, but it isn't one-size fits all. (which is why tunable consistency/replication is ideal)
Anyway...
I appreciate all the work the YCSB crew put in to produce the benchmark. Thank you. And it is not to say you should take the benchmarks with a grain of salt, but you may need some mustard to go with the HBase autoflush bologna.
No comments:
Post a Comment