Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cassandra Hits One Million Writes Per Second on Google Compute Engine

googlecloudplatform.blogspot.com

Post reply on HN
Print view View original

11–20 of 41 posts

Re: Cassandra Hits One Million Writes Per Second on Google Compute Engine

#11

Post by ivansmf »

arielweisberg wrote: post #5

I like Cassandra and I think it's something you should look at hard if what you want is an eventually consistent database because multi-DC and availability are important to the problem you are solving. The blog post fails to make that case anywhere. I am guessing that is because it is an ad for GCE and not Cassandra. I know these kinds of benchmarks are necessary to get and keep your widget on the map, but they alway…

I ran those tests. One of the challenges in benchmarking is to pick the workload, so I chose one our customers do more often. For IO testing I prefer to run FIO tests, and for networking iperf, but unless you do this for a living it is hard to relate to microbenchmarks. It would also leave out the Java settings.

The reason I left the tests running for an hour is because at some point Cassandra gets into compactions, which are CPU and read intensive. The engine memory maps the files, so the IO subsystem backing the storage sees a lot of random IOs as page faults kick in. Streaming IOs get in during fsyncs. Again, easier to see using FIO.

The cluster was brought up at once, and I added the ramp up time on the chart so folks could see it.

I had three goals for this test: - Show low latency is possible with remote storage and proper capacity planning. This is why I used quorum commit, which forces the client to wait for at least two nodes to commit. - Share the settings I used. If you open the GIST and download the tarball you'll find all changes I did to the cassandra.yaml and cassandra-env.sh. This benefits our customers directly because it gives them a starting point. - Recommend that customers look at all samples, not only the middle 80% the stress tool reports. Otherwise the cluster looks much better than it is.

Thank you for the comments! I can assure you I read them, and I'll incorporate suggestions as possible.

Re: Cassandra Hits One Million Writes Per Second on Google Compute Engine

#12

Post by ithkuil »

t0mas88 wrote: post #4
outside1234 wrote: post #2

Is anyone actually using Google Compute Engine? I haven't bumped into anyone using it and would love to hear what the real world experience is with it.

I've tested it, network latency is by far not as good as for example Rackspace Cloud. And tool-support is very far behind Amazon and OpenStack. Basically Google Compute Engine is something that might have had a chance years ago but at this point in time it's quite pointless because it has no upsides compared to existing far better established solutions and is behind on features, documentation, tools and user-support.…

> And I expect that not to get better soon if nobody is using it...

If the people running GCE see that people are not using it because it sucks at X, they will probably do their best to address X. So complaining is a good thing, especially if it contains enough information in order to understand the problem (e.g. latency from/to where? internal network or external network?)

Re: Cassandra Hits One Million Writes Per Second on Google Compute Engine

#13

Post by ivansmf »

thrownaway2424 wrote: post #9
jasonlotito wrote: post #7

Earlier quoted context omitted.

Keep in mind, it's not "One Million Writes Per Second," it's "One Million Writes Per Second on Google Compute Engine" with "Google Compute Engine" being the key point to the article. The "one million writes per second" for Cassandra has been written about before (in this case, on AWS): http://techblog.netflix.com/2011/11/benchmarking-cassandra-s...

That post doesn't mention anything about tail latency, while the GCE thing does point to P95 latency < 100ms consistently, which is nice.

I wrote the test - Yep. Tail latency is one of the key things here. And I took 100% of all samples, as opposed to the middle 80% the tool usually reports.

Re: Cassandra Hits One Million Writes Per Second on Google Compute Engine

#14

Post by ivansmf »

roncohen wrote: post #10

Did anyone actually care to see if that data can be read back? :) https://dev.mysql.com/doc/refman/5.0/en/blackhole-storage-en...

This is hilarious - it would have saved me days of work. Here is the GIST with my step by step. https://gist.github.com/ivansmf/6ec2197b69d1b7b26153

Re: Cassandra Hits One Million Writes Per Second on Google Compute Engine

#15

Post by ivansmf »

ivansmf wrote: post #11
arielweisberg wrote: post #5

I like Cassandra and I think it's something you should look at hard if what you want is an eventually consistent database because multi-DC and availability are important to the problem you are solving. The blog post fails to make that case anywhere. I am guessing that is because it is an ad for GCE and not Cassandra. I know these kinds of benchmarks are necessary to get and keep your widget on the map, but they alway…

I ran those tests. One of the challenges in benchmarking is to pick the workload, so I chose one our customers do more often. For IO testing I prefer to run FIO tests, and for networking iperf, but unless you do this for a living it is hard to relate to microbenchmarks. It would also leave out the Java settings. The reason I left the tests running for an hour is because at some point Cassandra gets into compactions,…

Here is the GIST link: https://gist.github.com/ivansmf/6ec2197b69d1b7b26153

Re: Cassandra Hits One Million Writes Per Second on Google Compute Engine

#16

Post by primitivesuave »

In the end these are just eye-popping benchmarks. What really drives people to your platform is developer friendliness and the speed by which you can get started. Google App Engine has this, but there's not a lot of information floating around the media and blogosphere as there is for Heroku and DigitalOcean.

I use Google App Engine for auto scaling an API and it works brilliantly. Super easy to set up and develop on. But recently I started preferring DigitalOcean simply because there is a community that is constantly posting tutorials and answering questions. To me, that's more valuable than the distant prospect of handling 1M writes/second.

Re: Cassandra Hits One Million Writes Per Second on Google Compute Engine

#17

Post by capkutay »

Can anyone comment on Cassandra's performance versus HBase? HBase adds the complexity of dealing with the whole HStack and clustering can be a pain if you have no need to use HDFS/Zookeeper in the first place. Cassandra seems nice because its a single platform and each node is an equal member of the cluster, no need for designating HMaster, data nodes, etc. I was just wondering if Cassandra's widely regarded as less performant.

I know the true answer lies in my exact use cases and weeks of initial testing, but it would be nice to hear someone's opinion first.

Re: Cassandra Hits One Million Writes Per Second on Google Compute Engine

#18

Post by oamoruwa »

The 1M writes/second capability is a valuable benchmark in analyzing continuously variable and real-time information, similar to CFD analysis are high-throughput computational analysis where you end up dealing with multi-variate dimensions of data continuously in flux.

Re: Cassandra Hits One Million Writes Per Second on Google Compute Engine

#19

Post by coolsunglasses »

capkutay wrote: post #17

Can anyone comment on Cassandra's performance versus HBase? HBase adds the complexity of dealing with the whole HStack and clustering can be a pain if you have no need to use HDFS/Zookeeper in the first place. Cassandra seems nice because its a single platform and each node is an equal member of the cluster, no need for designating HMaster, data nodes, etc. I was just wondering if Cassandra's widely regarded as less…

Honestly, just google "HBase vs. Cassandra" and go from there.

Just keep in mind that between the two, Cassandra has improved a lot more than HBase has.

HBase can be an easy choice if you already have a Hadoop cluster and want to roll the results of Map-Reduce jobs into HBase keys.

The DataStax crew has a decent stack for turning batch/OLAP jobs into queryable keys.

If you have no need of that, want tunable consistency, favor write availability over read performance, then Cassandra might be a fit.

Just uh, don't pretend Cassandra clusters are necessarily trivial to manage just because they're homogenous.

IMHO: put off moving to any of these technologies as long as possible.

Re: Cassandra Hits One Million Writes Per Second on Google Compute Engine

#20

Post by [deleted] »

coolsunglasses wrote: post #19
capkutay wrote: post #17

Can anyone comment on Cassandra's performance versus HBase? HBase adds the complexity of dealing with the whole HStack and clustering can be a pain if you have no need to use HDFS/Zookeeper in the first place. Cassandra seems nice because its a single platform and each node is an equal member of the cluster, no need for designating HMaster, data nodes, etc. I was just wondering if Cassandra's widely regarded as less…

Honestly, just google "HBase vs. Cassandra" and go from there. Just keep in mind that between the two, Cassandra has improved a lot more than HBase has. HBase can be an easy choice if you already have a Hadoop cluster and want to roll the results of Map-Reduce jobs into HBase keys. The DataStax crew has a decent stack for turning batch/OLAP jobs into queryable keys. If you have no need of that, want tunable consisten…

[deleted]
Post reply on HN