21–30 of 41 posts
Earlier quoted context omitted.
That post doesn't mention anything about tail latency, while the GCE thing does point to P95 latency < 100ms consistently, which is nice.
I wrote the test - Yep. Tail latency is one of the key things here. And I took 100% of all samples, as opposed to the middle 80% the tool usually reports.
In the end these are just eye-popping benchmarks. What really drives people to your platform is developer friendliness and the speed by which you can get started. Google App Engine has this, but there's not a lot of information floating around the media and blogosphere as there is for Heroku and DigitalOcean. I use Google App Engine for auto scaling an API and it works brilliantly. Super easy to set up and develop on…
The 1M writes/second would be a capability more specifically focused for users/developers interested in using already existing platforms such AWS's High Performance Computing platforms and the like.
Now I would like to see how many reads it can do at the same time. Cassandra does mostly-sequential writes, however most reads do couples of disk seeks.
In the end these are just eye-popping benchmarks. What really drives people to your platform is developer friendliness and the speed by which you can get started. Google App Engine has this, but there's not a lot of information floating around the media and blogosphere as there is for Heroku and DigitalOcean. I use Google App Engine for auto scaling an API and it works brilliantly. Super easy to set up and develop on…
Looks like a great way for DO to both build a sense community and also appear high in Google rankings when you search for something. I googled some Linux question earlier today, and ended up on a DO page -- refreshing their brand in my mind.
Earlier quoted context omitted.
I wrote the test - Yep. Tail latency is one of the key things here. And I took 100% of all samples, as opposed to the middle 80% the tool usually reports.
What was the network utilization during the test? If these machines were lightly loaded (< 30% utilized) then the tail latency isn't surprising. :)
Latency spikes come from Cassandra flushing data to disk (large sequential IO), Java garbage collection and heap resize, and page faults during compactions (random reads).
What I did to even traffic out was to enable trickle_fsync and size the flushes, set Java's max and min heap sizes, as well as to tune the Java heap ergonomics. I treated random reads as a fact of life - I did nothing to tune that.
In the end these are just eye-popping benchmarks. What really drives people to your platform is developer friendliness and the speed by which you can get started. Google App Engine has this, but there's not a lot of information floating around the media and blogosphere as there is for Heroku and DigitalOcean. I use Google App Engine for auto scaling an API and it works brilliantly. Super easy to set up and develop on…
I've been developing on GAE since it was released in 2008 and recently became "Google Cloud Platform Certified".
I'm considering offering a "Google App Engine for Startups" class/workshop (at something like General Assembly). The class would primarily include details about the platform's architecture and best practices for building high-scalability apps (like, say, Snapchat) so your app can "scale without thinking twice." Do you think there'd be some interest in a class like this?
Can anyone comment on Cassandra's performance versus HBase? HBase adds the complexity of dealing with the whole HStack and clustering can be a pain if you have no need to use HDFS/Zookeeper in the first place. Cassandra seems nice because its a single platform and each node is an equal member of the cluster, no need for designating HMaster, data nodes, etc. I was just wondering if Cassandra's widely regarded as less…
I like Cassandra and I think it's something you should look at hard if what you want is an eventually consistent database because multi-DC and availability are important to the problem you are solving. The blog post fails to make that case anywhere. I am guessing that is because it is an ad for GCE and not Cassandra. I know these kinds of benchmarks are necessary to get and keep your widget on the map, but they alway…
I ran those tests. One of the challenges in benchmarking is to pick the workload, so I chose one our customers do more often. For IO testing I prefer to run FIO tests, and for networking iperf, but unless you do this for a living it is hard to relate to microbenchmarks. It would also leave out the Java settings. The reason I left the tests running for an hour is because at some point Cassandra gets into compactions,…
That is 8 terabytes (n1-standard-8) of RAM for 18.6 * RF gigabytes of data or am I wrong? The entire thing should fit in the page cache assuming compaction keeps up. There are no deletes so no tombstones. On an overwrite workload I don't know what the space amplification will be and whether you will actually run out of RAM for caching.
90% of people who read a benchmark aren't going to look at it the way I do. They don't have a cost model that says how fast things should be and how much they should cost. I do and when things are fitting in memory I have a different a different set of expectations. If I am looking at the wrong instance type please let me know.
For the workload you described Cassandra shouldn't be doing random IO. I would expect there to be three + N streams of sequential IO. The write ahead log, memtable flushing, compaction output, and N streams reading tables for compaction.
All the write IO can be deferred and rescheduled heavily because fsyncs are infrequent. Reads for compaction are done by background tasks and shouldn't effect foreground latency.
If read ahead is not working for compaction (and killing disk throughput) that may be something that needs to be addressed. Compaction should be requesting IOs large enough to amortize the cost of seeking. Page faults in memory mapped files don't stop read ahead and I think that the kernel will even detect sequential access and double read ahead. For a workload like this with no random reads you could configure the kernel to read ahead 2-4 megabytes and the page cache would probably absorb it fine.
For the workload you described the only fsyncs would be the log (every 10 seconds) and memtable flushes/compaction finishing and that would normally be so infrequent as to not move the needle on overall IO capacity (although obviously it consumes sequential IO). You set trickle fsync so we are still talking about 10 megabyte writes.
Granted there are many things I don't know about Cassandra. I've plumbed a lot of it, but it isn't my day job. Using a 24 gigabyte heap and a 600 megabyte young generation is questionable to me. I think Cassandra can do better, and I also think there are several tools that would do the same job with 10-20x less nodes by exploiting the fact that they never have to do random reads from disk.
The "one million" number grabs some attention, but this isn't that special in my opinion. Doing 3333 writes/sec per node is not that hard with Cassandra, actually it can be much faster if you use a setup with fast local storage and split for example the data disk from the commit log. The Google article reads as if they used network storage and 1 volume per node, both are bad ideas for Cassandra as documented by Datas…
Keep in mind, it's not "One Million Writes Per Second," it's "One Million Writes Per Second on Google Compute Engine" with "Google Compute Engine" being the key point to the article. The "one million writes per second" for Cassandra has been written about before (in this case, on AWS): http://techblog.netflix.com/2011/11/benchmarking-cassandra-s...
According to Netflix article the AWS experiment did run at a cost $561 per 2h, that is ~$280 per hour. Perhaps they were not utilized the cluster fully in those 2h in which case we should multiply the 1h test that performed 500k inserts per second, in that case the cost would be $182*2 = ~$365 per 1h.
GCE test did run at the cost of $330 per hour. Give or take few dollars difference if anything it's surprising GCE can do at roughly the same cost what AWS was capable of 2+ years ago.
Saying all that GCE guys did a great effort. I wonder though how much speed you can squeeze from AWS and at what cost now when AWS is sporting SSD disks.
Earlier quoted context omitted.
What was the network utilization during the test? If these machines were lightly loaded (< 30% utilized) then the tail latency isn't surprising. :)
Network average utilization was low by design. Keeping it steady was more important than low, though, and harder too. Latency spikes come from Cassandra flushing data to disk (large sequential IO), Java garbage collection and heap resize, and page faults during compactions (random reads). What I did to even traffic out was to enable trickle_fsync and size the flushes, set Java's max and min heap sizes, as well as to…