Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Revisiting 1M Writes per second

techblog.netflix.com

Post reply on HN
Print view View original

21–30 of 40 posts

Re: Revisiting 1M Writes per second

#21

Post by Retric »

opendais wrote: post #15
arrryarr wrote: post #14

How many companies can possibly afford the cost and management pain of running a 285 node database? Few would have 285 servers of any type? So if this is what it takes to get 1M writes on Cassandra that is some poor ROI. 10K writes/sec is not impressive.

How would you build out at a metrics cluster that needs 1M writes/sec? 285 nodes that are easily automated to create/destroy/monitor doesn't seem like a management pain to me, personally. Just depends if you have the write internal tools built.

I have seen single servers do 250k writes/second. So it really depends on the data and access patterns more than some arbitrary one size fit's all solution.

Re: Revisiting 1M Writes per second

#23

Post by alex_sf »

JonoBB wrote: post #22

Couldn't help but notice: $398.70 per hour = $9568.80 per day = ~$3.5m per annum. They obviously get a discount...but still. What kind of discount do guys like this get?

I'd be surprised if anyone got discounts as deep as Netflix considering their usage.

For what it's worth, 3.5m/yr is about 0.08% their revenue.

Re: Revisiting 1M Writes per second

#24

Post by JonoBB »

alex_sf wrote: post #23
JonoBB wrote: post #22

Couldn't help but notice: $398.70 per hour = $9568.80 per day = ~$3.5m per annum. They obviously get a discount...but still. What kind of discount do guys like this get?

I'd be surprised if anyone got discounts as deep as Netflix considering their usage. For what it's worth, 3.5m/yr is about 0.08% their revenue.

Well, that puts things in perspective.

Re: Revisiting 1M Writes per second

#25

Post by rsynnott »

Retric wrote: post #21
opendais wrote: post #15

Earlier quoted context omitted.

How would you build out at a metrics cluster that needs 1M writes/sec? 285 nodes that are easily automated to create/destroy/monitor doesn't seem like a management pain to me, personally. Just depends if you have the write internal tools built.

I have seen single servers do 250k writes/second. So it really depends on the data and access patterns more than some arbitrary one size fit's all solution.

I suspect systems where the dataset, or at least the indexes, largely fit in memory?

Re: Revisiting 1M Writes per second

#26

Post by rsynnott »

JonoBB wrote: post #22

Couldn't help but notice: $398.70 per hour = $9568.80 per day = ~$3.5m per annum. They obviously get a discount...but still. What kind of discount do guys like this get?

If they were doing this for real, they'd be using reservations; with one-year reservations, the i2.xlarge bit (the servers) cost $905k/year.

Re: Revisiting 1M Writes per second

#27

Post by rsynnott »

threeseed wrote: post #20
arrryarr wrote: post #14

How many companies can possibly afford the cost and management pain of running a 285 node database? Few would have 285 servers of any type? So if this is what it takes to get 1M writes on Cassandra that is some poor ROI. 10K writes/sec is not impressive.

I don't understand your point. The type of companies who could afford this are the types of companies who need 1M writes/second. Which are few and far between. And yes 10K is not that impressive but 1M is. And with Cassandra you could continue to improve that number just be rolling out more nodes.

They probably mean 10K writes/sec/node, which would be correct assuming a replication factor of three. It doesn't sound huge, but if it were sustained, with a key set vastly larger than memory, it's not too bad.

Re: Revisiting 1M Writes per second

#28

Post by jandrewrogers »

threeseed wrote: post #20
arrryarr wrote: post #14

How many companies can possibly afford the cost and management pain of running a 285 node database? Few would have 285 servers of any type? So if this is what it takes to get 1M writes on Cassandra that is some poor ROI. 10K writes/sec is not impressive.

I don't understand your point. The type of companies who could afford this are the types of companies who need 1M writes/second. Which are few and far between. And yes 10K is not that impressive but 1M is. And with Cassandra you could continue to improve that number just be rolling out more nodes.

It is quite cheap to average a million writes per second. I've done it with five servers on AWS, and that was spatially indexing billions of GeoJSON polygons through storage while running queries against the index.

Many companies need far in excess of a million writes per second. Basically, most machine-generated data sources, whether it is personal location data or any other kind of telemetry. Many companies that do not generate that data themselves buy and consume it. I know of companies doing over a billion writes per second.

Cassandra is pretty good for this type of thing among open source software but it is not nearly as efficient as it could be in terms of write throughput. If the storage engine is correctly designed, you should be able to drive 10GbE all the way through storage -- call it 1 GB/sec per node. However, that does mean you can't do things like mmap()-ing files; those interfaces are slow due to poor scheduling by the OS when the throughput is very high.

Re: Revisiting 1M Writes per second

#29

Post by Retric »

rsynnott wrote: post #25
Retric wrote: post #21

Earlier quoted context omitted.

I have seen single servers do 250k writes/second. So it really depends on the data and access patterns more than some arbitrary one size fit's all solution.

I suspect systems where the dataset, or at least the indexes, largely fit in memory?

Most of the active data-set fit in ram, but considering you can get 100+GB of RAM that's generally not much of an issue. Also a single mid range SSD can break 120K writes per second so 0+1 RAID arrays can get crazy fast for less money than you might think.

Re: Revisiting 1M Writes per second

#30

Post by corysama »

jandrewrogers wrote: post #28
threeseed wrote: post #20

Earlier quoted context omitted.

I don't understand your point. The type of companies who could afford this are the types of companies who need 1M writes/second. Which are few and far between. And yes 10K is not that impressive but 1M is. And with Cassandra you could continue to improve that number just be rolling out more nodes.

It is quite cheap to average a million writes per second. I've done it with five servers on AWS, and that was spatially indexing billions of GeoJSON polygons through storage while running queries against the index. Many companies need far in excess of a million writes per second. Basically, most machine-generated data sources, whether it is personal location data or any other kind of telemetry. Many companies that do…

Out of curiosity, what would a correctly designed storage engine do to get better throughput than mmap()ing files?
Post reply on HN