Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Handling 1M Requests per Minute with Go (2015)

marcio.io

Post reply on HN
Print view View original

31–39 of 39 posts

Re: Handling 1M Requests per Minute with Go (2015)

#31

Post by meritt »

robbles wrote: post #29
meritt wrote: post #26

They're uploading each POST payload to S3 at a rate of up to 1M uploads a minute? They're going to go broke from S3 operational fees. PUT fees are $0.005 per 1k, or $5/minute, or $7200/day S3 is an absolutely terrible financial choice for systems that need to store a vast number of tiny files.

They're batching the requests into larger files on S3. The 1M refers to the number of HTTP requests hitting their server.

Can you show me where in the post that is described because I do not see it. All I see is a description of how they moved the UploadToS3 aspect to a job queue, but it's still sending individual files to S3.

Re: Handling 1M Requests per Minute with Go (2015)

#32

Post by natbobc »

meritt wrote: post #28
dx034 wrote: post #27

Earlier quoted context omitted.

Storing millions of tiny files in any filesystem is a terrible choice.

Fair point, I was mostly focused on the absurd cost for that specific implementation. What would you suggest as an alternative? A document-oriented database?

If you're on AWS I would probably go with DynamoDB, if you're on GCP Datastore. They aren't drop-in replacements for one another but the way you architect your system will be similar(ish). The main benefit is that it'll cost less upfront and require less to manage. Now that AWS have simplified back-ups it's a pretty simple system to operate. If you're looking for better controls over latency then I'd probably go with Cassandra.

There's a big caveat to any NoSQL database and that's how you handle aggregates/roll-ups. With a standard database it's easy to write these queries. If you do it without thinking on a NoSQL system it'll cost you in performance and where billed per access, money. There's a few ways to address this;

- batch ala map-reduce.

- streaming ala Apache Beam, Spark, etc.

- in query counting (aka sharded counters).

Re: Handling 1M Requests per Minute with Go (2015)

#34

Post by phillipcarter »

35bge57dtjku wrote: post #33

It's strange that they seem to think this is an amazing amount of scaling and then go on to reveal that it's actually with 20 servers. And it's not like their solution is novel.

833 requests per second per server (though it's not likely that the load is perfectly distributed like that).

I'm ... not very impressed.

Re: Handling 1M Requests per Minute with Go (2015)

#35

Post by [deleted] »

phillipcarter wrote: post #34
35bge57dtjku wrote: post #33

It's strange that they seem to think this is an amazing amount of scaling and then go on to reveal that it's actually with 20 servers. And it's not like their solution is novel.

833 requests per second per server (though it's not likely that the load is perfectly distributed like that). I'm ... not very impressed.

[deleted]

Re: Handling 1M Requests per Minute with Go (2015)

#36

Post by teh_klev »

35bge57dtjku wrote: post #33

It's strange that they seem to think this is an amazing amount of scaling and then go on to reveal that it's actually with 20 servers. And it's not like their solution is novel.

Well actually... :)

If you read to the end:

"After we had properly configured our cluster and the auto-scaling settings, we were able to lower it even more to only 4x EC2 c4.Large instances and the Elastic Auto-Scaling set to spawn a new instance if CPU goes above 90% for 5 minutes straight."

From last time's discussion on this (https://news.ycombinator.com/item?id=9845820):

This comment is kind of insightful:

https://news.ycombinator.com/item?id=9846606

"I was confused by the numbers at first, so: that's 17k requests per second, spread out over 4 dual core Xeon (Haswell) machines, which works out to just over 4000 requests/s per machine. It's still a respectable number, but it's much closer to what one would expect given the task."

"Don't get me wrong, the most interesting part is definitely the implementation and as a Go noob I found it very useful - it's just a bit misleading for the headline to sum your request rate across all parallelized machines."

Re: Handling 1M Requests per Minute with Go (2015)

#37

Post by meritt »

natbobc wrote: post #32
meritt wrote: post #28

Earlier quoted context omitted.

Fair point, I was mostly focused on the absurd cost for that specific implementation. What would you suggest as an alternative? A document-oriented database?

If you're on AWS I would probably go with DynamoDB, if you're on GCP Datastore. They aren't drop-in replacements for one another but the way you architect your system will be similar(ish). The main benefit is that it'll cost less upfront and require less to manage. Now that AWS have simplified back-ups it's a pretty simple system to operate. If you're looking for better controls over latency then I'd probably go with…

I was looking at the DynamoDB costs just to get an idea and I think it'd actually be a lot worse for this scenario:

- Up to 1M writes per minute, or ~16k/second. Obviously they don't operate at full capacity 24x7 but let's assume they do for worst-case planning. That means they need 16k WCU, or $7500/mo.

- Their RCU would obviously be significantly lower, let's say just $500/mo for analysis.

- Storage is going to cost $0.25 per GB-month. Let's say a very small payload of 100 bytes per object + 100 bytes for Indexed overhead. 200 bytes * 1M/minute * 1440 minutes/day = +288 GB/day. By the end of month #1, you're looking at $2,160/mo. By year #1 (105TB) you're spending $26k/mo.

At the end of the day, they're going to need to massively scale down their write throughput and constantly perform rollups/aggregations or they'll go broke if they ever want to simply store the raw/original data using S3 or DynamoDB.

Cost analyses like these make me very hesitant to use S3 or DynamoDB for an operation like in this article. Sure, individual operations are a fraction of a penny, but it grows very quickly.

It seems like it'd be far more sensible to just get a few EC2 instances and run something (cassanda, sqlite, spark, hadoop) locally.

Re: Handling 1M Requests per Minute with Go (2015)

#38

Post by kureikain »

Last year I wrote a Go service handle JSON posting with payload of ~ 100KB and insert them into MongoDB. It requires trivial amount of work and on a single t2.small server it can handle up to 200K RPM just fine. So this number 1M request on 4x c4.Large mean 250K RPM on a single c4.large. Maybe their payload is too big and writing data to S3 is probably slower than to a MongoDB cluster.

I was so happy at Go performance and since that day I realize just how slow other language is. I found Java is quite comparable to Go as well. I was coming from Ruby/PHP and I didn't realize how fast other languages are until I tried it.

Re: Handling 1M Requests per Minute with Go (2015)

#39

Post by natbobc »

meritt wrote: post #37
natbobc wrote: post #32

Earlier quoted context omitted.

If you're on AWS I would probably go with DynamoDB, if you're on GCP Datastore. They aren't drop-in replacements for one another but the way you architect your system will be similar(ish). The main benefit is that it'll cost less upfront and require less to manage. Now that AWS have simplified back-ups it's a pretty simple system to operate. If you're looking for better controls over latency then I'd probably go with…

I was looking at the DynamoDB costs just to get an idea and I think it'd actually be a lot worse for this scenario: - Up to 1M writes per minute, or ~16k/second. Obviously they don't operate at full capacity 24x7 but let's assume they do for worst-case planning. That means they need 16k WCU, or $7500/mo. - Their RCU would obviously be significantly lower, let's say just $500/mo for analysis. - Storage is going to cos…

I don't think it's a straight-forward decision and is influenced by a number of things;

1. desired time to market.

2. operating cost.

3. people cost.

If you're dealing at that level of traffic you're not going to go broke unless you lack a decent business model. One of the companies I worked at was spending $100,000+ per month on AWS (more than they should've for the incoming traffic level but they survived for a number of years). A friend was working at another company that was spending $3,000,000+/month. The motivation for using DynamoDB/Datastore over say S3 is that you can do range queries, field sub-selection, offloading of systems administration, some elements of security, etc.

Most traffic patterns have windows and tend to be diurnal in nature. This depends on the number of timezones your service covers. Obvious exception being IoT type time-series. So you might experience peaks of 16k/s but that will be for perhaps a few windows in the day. I also wouldn't be treating DynamoDB as permanent "long-term" storage for granular data. I would be using a sliding window of whatever I consider to be a valuable retention period (e.g. sliding window of 30 days). If you don't you'd need to scale C* as well.

In order to run Cassandra and get streaming aggregation via Spark you need to run "2 regions". One as the primary online region for serving requests and a second region for stream processing using Spark. Minimum recommended memory requirement is 16GB for a C* node. For a C* + Spark node minimum is 32GB. The more the better for both. If you've selected your keys well C* will scale almost linearly with the addition of nodes. Using Netflix 1,000,000 writes/second as a model you could roughly state that dividing nodes (285) by 60 should yield an approximately similar throughput requirement to the articles. So about 5-6 t2.xlarge nodes could potentially work for the FE cluster, and whatever gives you adequate streaming performance for the streaming cluster, as a minimum 3-5 x t2.2xlarge. Let's say ~$2,000/month.

Another thing to consider is that the management and maintenance of a C* cluster doesn't come for free either. So figure you'll need a minimum of 2 sys-admins to administer the cluster. Depending on where you are that'll likely cost the company upwards of $400,000 per year or $33,000k/month once you combine base salary, pension contributions, taxes, etc. You could say 1 sysadmin but then you're going to burn out your sysadmins if 1 guy is always on-call.

So assuming a sliding window of 30 days that's $2,160/day for DynamoDB. With a C* cluster you're looking at a minimum of $1,100/day. More depending on how well it performs for what your system is doing. If you're not using a sliding window you'd also need to scale you're C* cluster as well. In order to configure the C* cluster in a way that provides back-ups, auto-recovery, etc you're probably looking at about 2w-2m depending on experience. Whereas DynamoDB time to market will be probably a 1-2w.

Post reply on HN