If this system is merely decoding JSON and writing payloads to S3 for asynchronous data processing, why not have your clients write directly to S3?
Handling 1M Requests per Minute with Go
11–20 of 109 posts
Re: Handling 1M Requests per Minute with Go
#1216k requests per second isn't fast.
16k requests per second is not fast if you're talking about fetching a page or doing a minimal amount of I/O.
16k requests per second is worth writing about if you're talking about a process with substantial side effects (S3 I/O).
Re: Handling 1M Requests per Minute with Go
#13>> But since the beginning, our team knew that we should do this in Go because during the discussion phases we saw this could be potentially a very large traffic system I don't get this reasoning.
Go is 1-2 orders of magnitude faster than Ruby, and much easier to write concurrent code in.
It makes perfect sense to me. What would you have recommended them, for a reasonably-high-performance server implementation? (Please don't say C.)
https://benchmarksgame.alioth.debian.org/u64q/benchmark.php?...
Re: Handling 1M Requests per Minute with Go
#1416k requests per second isn't fast.
16k requests per second is not fast if you're talking about fetching a page or doing a minimal amount of I/O. 16k requests per second is worth writing about if you're talking about a process with substantial side effects (S3 I/O).
It's just pass through. The limit in this instance is probably packets per second of the legacy AWS network. Why is this difficult?
Re: Handling 1M Requests per Minute with Go
#15What's the trick to resubmit without getting stuck going to the first post? For example, I submitted this story 2 hours before this one:
https://news.ycombinator.com/item?id=9844826
In the past, when I try to submit a story, even if it's a couple days old, the submission is ignored, and my vote is added to the original.
Re: Handling 1M Requests per Minute with Go
#16What's the trick to resubmit without getting stuck going to the first post? For example, I submitted this story 2 hours before this one: https://news.ycombinator.com/item?id=9844826 In the past, when I try to submit a story, even if it's a couple days old, the submission is ignored, and my vote is added to the original.
The slash at the end of the URL made the difference in this case.
Re: Handling 1M Requests per Minute with Go
#17>> But since the beginning, our team knew that we should do this in Go because during the discussion phases we saw this could be potentially a very large traffic system I don't get this reasoning.
Go is 1-2 orders of magnitude faster than Ruby, and much easier to write concurrent code in. It makes perfect sense to me. What would you have recommended them, for a reasonably-high-performance server implementation? (Please don't say C.) https://benchmarksgame.alioth.debian.org/u64q/benchmark.php?...
Lua would beat go every day of the week.
Re: Handling 1M Requests per Minute with Go
#18>> But since the beginning, our team knew that we should do this in Go because during the discussion phases we saw this could be potentially a very large traffic system I don't get this reasoning.
It's less about Go being incredibly fast than about Ruby being anomalously slow. It makes sense if you start from the presumption that Go is the only "fast" language they're considering.
Re: Handling 1M Requests per Minute with Go
#19Earlier quoted context omitted.
16k requests per second is not fast if you're talking about fetching a page or doing a minimal amount of I/O. 16k requests per second is worth writing about if you're talking about a process with substantial side effects (S3 I/O).
It's just pass through. The limit in this instance is probably packets per second of the legacy AWS network. Why is this difficult?
Read the analysis. First attempt was "just a pass through".
Re: Handling 1M Requests per Minute with Go
#20>> But since the beginning, our team knew that we should do this in Go because during the discussion phases we saw this could be potentially a very large traffic system I don't get this reasoning.
It's less about Go being incredibly fast than about Ruby being anomalously slow. It makes sense if you start from the presumption that Go is the only "fast" language they're considering.
But that would prevent the knee-jerk tribal language war response.