Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Handling 1M Requests per Minute with Go

marcio.io

Post reply on HN
Print view View original

81–90 of 109 posts

Re: Handling 1M Requests per Minute with Go

#81

Post by vidarh »

Ao7bei3s wrote: post #13
fasteo wrote: post #6

>> But since the beginning, our team knew that we should do this in Go because during the discussion phases we saw this could be potentially a very large traffic system I don't get this reasoning.

Go is 1-2 orders of magnitude faster than Ruby, and much easier to write concurrent code in. It makes perfect sense to me. What would you have recommended them, for a reasonably-high-performance server implementation? (Please don't say C.) https://benchmarksgame.alioth.debian.org/u64q/benchmark.php?...

> Go is 1-2 orders of magnitude faster than Ruby

That may be true for some types of code, but for an app that's predominantly shuffling data over the network you should be spending most of the time in kernel space executing syscalls, and then language differences are largely irrelevant.

> and much easier to write concurrent code in.

How? Writing concurrent code in Ruby is trivial since 1.9.x (prior to 1.9 you had to battle the green threads in MRI for some stuff), which isn't exactly new.

Re: Handling 1M Requests per Minute with Go

#82

Post by nindalf »

meir_yanovich wrote: post #78

Can you please explain why to use go and not c++/c forget about language syntax / compilation complexity. say i know both very well , now why to go with "GO" ? thanks

There are a few benefits of Go I can think of

* Fewer lines of code, fewer gotchas and hence easier to reason about and maintain.

* Powerful concurrency primitives (channels, select) built right into the language, rather than a library. A scalable producer-consumer implementation would probably be 100 lines of Go code.

* If your application isn't too latency sensitive (game server, frequency trading etc) then the GC simplifies matters. Its guaranteed to run for a maximum of 10ms out of every 50ms which is good enough for most applications. (but typically runs for around 1ms)

* Some of the tooling around the language is great. There are some great articles (I remember one posted to HN yesterday) about how people wrangled a lot more performance out of their code using the profile tool, for instance.

* Miscellaneous goodies like testing out of the box, an extensive standard library and being able to compile in 1/10th of the time.

An example of a service migrated from C++ to Go - dl.google.com - http://talks.golang.org/2013/oscon-dl.slide#1

These are the benefits I could think of if a programmer knows C++ and Go equally well. However, suppose he has to work with fellow programmers who aren't comfortable with either, Go would be a superior choice. It would take a week to learn most of Go and perhaps a month to grok it. I think C++ takes much, much longer than that to learn properly.

Re: Handling 1M Requests per Minute with Go

#83

Post by vidarh »

fasteo wrote: post #6

>> But since the beginning, our team knew that we should do this in Go because during the discussion phases we saw this could be potentially a very large traffic system I don't get this reasoning.

They're unfamiliar with network programming, and so picked a language that seems fast to them, is my assessment.

Based on their description, they should be IO bound. If they're IO bound, their system should spend the vast majority of time in the kernel executing system calls.

If they do that, then whichever language they use has very little relevance to whether or not their system ends up being fast.

EDIT: As an example, my first ever production Ruby app was a messaging middleware server that processed about 3 million messages a day (34/sec) on 10% of a single mid-range 2005-era Xeon core. It was totally unoptimized, and used select() instead of the more modern, faster alternatives. Also, only about 10% of that time was spent in user space. 90% of it was in the kernel, executing system calls, so most meaningful optimization would be in improving the usage of system calls that are exactly the same across languages.

At the time, going beyond a single core required multiple instances, as Ruby lacked native threads back then. But processing 500-600/sec across 2-3 processes on a dual core 2005-era Xeon would not have been challenging even then with that quite naive and dated approach.

As someone else pointed out, they're doing ~4k/sec on a modern dual-core Xeon. Based on the above I'd expect it to be fairly easy to match their ~4K/sec with Ruby today.

Re: Handling 1M Requests per Minute with Go

#85

Post by vidarh »

fixxer wrote: post #12
mstump wrote: post #9

16k requests per second isn't fast.

16k requests per second is not fast if you're talking about fetching a page or doing a minimal amount of I/O. 16k requests per second is worth writing about if you're talking about a process with substantial side effects (S3 I/O).

If the language implementation is an issue when doing this, you're doing something wrong.

I'm working on a contract doing this exact thing at the moment, and unsurprisingly the Ruby code that's in the picture is responsible for just a tiny little fraction of the overall latency.

You should spend most of the time waiting on S3, pretty much no matter what language you choose. If you don't, it's not the fault of whichever language you use.

In fact, S3 performance is so dominant in this type of scenario if you do things properly, that if you want to optimize for speed and can afford to risk data loss (or in our case, the clients poll for confirmation and re-upload data in the rare event of loss), it's generally proving substantially better to write to local disk and do uploading to S3 in the background.

Re: Handling 1M Requests per Minute with Go

#87

Post by sinzone »

fasteo wrote: post #51
Ao7bei3s wrote: post #13

Earlier quoted context omitted.

Go is 1-2 orders of magnitude faster than Ruby, and much easier to write concurrent code in. It makes perfect sense to me. What would you have recommended them, for a reasonably-high-performance server implementation? (Please don't say C.) https://benchmarksgame.alioth.debian.org/u64q/benchmark.php?...

>>> It makes perfect sense to me Yes. Other than some very basic scripts, I have never worked with Ruby, so I was not aware that it is so slow. Thanks for pointing that out. As for my recommendation, it is pretty standard worker architecture: Their system seems to be an ingesting-only system, that is, the clients are getting an empty HTTP 200 OK response. Given this, I would put openresty (nginx) in the front, with s…

we use the same technology with KONG [1]

[1] https://github.com/mashape/kong

Re: Handling 1M Requests per Minute with Go

#88

Post by mstump »

fixxer wrote: post #68
mstump wrote: post #62

Earlier quoted context omitted.

No, it is just pass through, they just did a crappy job of handling concurrency.

Maybe they should hire somebody smart like you.

To be snide, maybe they should, I am an expert in this topic this is what I do all day every day.

They're doing development without understanding how computers work, where the bottlenecks are, or what the maximum theoretical throughput for the use-case is. They ended up with something slightly better than the horrible situation they were in, and are celebrating a inefficient solution as a technical triumph.

Re: Handling 1M Requests per Minute with Go

#89

Post by threeseed »

darksaints wrote: post #28

Does anybody here have any experience with Go's garbage collection pauses with large stack sizes? I've got a scala app that regularly consumes about 48G of ram, and I'm very happy with the response times during heavy loads like this, but the P99.5 is abysmal because of garbage collection. I've tried tuning it, but it doesn't seem like anything I do helps. I'll probably end up using an Azul JVM but I'm curious how oth…

For a JVM heap size greater than 32GB you should be using G1GC.

Tuning GC for Spark: https://www.youtube.com/watch?v=drmJDISLkf4

In the Big Data space we have dozens of machines all with very large stack sizes (I run mine with 250GB) and don't run into any major stop the world pauses.

Re: Handling 1M Requests per Minute with Go

#90

Post by fixxer »

mstump wrote: post #88
fixxer wrote: post #68

Earlier quoted context omitted.

Maybe they should hire somebody smart like you.

To be snide, maybe they should, I am an expert in this topic this is what I do all day every day. They're doing development without understanding how computers work, where the bottlenecks are, or what the maximum theoretical throughput for the use-case is. They ended up with something slightly better than the horrible situation they were in, and are celebrating a inefficient solution as a technical triumph.

> inefficient solution as a technical triumph

They were able to solve their problem in a single process balanced over 4 boxes without ever having to hire someone like you, despite your expertise.

Could they have increased throughput? Absolutely. It would have involved a different architecture with more complexity & time, and it also would have relied on skills beyond what was immediately available. I'm guessing their line count is around ~200 for the core functionality.

Can you share some actual technical points where they made an error? I would really like to see you demonstrate expertise beyond these uninspiring generalities.

Post reply on HN