GitHub's Unicorn Setup • Hacker News Board

Re: GitHub's Unicorn Setup

#41

Post by mojombo » Sat, Oct 10, 2009, 3:45 AM UTC

antonovka wrote: post #39

joevandyk wrote: post #22

Earlier quoted context omitted.

The 16 worker processes only handle dynamic requests, not static. With proper http caching, ESI, etc, it's quite likely that amount amount of concurrency is more than enough for their needs. Say that when all the workers are being utilized, the average dynamic request takes 50ms (which is on the high end). That means that each box can handle 320 dynamic requests per second. Which is a decent amount, and I'd be surpri…

If they are using http caching and ESI properly, the average request time would be significantly lower, and the requests per second would probably be > 500/s on that box. 500 req/s is pretty abysmal for a 8 core 16 gigabyte server. I realize that some of that has to do with Ruby performance, but yikes -- that is frighteningly bad scaling, regardless of the cause.

A number like 500 req/s is irrelevant without an understanding of what the site has to do. For a simple site that does simple database lookups, I'd expect more than that. For a site like GitHub that has to do database lookups and pull large amounts of data from Git repositories, it's an entirely different story. Without knowing the split between cache hits and misses, the number is even more irrelevant. You're arguing in abstracts, whereas I am constrained to actually working with real life. I'd love to see some examples of the type of sites you're running and the solutions you employ wherein 500 req/s of dynamic requests on an 8 core machine are considered "frighteningly bad scaling."

Re: GitHub's Unicorn Setup

#42

Post by [deleted] » Sat, Oct 10, 2009, 3:53 AM UTC

antonovka wrote: post #20

mojombo wrote: post #17

Earlier quoted context omitted.

Each box can handle plenty more than 16 workers, but we currently don't need that many, so it's pointless to run them. Our frontend machines generally sit about 60% idle. We wanted to have plenty of headroom on our new setup. Also, our Ruby application code spends very little time blocking on external resources (compared to time spent in Ruby execution), so having async execution on these wouldn't give us that much m…

Each box can handle plenty more than 16 workers, but we currently don't need that many, so it's pointless to run them. Our frontend machines generally sit about 60% idle. We wanted to have plenty of headroom on our new setup. At those numbers, it sounds like you'd max out at roughly 32 workers (so 32 concurrent requests), or am I missing something important? It's true that running Ruby code is generally more expensiv…

[deleted]

Re: GitHub's Unicorn Setup

#43

Post by jasonwatkinspdx » Sat, Oct 10, 2009, 4:26 AM UTC

blasdel wrote: post #12

I don't understand why people insist on architectures where otherwise-independent processes share a single socket. You're already running a reverse proxy in front of them! There's no reason each Unicorn couldn't be listening on a different port. Does that third layer of local load-balancing between the HTTP proxy and the event-driven app server actually get you anything?

Sharing queues wins. It's as simple as that.

It's a general result in queue theory that you want one global queue as early as possible in the system. This is because requests don't take exactly the average service time to clear backends: it's a distribution. The more workers that can pull from a queue, the less the worst case service times affect the average service time.

One queue per worker with no global queue is the worst configuration and should be avoided if at all possible. Anyone who's run large reverse proxy installs knows this pain well.

The ideal system would be for the balancer machine(s) to hold the requests, and for backends to pull them in a sort of ping/pong fashion. I gather fuzed runs in a pattern like this, though I've not used it.

Telcom folks have analyzed this stuff in detail for the better part of a century. There's a lot of theory out there and it's surprisingly practical and applicable to real world web applications.

Re: GitHub's Unicorn Setup

#44

Post by chadr » Sat, Oct 10, 2009, 4:29 AM UTC

caseyf wrote: post #36

fizx wrote: post #32

Earlier quoted context omitted.

Sites do tend to lag for at least several seconds after the restart though.

Yeah. We load balance across 5 Apache/Passengers and I do rolling deploys (all in Capistrano) by removing a Passenger from load balancing, updating the app, restarting Apache, and adding it back into load balancing with a 10 second delay between each. We tried the Passenger touch restart.txt and that didn't go well at all when we were under load.

Have you tried contacting the Passenger developers about this? I won't be surprised if they come out with a fix for this.

Re: GitHub's Unicorn Setup

#45

Post by jasonwatkinspdx » Sat, Oct 10, 2009, 4:30 AM UTC

wmf wrote: post #13

blasdel wrote: post #12

I don't understand why people insist on architectures where otherwise-independent processes share a single socket. You're already running a reverse proxy in front of them! There's no reason each Unicorn couldn't be listening on a different port. Does that third layer of local load-balancing between the HTTP proxy and the event-driven app server actually get you anything?

Replacing N ports with one simplifies configuration. I never understood the complex HAProxy in front of Apache in front of Nginx in front of Mongrel type setups that seem to be popular in the Rails world. Why not just use Unicorn? What value is GitHub getting from having Nginx in front?

Because Ruby 1.8 threading sucks you pay a large memory price (ie a process) for each concurrent request in flight. A fronting proxy allows your backends to write out the response as fast as possible and move on to another request while the proxy spoon feeds the response to slow clients.

Also, nginx is going to be more efficient for serving static files, though most larger apps will have broken such requests out to a separate set of domains likely serviced by a cdn.

Re: GitHub's Unicorn Setup

#46

Post by grandalf » Sat, Oct 10, 2009, 4:51 AM UTC

mrshoe wrote: post #21

mojombo wrote: post #18

Earlier quoted context omitted.

Why is that shocking? Can you elaborate?

Sure, now that HN has decided that enough time has elapsed to allow me to reply. :-) Actually, antonovka did an excellent job explaining it above. The most important aspect is that threads are "lighter weight" than processes. They use less memory and are quicker to context switch (usually). The result of this lighter weight is that you can spawn more threads than you could processes, on the same hardware. And when yo…

I think you missed the point. The point was not that github stopped using haproxy, but that it no longer needs to use it.

Assuming haproxy can work with unix sockets, github could configure nginx/unicorn to use it, but with unicorn its current work load does not require additional load balancing acrobatics beyond what the unix socket does.

Similarly, many rails sites use thin/nginx or mongrel/nginx and do not even have a workload that necessitates haproxy.

Plus, haproxy adds another layer of complexity, configuration, and management, which is nice to avoid if you can.

Re: GitHub's Unicorn Setup

#47

Post by boundlessdreamz » Sat, Oct 10, 2009, 4:55 AM UTC

caseyf wrote: post #36

fizx wrote: post #32

Earlier quoted context omitted.

Sites do tend to lag for at least several seconds after the restart though.

Yeah. We load balance across 5 Apache/Passengers and I do rolling deploys (all in Capistrano) by removing a Passenger from load balancing, updating the app, restarting Apache, and adding it back into load balancing with a 10 second delay between each. We tried the Passenger touch restart.txt and that didn't go well at all when we were under load.

Can you share your setup please?

Re: GitHub's Unicorn Setup

#48

Post by davidw » Sat, Oct 10, 2009, 5:34 AM UTC

mrshoe wrote: post #21

mojombo wrote: post #18

Earlier quoted context omitted.

Why is that shocking? Can you elaborate?

Sure, now that HN has decided that enough time has elapsed to allow me to reply. :-) Actually, antonovka did an excellent job explaining it above. The most important aspect is that threads are "lighter weight" than processes. They use less memory and are quicker to context switch (usually). The result of this lighter weight is that you can spawn more threads than you could processes, on the same hardware. And when yo…

Switching to threads from processes is probably "premature optimization" for Ruby - it's just not going to buy you all that much.

Re: GitHub's Unicorn Setup

#49

Post by davidw » Sat, Oct 10, 2009, 5:36 AM UTC

seiji wrote: post #4

dschobel wrote: post #3

When the Unicorn master starts, it loads our app into memory. As soon as it’s ready to serve requests it forks 16 workers. Those workers then select() on the socket, only serving requests they’re capable of handling. In this way the kernel handles the load balancing for us. Wasn't there a heated debate here just the other day about the prefork model? Guess it's at least back en vogue @github.

It's connection pooling, not a fork-per-accept web server. Each worker is select/epoll/kqueue'ing for individual requests.

Except that Apache (and Passenger) don't just start up N worker processes - they can dynamically allocate them as the need arises. Perhaps this is less important for a site like github, which can plan exactly how many to use, but if you have more than one site on the same server, it's really convenient, and, IMO, makes Passenger the best choice for deploying Rails for most people.

So it appears as if this setup replicates a small portion of what Apache does without all the bells and whistles.

Re: GitHub's Unicorn Setup

#50

Post by davidw » Sat, Oct 10, 2009, 5:41 AM UTC

antonovka wrote: post #9

aaronblohowiak wrote: post #6

Earlier quoted context omitted.

Right, but it is being handled by processes and not threads, which the argument was about the other day.

The argument encompassed two aspects of the model: 1) "Threads are out -- processes are better than threads." 2) Process-per-connection architectures. The first is demonstrably false -- for instance, look at Erlang, which maps lightweight erlang processes to operating system threads, providing SMP scalability at a low cost without running into Github's issues with mongrel "thread-killing". More broadly used, look at…

Erlang is not really an argument for threads.

The Erlang model is to utilize poll/select/whatever together with a language that contains its own scheduler.

If you execute arbitrary python/ruby/tcl/php code, there's a chance it will block. If you execute arbitrary Erlang code, the chance of that is very low. This means that you can handle all kinds of long running calculations in the same OS process, while you continue to handle incoming requests.

The fact that Erlang farms out a few threads is a later optimization added to the language several years back in order to take advantage of SMP. However, it only creates a number of threads to match the number of processes, IIRC. After that it shouldn't spawn any more, so I don't think you would particularly call it an example of a 'threaded model'. Erlang's big advantage is the internal scheduler, and a systematic approach to writing code that will never block the Erlang OS process.