GitHub's Unicorn Setup • Hacker News Board

Re: GitHub's Unicorn Setup

#11

Post by defunkt » Fri, Oct 09, 2009, 9:21 PM UTC

cakeface wrote: post #5

Its great to see companies sharing what they've learned from experience about system architecture. So many times this sort of stuff is very difficult to plan out and the only real way to get it right is through experimentation. Having first hand descriptions like this is a great resource if you are setting something up the first time. I like that they posted their unicorn config file too!

Also it doubles as documentation for the members of our team who aren't familiar with this part of the system :)

Re: GitHub's Unicorn Setup

#12

Post by blasdel » Fri, Oct 09, 2009, 9:44 PM UTC

I don't understand why people insist on architectures where otherwise-independent processes share a single socket.

You're already running a reverse proxy in front of them! There's no reason each Unicorn couldn't be listening on a different port. Does that third layer of local load-balancing between the HTTP proxy and the event-driven app server actually get you anything?

Re: GitHub's Unicorn Setup

#13

Post by wmf » Fri, Oct 09, 2009, 9:53 PM UTC

blasdel wrote: post #12

I don't understand why people insist on architectures where otherwise-independent processes share a single socket. You're already running a reverse proxy in front of them! There's no reason each Unicorn couldn't be listening on a different port. Does that third layer of local load-balancing between the HTTP proxy and the event-driven app server actually get you anything?

Replacing N ports with one simplifies configuration.

I never understood the complex HAProxy in front of Apache in front of Nginx in front of Mongrel type setups that seem to be popular in the Rails world. Why not just use Unicorn? What value is GitHub getting from having Nginx in front?

Re: GitHub's Unicorn Setup

#14

Post by defunkt » Fri, Oct 09, 2009, 9:54 PM UTC

wmf wrote: post #13

blasdel wrote: post #12

I don't understand why people insist on architectures where otherwise-independent processes share a single socket. You're already running a reverse proxy in front of them! There's no reason each Unicorn couldn't be listening on a different port. Does that third layer of local load-balancing between the HTTP proxy and the event-driven app server actually get you anything?

Replacing N ports with one simplifies configuration. I never understood the complex HAProxy in front of Apache in front of Nginx in front of Mongrel type setups that seem to be popular in the Rails world. Why not just use Unicorn? What value is GitHub getting from having Nginx in front?

Unicorn is not for slow clients or static assets. That's what nginx is for. See http://unicorn.bogomips.org/PHILOSOPHY.html for info on Unicorn and slow clients.

nginx also has features like ESI, serving from memcached, and rate limiting which Unicorn does not (and doesn't need).

Re: GitHub's Unicorn Setup

#15

Post by defunkt » Fri, Oct 09, 2009, 9:55 PM UTC

blasdel wrote: post #12

I don't understand why people insist on architectures where otherwise-independent processes share a single socket. You're already running a reverse proxy in front of them! There's no reason each Unicorn couldn't be listening on a different port. Does that third layer of local load-balancing between the HTTP proxy and the event-driven app server actually get you anything?

If each Unicorn worker listened on a different port, we'd have to use one of nginx's load balancing strategies (unless I'm misreading your comment).

We have not had success with them in the past which is why we employed HAProxy with mongrel.

Re: GitHub's Unicorn Setup

#16

Post by mrshoe » Fri, Oct 09, 2009, 10:16 PM UTC

antonovka wrote: post #9

aaronblohowiak wrote: post #6

Earlier quoted context omitted.

Right, but it is being handled by processes and not threads, which the argument was about the other day.

The argument encompassed two aspects of the model: 1) "Threads are out -- processes are better than threads." 2) Process-per-connection architectures. The first is demonstrably false -- for instance, look at Erlang, which maps lightweight erlang processes to operating system threads, providing SMP scalability at a low cost without running into Github's issues with mongrel "thread-killing". More broadly used, look at…

Where were all you people in this thread the other day when I seemed to be the only one who understood the other side of this debate?!? :-P

I'm also very curious about the 16 workers per box model. Frankly, that github would ditch haproxy in favor of a pre-fork Ruby web server is shocking.

I'm just glad we have CherryPy in the Python world.

Re: GitHub's Unicorn Setup

#17

Post by mojombo » Fri, Oct 09, 2009, 10:31 PM UTC

antonovka wrote: post #9

aaronblohowiak wrote: post #6

Earlier quoted context omitted.

Right, but it is being handled by processes and not threads, which the argument was about the other day.

The argument encompassed two aspects of the model: 1) "Threads are out -- processes are better than threads." 2) Process-per-connection architectures. The first is demonstrably false -- for instance, look at Erlang, which maps lightweight erlang processes to operating system threads, providing SMP scalability at a low cost without running into Github's issues with mongrel "thread-killing". More broadly used, look at…

Each box can handle plenty more than 16 workers, but we currently don't need that many, so it's pointless to run them. Our frontend machines generally sit about 60% idle. We wanted to have plenty of headroom on our new setup.

Also, our Ruby application code spends very little time blocking on external resources (compared to time spent in Ruby execution), so having async execution on these wouldn't give us that much more efficiency. It's true that running Ruby code is generally more expensive (dollar-wise) per given task than other languages, but we made a conscious decision to make that tradeoff for the productivity gains that Ruby affords us. We use Erlang and EventMachine in other pieces of the architecture where they make sense.

The simple fact is that Unicorn works better for us (and our specific use case) than anything else we've tried, and so we're using it.

Re: GitHub's Unicorn Setup

#18

Post by mojombo » Fri, Oct 09, 2009, 10:33 PM UTC

mrshoe wrote: post #16

antonovka wrote: post #9

Earlier quoted context omitted.

The argument encompassed two aspects of the model: 1) "Threads are out -- processes are better than threads." 2) Process-per-connection architectures. The first is demonstrably false -- for instance, look at Erlang, which maps lightweight erlang processes to operating system threads, providing SMP scalability at a low cost without running into Github's issues with mongrel "thread-killing". More broadly used, look at…

Where were all you people in this thread the other day when I seemed to be the only one who understood the other side of this debate?!? :-P I'm also very curious about the 16 workers per box model. Frankly, that github would ditch haproxy in favor of a pre-fork Ruby web server is shocking. I'm just glad we have CherryPy in the Python world.

Why is that shocking? Can you elaborate?

Re: GitHub's Unicorn Setup

#19

Post by joevandyk » Fri, Oct 09, 2009, 10:51 PM UTC

latortuga wrote: post #10

joevandyk wrote: post #8

Earlier quoted context omitted.

You run "touch #{RAILS_ROOT}/tmp/restart.txt". That will restart the rails processes. No connections are dropped, and I've not seen any downtime.

To clarify, my understanding is that while rails is restarting, Passenger will queue all the requests that come in and begin processing them as soon as rails is ready. But yeah, zero downtime, it's pretty awesome.

that is correct.

Re: GitHub's Unicorn Setup

#20

Post by antonovka » Fri, Oct 09, 2009, 10:52 PM UTC

mojombo wrote: post #17

antonovka wrote: post #9

Earlier quoted context omitted.

The argument encompassed two aspects of the model: 1) "Threads are out -- processes are better than threads." 2) Process-per-connection architectures. The first is demonstrably false -- for instance, look at Erlang, which maps lightweight erlang processes to operating system threads, providing SMP scalability at a low cost without running into Github's issues with mongrel "thread-killing". More broadly used, look at…

Each box can handle plenty more than 16 workers, but we currently don't need that many, so it's pointless to run them. Our frontend machines generally sit about 60% idle. We wanted to have plenty of headroom on our new setup. Also, our Ruby application code spends very little time blocking on external resources (compared to time spent in Ruby execution), so having async execution on these wouldn't give us that much m…

Each box can handle plenty more than 16 workers, but we currently don't need that many, so it's pointless to run them. Our frontend machines generally sit about 60% idle. We wanted to have plenty of headroom on our new setup.

At those numbers, it sounds like you'd max out at roughly 32 workers (so 32 concurrent requests), or am I missing something important?

It's true that running Ruby code is generally more expensive (dollar-wise) per given task than other languages, but we made a conscious decision to make that tradeoff for the productivity gains that Ruby affords us.

It's really easy to measure performance, but productivity claims seem difficult to support and highly susceptible to confirmation bias.

I think that Ruby, Scala, and Clojure (just for instance) would stack up against each other very well in terms of productivity, but I wouldn't know how to prove it.