Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub's Unicorn Setup

github.com

Post reply on HN
Print view View original

31–40 of 64 posts

Re: GitHub's Unicorn Setup

#31

Post by joevandyk »

antonovka wrote: post #27
joevandyk wrote: post #25

Earlier quoted context omitted.

The github developers are great ruby developers, and I'd be very surprised if they would be as productive in the short term in a different language. I do not see a business case for switching platforms for them. You seem hung up on the number of concurrent requests. An 8 core machine can only do 8 things at once, no matter if you are using threads or processes.

You seem hung up on the number of concurrent requests. An 8 core machine can only do 8 things at once, no matter if you are using threads or processes. You can't assume that the threads are 100% CPU bound. If they were 100% CPU bound, you would have a point -- 8 cores, 8 CPU-bound processes, no processing time left over -- that is where event based and hybrid thread/event-based architectures excel. In reality, webapp…

I think we all understand the difference between threads and processes and where context switching fits in.

In Linux 2.6, there's not a lot of difference between switching between processes and threads.

From the github dude: "Our Ruby application code spends very little time blocking on external resources." That is why using threads in this particular case won't help.

Re: GitHub's Unicorn Setup

#32

Post by fizx »

latortuga wrote: post #10
joevandyk wrote: post #8

Earlier quoted context omitted.

You run "touch #{RAILS_ROOT}/tmp/restart.txt". That will restart the rails processes. No connections are dropped, and I've not seen any downtime.

To clarify, my understanding is that while rails is restarting, Passenger will queue all the requests that come in and begin processing them as soon as rails is ready. But yeah, zero downtime, it's pretty awesome.

Sites do tend to lag for at least several seconds after the restart though.

Re: GitHub's Unicorn Setup

#33

Post by antonovka »

mbrubeck wrote: post #29
antonovka wrote: post #9

Earlier quoted context omitted.

The argument encompassed two aspects of the model: 1) "Threads are out -- processes are better than threads." 2) Process-per-connection architectures. The first is demonstrably false -- for instance, look at Erlang, which maps lightweight erlang processes to operating system threads, providing SMP scalability at a low cost without running into Github's issues with mongrel "thread-killing". More broadly used, look at…

"Look at Erlang, which maps lightweight erlang processes to operating system threads, providing SMP scalability at a low cost..." Erlang can use native threads, but they do not map to its lightweight processes. Erlang's lightweight processes are basically green threads. Native threads are used for SMP scheduling only. So on a four-core system, you might have thousands of Erlang lightweight processes running on just f…

Erlang can use native threads, but they do not map to its lightweight processes.

That's not what I said; I said: "[Erlang] maps lightweight erlang processes to operating system threads."

Erlang processes are green threads mapped to OS threads. They're called processes in erlang, but they're M:N scheduled threads.

So on a four-core system, you might have one thousands of Erlang lightweight processes running on just four OS threads in one OS process.

Or you could have thousands of Erlang lightweight processes mapped to 185 OS threads (not that it will, but it could). The fact that they're green threads is irrelevant in terms of scalability -- they're still threads, but even less resource intensive than a direct mapping to OS threads. M:N scheduled green threading is also very hard to do in the general case, which is why you don't see more of it (see the abandoned attempts to implement generic M:N OS:green thread scheduling in FreeBSD, for instance).

You could also be using Scala, where an actor only maps to a thread when it's blocked inside of its react loop, but in that case will consume a whole thread.

What's your point? :)

Re: GitHub's Unicorn Setup

#34

Post by antonovka »

joevandyk wrote: post #31
antonovka wrote: post #27

Earlier quoted context omitted.

You seem hung up on the number of concurrent requests. An 8 core machine can only do 8 things at once, no matter if you are using threads or processes. You can't assume that the threads are 100% CPU bound. If they were 100% CPU bound, you would have a point -- 8 cores, 8 CPU-bound processes, no processing time left over -- that is where event based and hybrid thread/event-based architectures excel. In reality, webapp…

I think we all understand the difference between threads and processes and where context switching fits in. In Linux 2.6, there's not a lot of difference between switching between processes and threads. From the github dude: "Our Ruby application code spends very little time blocking on external resources." That is why using threads in this particular case won't help.

I think we all understand the difference between threads and processes and where context switching fits in.

You might, but I don't think that's the general case.

From the github dude: "Our Ruby application code spends very little time blocking on external resources." That is why using threads in this particular case won't help.

It hits a database, loads content from memcached, and waits on HTTP requests to complete, does it not?

I'd be surprised if the usage modeled a pure event-driven sendfile() based static-only file server (such as lighttpd), for instance (where threads vs. processes is moot).

Moreover, a fair amount of effort has been expended on a complex architecture to push only a very specific type of requests to Unicorn.

Re: GitHub's Unicorn Setup

#35

Post by chadr »

fizx wrote: post #30
atambo wrote: post #28

Is there any benefit in using unicorn over passenger?

Unicorn doesn't make your site very slow for the 5 seconds-2 minutes (YMMV) after you deploy.

It seems to me like the Passenger guys could easily add an option so that on a "touch tmp/restart.txt" a set of new worker processes is started before the old ones are killed off. I imagine this would make this slowness a thing of the past. For the record, my apps experiencing this momentary queueing and slowness on restart (5 seconds max).

Re: GitHub's Unicorn Setup

#36

Post by caseyf »

fizx wrote: post #32
latortuga wrote: post #10

Earlier quoted context omitted.

To clarify, my understanding is that while rails is restarting, Passenger will queue all the requests that come in and begin processing them as soon as rails is ready. But yeah, zero downtime, it's pretty awesome.

Sites do tend to lag for at least several seconds after the restart though.

Yeah. We load balance across 5 Apache/Passengers and I do rolling deploys (all in Capistrano) by removing a Passenger from load balancing, updating the app, restarting Apache, and adding it back into load balancing with a 10 second delay between each. We tried the Passenger touch restart.txt and that didn't go well at all when we were under load.

Re: GitHub's Unicorn Setup

#38

Post by mbrubeck »

antonovka wrote: post #33
mbrubeck wrote: post #29

Earlier quoted context omitted.

"Look at Erlang, which maps lightweight erlang processes to operating system threads, providing SMP scalability at a low cost..." Erlang can use native threads, but they do not map to its lightweight processes. Erlang's lightweight processes are basically green threads. Native threads are used for SMP scheduling only. So on a four-core system, you might have thousands of Erlang lightweight processes running on just f…

Erlang can use native threads, but they do not map to its lightweight processes. That's not what I said; I said: "[Erlang] maps lightweight erlang processes to operating system threads." Erlang processes are green threads mapped to OS threads. They're called processes in erlang, but they're M:N scheduled threads. So on a four-core system, you might have one thousands of Erlang lightweight processes running on just fo…

Sorry, I misunderstood what you were saying. You are right.

Re: GitHub's Unicorn Setup

#39

Post by antonovka »

joevandyk wrote: post #22
antonovka wrote: post #9

Earlier quoted context omitted.

The argument encompassed two aspects of the model: 1) "Threads are out -- processes are better than threads." 2) Process-per-connection architectures. The first is demonstrably false -- for instance, look at Erlang, which maps lightweight erlang processes to operating system threads, providing SMP scalability at a low cost without running into Github's issues with mongrel "thread-killing". More broadly used, look at…

The 16 worker processes only handle dynamic requests, not static. With proper http caching, ESI, etc, it's quite likely that amount amount of concurrency is more than enough for their needs. Say that when all the workers are being utilized, the average dynamic request takes 50ms (which is on the high end). That means that each box can handle 320 dynamic requests per second. Which is a decent amount, and I'd be surpri…

If they are using http caching and ESI properly, the average request time would be significantly lower, and the requests per second would probably be > 500/s on that box.

500 req/s is pretty abysmal for a 8 core 16 gigabyte server. I realize that some of that has to do with Ruby performance, but yikes -- that is frighteningly bad scaling, regardless of the cause.

Re: GitHub's Unicorn Setup

#40

Post by blasdel »

defunkt wrote: post #15
blasdel wrote: post #12

I don't understand why people insist on architectures where otherwise-independent processes share a single socket. You're already running a reverse proxy in front of them! There's no reason each Unicorn couldn't be listening on a different port. Does that third layer of local load-balancing between the HTTP proxy and the event-driven app server actually get you anything?

If each Unicorn worker listened on a different port, we'd have to use one of nginx's load balancing strategies (unless I'm misreading your comment). We have not had success with them in the past which is why we employed HAProxy with mongrel.

You read me correctly, I don't see why load balancing should be taking place down at the app server level.

Aren't you going to have multiple machines each with their own blessing of unicorns? You're still going to have to use some kind of load balancer in front of independent sockets.

Post reply on HN