A bit of context:
Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails
421–430 of 437 posts
Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails
#422Earlier quoted context omitted.
I've built fairly large haproxy based infrastructures, thank you very much. Doing this is not particularly challenging. Actually what I'd probably do for a setup like this would be to balance by the Host: header, and simply have the second layer be a suitable set of haproxy instances balancing each by least connections. Immediately vastly better than random.
Haproxy doesn't support dynamic configurations as far as I know, which is a serious problem if you're letting lots of people add/change domains and scale backends up/down dynamically. A Heroku haproxy would probably need to be restarted multiple times a second due to config changes. Nginx can do dynamic backends with lua & redis, but it can't use the built-in upstream backend balancing/failover logic if you do.
While it doesn't support dynamic configurations, it does support hot reconfiguration (the new daemon signals the old processes to gracefully finish up and shut down), and reconfigures very rapidly. You still don't want to restart it multiple times a second, but you don't need to:
A two layer approach largely prevents this from being a problem. You can afford total overkill in terms of the number of haproxies as they're so lightweight - running a few hundred individual haproxy instances with separate configs even on a single box is no big deal.
The primaries would rarely need to change configs. You can route sets customers to specific sets of second layer backends with ACL's on short substrings of the hostname (e.g. two letter combinations), so that you know which set of backends each hostname you handle maps to, and then further balance on the full host header within that set to enable the second layer to balance on least-connections to get the desired effect.
That lets you "just" rewrite the configs and hot-reconfigure the subset of second layer proxies handling customers that falls in the same set on modifications. If your customer set is large enough, you "just" break out the frontend into a larger number of backends.
Frankly, part of the beauty of haproxy is that it is so light that you could probably afford a third layer - a static primary layer grouping customers into buckets, a dynamic second layer routing individual hostnames (requiring reconfiguration when adding/removing customers in that bucket) to a third layer of individual customer-specific haproxies.
So while you would restart some haproxy multiple times a second, the restarts could trivially be spread out over a large pool of individual instances.
Alternatively, "throwing together" a second or third layer using iptables either directly or via keepalived - which does let you do dynamic reconfiguration trivially, and also supportes least-connections load balancing - is also fairly easy.
But my point was not to advocate this as the best solution for somewhere like Heroku - it doesn't take a very large setup before a custom solution starts to pay off.
My point was merely that even with an off the shelf solution like haproxy, throwing together a workable solution that beats random balancing is not all that hard - there's a large number of viable solutions -, so there really is no excuse not to for someone building a PaaS.
Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails
#423Heroku Support Request #76070 To whom it may concern, We are long time users of Heroku and are big fans of the service. Heroku allows us to focus on application development. We recently read an article on HN entitled 'Heroku's Ugly Secret' http://s831.us/11IIoMF We have noticed similar behavior, namely increasing dynos does not provide performance increases we would expect. We continue to see wildly different perform…
Heroku's response:
Hi Steve,
I've been reading through all the concerns from customers, and I want every single customer to feel that Heroku is transparent and responsive. Our job at Heroku is to make you successful. Getting to the bottom of this situation and giving you a clear and transparent understanding of what we’re going to do to make it right is our top priority. I am committing to the community to provide more information as soon as possible, including a blog post on http://blog.heroku.com.
Oren Teich Heroku GM
Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails
#424This is Oren Teich, I run Heroku. I've read through the OP, and all of the comments here. Our job at Heroku is to make you successful and we want every single customer to feel that Heroku is transparent and responsive. Getting to the bottom of this situation and giving you a clear understanding of what we’re going to do to make it right is our top priority. I am committing to the community to provide more information…
I've posted an update on our blog, with another to follow tomorrow:
https://blog.heroku.com/archives/2013/2/15/bamboo_routing_pe...
Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails
#425Earlier quoted context omitted.
Thing is though, it says that a Dyno is a Ubuntu virtual machine. In what sort of horrendous configuration can an ENTIRE VM serve only a SINGLE REQUEST AT A TIME?! That is utter madness, and the validity of the argument depends on whether it's the Heroku or this dude's fault that the VM is serving only a single request at a time (and it taking >1sec to handle a request).
Not Heroku's fault in this case, Rails (and any other single-threaded environment) can handle a single request at a time.
But one VM can host multiple Rails instances, each on different port. That's what Passenger or Unicorn do, acting as proxy of group of locally spawn Rails instances.
Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails
#426Earlier quoted context omitted.
how does it decide to queue at the dyno level anyway? Does it check for connection refusal at the TCP level?
The connection is accepted, and a single-threaded web server will do the queuing.
Oh so the server process hosting rails is itself queueing? Is that what they refer to as "dyno queueing"? I thought perhaps there was another server between the router and your apps server process.
Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails
#427So the issue here is two-fold: - It's very hard to do 'intelligent routing' at scale. - Random routing plays poorly with request times with a really bad tail (median is 50ms, 99th is 3 seconds) The solution here is to figure out why your 99th is 3 seconds. Once you solve that, randomized routing won't hurt you anymore. You hit this exact same problem in a non-preemptive multi-tasking system (like gevent or golang).
> The solution here is to figure out why your 99th is 3 seconds. Once you solve that, randomized routing won't hurt you anymore. You hit this exact same problem in a non-preemptive multi-tasking system (like gevent or golang). The Golang runtime uses non-blocking I/O to get around this problem.
Wow, it's annoying to get modded down for saying something correct. Don't believe me? Run a golang program with strace -f sometime.
You could write a pthreads-compliant threading library without using threads at all, just epoll.
Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails
#428The initial response from GM of Heroku - https://blog.heroku.com/archives/2013/2/15/bamboo_routing_pe...
Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails
#429OP is a friend of mine, and when I first heard of his problem I wondered if there might be an analytical solution to quantify the difference between intelligent vs naive routing. I took this problem as an opportunity to teach myself a bit of Queueing Theory[1], which is a fascinating topic! I'm still very much a beginner, so bear with me and I'd love to get any feedback or suggestions for further study. For this exam…
OT: I went to the website, had my eyes assaulted with multiple font colours and sizes and left immediately without even trying to read.
People, there is a compromise between Google "brain dead" simplicity and MySpace pages "psycho" look, that is easy to read but still functional.
Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails
#430Earlier quoted context omitted.
You'd have some sort of error threshold/rate limit[1] at which point the server is marked dead and falls out of the backends list, surely? Or even, an alarm threshold if responses are averaging /too fast/, based on your expected load & response times. I've not done any deployment/ops beyond hte trivial/theoretical though, so I don't know how this would work in reality.
No, lbs don't inspect established streams. The lb will periodically send requests to a known uri as a health heck instead. The problem is when the health check uri isn't indicative of availabilty. (Hint: it never is) Nope, don't do this either. Unless you like getting pages because things are working?
You can configure a LB to inspect layer 7 - in HAProxy this is done with an observe directive. Error 500s would then take the member out of a pool. You are right that the health check of a static file may put it right back into the pool, but you can give it a slow rise value so that it waits a few minutes to do that. I'm not saying this is easy to get right but it is definitely possible to at least reduce the frequency of selection of a server raising 500 errors.