Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

rapgenius.com

Post reply on HN
Print view View original

281–290 of 437 posts

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#281

Post by pinars »

nsrivast wrote: post #262

OP is a friend of mine, and when I first heard of his problem I wondered if there might be an analytical solution to quantify the difference between intelligent vs naive routing. I took this problem as an opportunity to teach myself a bit of Queueing Theory[1], which is a fascinating topic! I'm still very much a beginner, so bear with me and I'd love to get any feedback or suggestions for further study. For this exam…

Distributed load balancing is a tough problem with two pieces to it. One is the queueing theory part.

The other is the systems side to it. If you have multiple customers and multiple checkout lines, and if your customers act independently without seeing the lines (no feedback from servers, network failures and delays, implementation complexity), what do you do?

It isn't a trivial problem. The easy route is paying Cisco's load balancers millions of dollars, but those only scale so far.

The bigger internet companies spend years of development time trying to make distributed load balancing work, but the issues there are a bit more complicated than a few customers walking to checkout lines.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#282

Post by bshanks »

jhspaybar wrote: post #62
aristus wrote: post #24

Earlier quoted context omitted.

I do perf work at Facebook, and over time I've become more and more convinced that the most crucial metric is the width of the latency histogram. Narrowing your latency band --even if it makes the average case worse -- makes so many systems problems better (top of the list: load balancing) it's not even funny.

I can chime in here that I have had similar experiences at another large scale place :). Some requests would take a second or more to complete with the vast majority finishing in under 100MS. A solution was put in place that added about 5 MS to the average request, but also crushed the long tail(it just doesn't even exist anymore) and everything is hugely more stable and responsive.

Let's assume that it is unacceptable to have each dyno tell the router each time it finishes a request ( http://news.ycombinator.com/item?id=5217157 ). And also that the goal is to reduce worst-case latency. And also that we don't know a priori how many requests each dyno should queue before it is 'full' and rejects further requests ( http://news.ycombinator.com/item?id=5216771 ).

Proposal:

1) Once per minute (or less often if you have a zillion dynos), each dyno tells the router the maximum number of requests it had queued at any time over the past minute.

2) Using that information, the router recalculates a threshold once a minute that defines how many queued requests is "too many" (e.g. maybe if you have n dynos, you take the log(n)th-busiest-dyno's load as the threshold -- you want the threshold to only catch the tail).

3) When each request is sent to a dyno, a header fields is added that tells the dyno the current 'too many' threshold.

4) If the receiving dyno has too many, it passes the request back to the router, telling the router that it's busy ( http://news.ycombinator.com/item?id=5217157 ). The 'busy' dyno remembers that the router thinks it is 'busy'. The next time its queue is empty, it tells the router "i'm not busy anymore" (and repeats this message once per minute until it receives another request, at which point it assumes the router 'heard').

5) When a receiving dyno tells the router that it is busy, the router remembers this and stops giving requests to that dyno until the dyno tells it that it is not busy anymore.

I haven't worked on stuff like this myself, do you think that would work?

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#283

Post by bad_user »

I noticed problems with Heroku's router too.

However, contrary to the author, I'm serving 25,000 real requests per second with only 8 dynos.

The app is written in Scala and runs on top of the JVM. And I was dissatisfied that 8 dynos seem like too much for an app that can serve over 10K requests per sec on my localhost.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#285

Post by aaron695 »

fatbird wrote: post #64
joevandyk wrote: post #55

Earlier quoted context omitted.

Nope, requests could still get queued behind a dyno that's busy with a long request.

Sure, but the real issue the article identifies is that, under random routing, they need to keep doubling the number of dynos to halve the odds of bad queueing, which leads to absurd factor of 50 requirements to get back to what they had before. With round robin, the increase should be much more linear.

I think you are right.

I had to create a quick sim but it does pan out.

With round robin it's going to chose the dynos most likely to have the shortest queue given the simple information available. (longest time since it got something) so it's biased to putting stuff on empty queues.

Where as random picks randomly, so there's no bias to empty queues, so random should be less efficient.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#286

Post by skrebbel »

themgt wrote: post #274
pdenya wrote: post #272

Earlier quoted context omitted.

We're discussing Rails on Heroku specifically which, non-unicorn, should be a "next available checkout counter" situation. Ideally it should be possible to make this an optional behavior that you can choose to turn on for Rails apps.

I agree there should be a better way - it's just important to understand than Rails doesn't get any special treatment on a PaaS done correctly, so it's important to come up with a generic solution. I think part of the solution would be customizable option(i.e.. how many requests can each dyno handle simultaneously), probably combined with intelligently monitoring/balancing proxy load so new requests always go to the…

> it's just important to understand than Rails doesn't get any special treatment on a PaaS done correctly

Why is it only "done correctly" if it does not account for specific properties of the technology used by a particular customer?

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#287

Post by vicks711 »

nsrivast wrote: post #262

OP is a friend of mine, and when I first heard of his problem I wondered if there might be an analytical solution to quantify the difference between intelligent vs naive routing. I took this problem as an opportunity to teach myself a bit of Queueing Theory[1], which is a fascinating topic! I'm still very much a beginner, so bear with me and I'd love to get any feedback or suggestions for further study. For this exam…

I cant stop myself from saying this. You wrote all this instead of doing what?

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#288

Post by jacques_chester »

jules wrote: post #147
jemfinch wrote: post #132

Earlier quoted context omitted.

This is a knee-jerk reply. I know, because my knee jerked as well. Think about the problem a little more: if you have the data necessary to pick the min-of-two, then you have the data you need to do intelligent routing.

Not necessarily. Heroku claims that a global request queue is hard to scale, and therefore they switched to random load balancing. The comment above shows that a global request queue is not necessary. Lets say that minimum-of-n scales up to 10 dynos. If your application requires 40 dynos, you can have one front load balancer which dispatches the requests to 4 back load balancers, each of which has 10 dynos assigned t…

> Heroku claims that a global request queue is hard to scale, and therefore they switched to random load balancing.

I wish Heroku would tell us more about what they tried. I can imagine a few cock-a-mimie schemes off the top of my head; it would be good to know whether they thought of those.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#289

Post by krunaldo »

gleb wrote: post #168
nostrademons wrote: post #136

Earlier quoted context omitted.

Jeff Dean wrote a paper on it for CACM: http://cacm.acm.org/magazines/2013/2/160173-the-tail-at-scal... There's a relatively easy fix for Heroku. They should do random routing with a backup second request sent if the first request times fails to respond after a relatively short period of time (say, 95th percentile latency), killing any outstanding requests when the first response comes back in. The amount of bookkeep…

From experience, this is an incredibly effective way to DoS yourself. It was the default behaviour of nginx LB ages ago. Maybe only on EngineYard. Doesn't really matter as nobody uses nginx LB anymore. Even ignoring the POST requests problem (yup, it tried to replay those) properly cancelling a request on all levels of a multi-level rails stack is very hard/not possible in practice. So you end up DOSing the hard to s…

Nginx introduced least_conn lb method in 1.3.1 which makes it a bit better. http://nginx.org/en/docs/http/ngx_http_upstream_module.html#...

ha-proxy is a lot better than nginx + more flexible if you want to introduce non-http to your stack.

Shouldn't the request be canceled on all levels if you cut the HTTP connection to the frontend?

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#290

Post by jacques_chester »

benjamincburns wrote: post #179

This kind of validates an idea I've been flirting with: a Heroku-like service which routes requests via AMQP or similar message broker and actually exposes the routing dynamics to the client apps. From a naive, inexperienced view the idea of having web nodes "pull" requests from a central queue rather than the queue taking uneducated guesses seems to be a no-brainer. I can see this making long-running requests (keep-…

It has: https://github.com/paulj/trapeze

You might also be interested in what's been done in the literature; look under "Staged Event Driven Architecture" aka SEDA.

Mongrel2 and OK Web Server both support the SEDA approach.

Post reply on HN