Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

rapgenius.com

Post reply on HN
Print view View original

161–170 of 437 posts

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#161

Post by jholman »

awj wrote: post #139
jholman wrote: post #100

Earlier quoted context omitted.

Re the distribution, absolutely. That "FIFTY TIMES" is totally due to the width of the distribution. Although, you know, even if their app was written such that every single request took exactly 100ms of dyno time, this random routing would create the problem all over again, to some degree. As for the intelligent routing, could you explain the problem? The goal isn't to predict which request will take a long time, th…

I think the problem is that any servers which can handle concurrent requests now need to decide how many requests they can handle. Since most application servers seem to have concurrency values of "1, ever" or "I dunno, lots" this is a hard problem. Your solution would likely work if you had some higher level (application level? not real up on Heroku) at which you could specify a push vs. pull mechanism for request r…

Yeah, I dunno squit about Heroku either.

Given that, according to TFA (and it's consistent with some other things I've read) Heroku's bread and butter is Rails apps, and given that, according to TFA, Rails is single-threaded, that (valid) point about concurrency in a single dyno is perhaps not that relevant? You'd think that Heroku would continue to support the routing model that almost all of their marketing and documentation advertises, right? Even if it's a configurable option, and it only works usefully with single-threaded servers?

And if you did do it pull-based, it wouldn't be Heroku's problem to decide how many concurrent requests to send. Leave it to the application (or whatever you call the thing you run on a dyno).

And it doesn't need to be pull-based, if the router can detect HTTP connections closing in dynos, or whatever.

But the idea of pull-based work distribution is pretty straightforward. It's called a message queue.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#162

Post by eignerchris_ »

Thanks for calling this out. As you said, random routing is about as naive as it gets. They need to make upgrades to the routing mesh - expose some internal stats about dyno performance and route accordingly. Even if the stats were rudimentary, anything would be an improvement over random.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#164

Post by rapind »

I'd been using Heroku since forever, but bailed on them for a high traffic app last year (Olympics related) due to poor performance once we hit a certain load (adding dynos made very little difference). We were paying for their (new at the time) critical app support, and I brought up that it appears to be failing at a routing level continuously. And this was with a Sinatra app served by Unicorn (which at the time at least was considered unsupported).

We went with a metal cluster setup and everything ran super smooth. I never did figure out what the problem was with Heroku though and this article has been a very illuminating read.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#165

Post by ijl »

zrail wrote: post #47

For those of you looking to migrate to other, barer hosting solutions like AWS or another VPS provider, I've put together a Capistrano add-on that let's you use Heroku-style buildpacks to deploy with Nginx doing front-end proxy. I use it for half a dozen apps on my VPSs and it works swimmingly well. https://github.com/peterkeen/capistrano-buildpack

Thank you.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#166

Post by gojomo »

jemfinch wrote: post #127
zenazn wrote: post #13

Randomized routing isn't all bad. In fact, if Heroku were to switch from purely random routing to minimum-of-two random routing, they'd perform asymptotically better [1]. [1]: http://www.eecs.harvard.edu/~michaelm/postscripts/mythesis.p...

If Heroku had the data needed to do minimum-of-two random routing, they'd have the data needed to do intelligent routing. The problem is not the algorithm itself: "decrement and reheap" isn't going to be a performance bottleneck. The problem is tracking the number of requests queued on the dyno.

If Heroku had the data needed to do minimum-of-two random routing, they'd have the data needed to do intelligent routing.

Not strictly true; imagine that they can query the load state of a dyno, but at some non-zero cost. (For example, that it requires contacting the dyno, because the load-balancer itself is distributed and doesn't have a global view.)

Then, contacting 2, and picking the better of the 2, remains a possible win compared to contacting more/all.

See for example the 'hedged request' strategy, referenced in a sibling thread by nostradaemons from a Jeff Dean Google paper, where 2 redundant requests are issued and the slower-to-respond is discarded (or even actively cancelled, in the 'tiered request' variant).

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#167

Post by haddr »

I would like to see why actually Heroku fell back to random routing. It doesn't really make sense. Of course this all routing stuff is really tricky, but on the other hand there is a lot of work done (look at TCP agorithms). When I was studying ZeroMQ routing based stuff for one project, I came across "credit-based flow control" pattern, that could make perfect sense in this kind of situation (Publisher-Subscriber scenario). Why not implementing such thing?

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#168

Post by gleb »

nostrademons wrote: post #136
harshreality wrote: post #80

Earlier quoted context omitted.

I seem to recall Google mentioning on some blog several years ago that high variance in response latency degrades user experience much more than slightly higher average request times. I can't find the link though; if anyone has it, I'd be grateful.

Jeff Dean wrote a paper on it for CACM: http://cacm.acm.org/magazines/2013/2/160173-the-tail-at-scal... There's a relatively easy fix for Heroku. They should do random routing with a backup second request sent if the first request times fails to respond after a relatively short period of time (say, 95th percentile latency), killing any outstanding requests when the first response comes back in. The amount of bookkeep…

From experience, this is an incredibly effective way to DoS yourself. It was the default behaviour of nginx LB ages ago. Maybe only on EngineYard. Doesn't really matter as nobody uses nginx LB anymore.

Even ignoring the POST requests problem (yup, it tried to replay those) properly cancelling a request on all levels of a multi-level rails stack is very hard/not possible in practice. So you end up DOSing the hard to scale lower levels of the stack (e.g. database) at the expense of the easy to scale LB.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#169

Post by tim_sw »

jemfinch wrote: post #132
tim_sw wrote: post #28

randomized routing is not necessarily bad if they look at 2 choices and pick the min. See http://en.wikipedia.org/wiki/2-choice_hashing and http://www.eecs.harvard.edu/~michaelm/postscripts/handbook20...

This is a knee-jerk reply. I know, because my knee jerked as well. Think about the problem a little more: if you have the data necessary to pick the min-of-two, then you have the data you need to do intelligent routing.

Assuming by intelligent, you mean minimally loaded, choice of 2 requires less bookkeeping than this. (choice of 2 requires only local (per-node level) info, where as most other intelligent load-balancing requires global info ie. the min, etc.)

Taking the case of minimally loaded, you need to keep track of how many active requests each node/replica is serving, as well as globally keeping track of the min. (which past a certain load, will suffer a lot of contention to update)

To do choice of 2, all you need is to keep track of active requests per node/replica.

Under spiky workloads, there is also a problem with choosing minimally loaded. The counter for numRequests of a node might not update fast enough, so that a bunch of requests will go to that node, quickly saturating its capacity.

Choice of 2 doesn't suffer this problem bec of its inherent randomization.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#170

Post by toast76 »

Cushman wrote: post #84
runarb wrote: post #74

Earlier quoted context omitted.

Is it so that a dyno can only handle a single user request at a time? Why dos it not use some kind of scheduling system to handle other task while one task is waiting on i/o?

It's not exactly so, if you use a server that spawns child processes: http://michaelvanrooijen.com/articles/2011/06/01-more-concur... you can potentially handle 3-4 requests per dyno at a time. That doesn't fix the root problem, though.

Investigating this approach now. It won't fix the problem, but will certainly reduce the occurrence of blocked dynos. Thx!

EDIT: will need to look into our memory perf though, looks like we'll need to do some work to get more than a couple of workers.

Post reply on HN