Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

rapgenius.com

Post reply on HN
Print view View original

291–300 of 437 posts

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#291

Post by nostrademons »

badgar wrote: post #150
nostrademons wrote: post #136

Earlier quoted context omitted.

Jeff Dean wrote a paper on it for CACM: http://cacm.acm.org/magazines/2013/2/160173-the-tail-at-scal... There's a relatively easy fix for Heroku. They should do random routing with a backup second request sent if the first request times fails to respond after a relatively short period of time (say, 95th percentile latency), killing any outstanding requests when the first response comes back in. The amount of bookkeep…

> There's a relatively easy fix for Heroku. They should do random routing with a backup second request sent if the first request times fails to respond after a relatively short period of time (say, 95th percentile latency), killing any outstanding requests when the first response comes back in. The amount of bookkeeping required for this is a lot less than full-on intelligent routing, but it can reduce tail latency d…

Yeah, I know. I figure that for incoming HTTP traffic it's relatively easy to balance the GET requests, and if they're doing anything remotely sane with HTTP those ought to be idempotent (if they're not, Googlebot will come along and delete their site ;-)).

For mutating requests, there's a solution as well, but it involves checksumming the request and passing the checksum along so that the database layer knows to discard duplicate requests that it's already handled. You need this anyway if there's any sort of retry logic in your application, though.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#292

Post by jacques_chester »

dblock wrote: post #198
benjamincburns wrote: post #179

This kind of validates an idea I've been flirting with: a Heroku-like service which routes requests via AMQP or similar message broker and actually exposes the routing dynamics to the client apps. From a naive, inexperienced view the idea of having web nodes "pull" requests from a central queue rather than the queue taking uneducated guesses seems to be a no-brainer. I can see this making long-running requests (keep-…

The pull model is very hard to implement because the router behaves like a proxy for a much larger set of dynos (think tens of thousands). When you have 10K clients yielding "i'm available" 10 times a second, you have a nightmare, it's not sustainable. A possible solution for the proxy and the dynos to agree on a protocol where the proxy passes a request to the dyno and the latter can give up with a status code that…

In Mongrel2, app servers subscribe to a named ZeroMQ queue and, when they're done, they send the response on a different queue.

You can actually configure an arbitrary number of different queues if you like, switching on request path and some other stuff I don't just now recall.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#293

Post by pdonis »

mattj wrote: post #12

So the issue here is two-fold: - It's very hard to do 'intelligent routing' at scale. - Random routing plays poorly with request times with a really bad tail (median is 50ms, 99th is 3 seconds) The solution here is to figure out why your 99th is 3 seconds. Once you solve that, randomized routing won't hurt you anymore. You hit this exact same problem in a non-preemptive multi-tasking system (like gevent or golang).

The solution here is to figure out why your 99th is 3 seconds.

I'm not sure this applies to the OP. His in-app measurements were showing all requests being handled very fast by the app itself; the variability in total response time was entirely due to the random routing.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#294

Post by themgt »

skrebbel wrote: post #286
themgt wrote: post #274

Earlier quoted context omitted.

I agree there should be a better way - it's just important to understand than Rails doesn't get any special treatment on a PaaS done correctly, so it's important to come up with a generic solution. I think part of the solution would be customizable option(i.e.. how many requests can each dyno handle simultaneously), probably combined with intelligently monitoring/balancing proxy load so new requests always go to the…

> it's just important to understand than Rails doesn't get any special treatment on a PaaS done correctly Why is it only "done correctly" if it does not account for specific properties of the technology used by a particular customer?

Because PaaS is a generic technology for running and scaling applications with a multitude of different language/framework/stacks, and many/most of those apps do not share the specific properties of single-threaded Rails (including many Ruby/Rails apps!)

And Rails 4 is going to bake-in "live streaming", making single-threaded app servers even more of an edge case.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#296

Post by gojomo »

nacho2sweet wrote: post #238

Why is everyone like against rapgenius.com for "forcing the issue with a public spat". They are the customer not getting a service they are paying for. I would be fucking pissed too. Heroku isn't being the darling the service they advertised. They tried to work on it with Heroku. This is useful information to most of you. Are most of you against Yelp?

The only person using the words 'force the issue with a public spat' is me, and I judged that as 'fair enough'. I'm not against RapGenius and I'm glad the issue is being discussed.

But we haven't seen Heroku's comments, and while some parts of RapGenius's complaint are compelling, I'm not sure their apparent conclusions - that 'intelligent routing' is needed, and its lack is screwing Heroku customers — are right. I strongly suspect some small tweaks, ideally with Heroku's help, but perhaps even without it, can fix most of RapGenius's concerns.

Perhaps there was a communication or support failure, which led to the public condemnation, or maybe that's just RG's style, to heighten the drama. (That's an observation without judgement; quarrels can benefit both parties in some attention- and personality-driven contexts.)

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#297

Post by nostrademons »

gleb wrote: post #168
nostrademons wrote: post #136

Earlier quoted context omitted.

Jeff Dean wrote a paper on it for CACM: http://cacm.acm.org/magazines/2013/2/160173-the-tail-at-scal... There's a relatively easy fix for Heroku. They should do random routing with a backup second request sent if the first request times fails to respond after a relatively short period of time (say, 95th percentile latency), killing any outstanding requests when the first response comes back in. The amount of bookkeep…

From experience, this is an incredibly effective way to DoS yourself. It was the default behaviour of nginx LB ages ago. Maybe only on EngineYard. Doesn't really matter as nobody uses nginx LB anymore. Even ignoring the POST requests problem (yup, it tried to replay those) properly cancelling a request on all levels of a multi-level rails stack is very hard/not possible in practice. So you end up DOSing the hard to s…

It's a capacity/latency tradeoff. Perhaps I'm biased by working at Google, where capacity is cheap and latency will kill you, but IIUC Heroku runs off of AWS and their database tier needs to scale horizontally anyway, so reserving sufficient overflow capacity should simply be a matter of throwing money at the problem.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#299

Post by skrebbel »

themgt wrote: post #294
skrebbel wrote: post #286

Earlier quoted context omitted.

> it's just important to understand than Rails doesn't get any special treatment on a PaaS done correctly Why is it only "done correctly" if it does not account for specific properties of the technology used by a particular customer?

Because PaaS is a generic technology for running and scaling applications with a multitude of different language/framework/stacks, and many/most of those apps do not share the specific properties of single-threaded Rails (including many Ruby/Rails apps!) And Rails 4 is going to bake-in "live streaming", making single-threaded app servers even more of an edge case.

That's like saying Craigslist did it correctly and AirBnB didn't because AirBnB is only tailoring to a specific segment of the world's supply and demand market.

Rails is very widely used. How can you consider that an edge case?

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#300

Post by stock_toaster »

newhouseb wrote: post #277
nsrivast wrote: post #262

OP is a friend of mine, and when I first heard of his problem I wondered if there might be an analytical solution to quantify the difference between intelligent vs naive routing. I took this problem as an opportunity to teach myself a bit of Queueing Theory[1], which is a fascinating topic! I'm still very much a beginner, so bear with me and I'd love to get any feedback or suggestions for further study. For this exam…

Queuing theory is cool, but I'm not 100% sure it actually applies here in a meaningful sense (although, disclaimer: I'm no more experienced here than you). A lot of queuing theory assumes that you must route a request to a handler immediately as you receive it, and that reassigning a request is a very expensive process. This intuitively explains why queueing theory is very big in router design - imagine that you send…

haproxy also has a few balance algos[1] that it can be configured to use. I would think something like static-rr would even be somewhat better than random.

[1]: http://cbonte.github.com/haproxy-dconv/configuration-1.5.htm...

Post reply on HN