Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

rapgenius.com

Post reply on HN
Print view View original

381–390 of 437 posts

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#381

Post by bambax »

eli wrote: post #373
bambax wrote: post #325

Earlier quoted context omitted.

What's the point of posting a link to the front page of your blog, where the most recent article is 15 days old (4 hours after the comment above)? What we want to know: - is the OP right or wrong? That is, did you switch from smart to naive routing, for all platforms, and without telling your existing or future customers? - if you did switch from smart to naive routing, what was the rationale behind it? (The OP is li…

What's the point of posting a link to the front page of your blog Well, he promised a detailed blog post, at which point that link will be extremely helpful. I do not think it is fair to expect an immediate detailed response to those questions. If I were CEO of Heroku, I wouldn't say anything definite until after talking to the engineers and product managers involved--even if I was already pretty sure what happened.…

I don't expect an immediate response; I would have been happy with just: "This is Heroku's CEO. I'm on it."

But a link, that doesn't point anywhere useful, introduced by a PR phrase that sounds a little like "Your call is important to us", was a little annoying, esp. after reading the OP where they say they have contacted Heroku multiple times on this issue.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#382

Post by jwrubel »

tibbon wrote: post #243
scottshea wrote: post #237

Earlier quoted context omitted.

To amplify, I do a ton of queue adjustment work with Unicorn at the backlog level. It is so frequent that we set up Unicorn startup to read from an ENV variable on Heroku that we set as needed. With two Unicorn workers we found that 25 was the best backlog threshold to accept (it refuses additional requests). When we were able to go to 5 Unicorn workers on Heroku we had to start to adjust that.

You don't happen to have any documentation for how to do that do you? Very curious. Never seen anything about setting up Unicorn like this prior (I'm just using 3 or 4 Unicorns/dyno currently)

Here's a gist of our unicorn.rb config (https://gist.github.com/apangeajwrubel/4953849) Using env variables lets us adjust parameters without a code push (still requires a restart). We saw dramatic reduction in 'long tail' requests when we dropped the backlog to 25. We're experimenting now (thanks to @scottshea) with even lower values. At some point the routing mesh will give up retrying and throw an H21 (https://devcenter.heroku.com/articles/error-codes#h21-backen...). One data point that would be useful from heroku is how many retries we get.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#383

Post by vidarh »

jholman wrote: post #106
46Bit wrote: post #85

Earlier quoted context omitted.

Several Rails apps I develop have been suffering from similar issues. Perhaps 2-3% of requests take 0.4-2s in just processing. If the allocation is a little intelligent, it'll not perform too badly and is less work than much harder optimization. Yet if it's random, it'll queue up horribly . I'm pissed. Spent way too much time unable to explain it to coworkers, thinking I just didn't understand Heroku's platform and t…

> Easiest fix: moving to EC2 next week. I've wanted to ever since these issues became evident but it's hard to make a good argument from handwaving about 'problems'. Of course, then you need to solve all these problems yourself. That sounds pretty easy, you'll have it done next week no problem! That was sarcastic, but this isn't: good luck, let us know how it goes.

Solving this problem is easy: Run haproxy, and he'll have detailed control over the balancing algorithm, including balancing by least connections and a number of other measures (if he, for example, wants to segregate the long running requests on a specific set of backends, it's trivial).

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#384

Post by pointful »

tibbon wrote: post #243
scottshea wrote: post #237

Earlier quoted context omitted.

To amplify, I do a ton of queue adjustment work with Unicorn at the backlog level. It is so frequent that we set up Unicorn startup to read from an ENV variable on Heroku that we set as needed. With two Unicorn workers we found that 25 was the best backlog threshold to accept (it refuses additional requests). When we were able to go to 5 Unicorn workers on Heroku we had to start to adjust that.

You don't happen to have any documentation for how to do that do you? Very curious. Never seen anything about setting up Unicorn like this prior (I'm just using 3 or 4 Unicorns/dyno currently)

To expand on this:

You have to remove the port declaration from the line for Unicorn in your Procfile, and then add a line like this to your unicorn.rb file to define the listener port along with adjusting the backlog size:

listen ENV['PORT'], :backlog => Integer(ENV['UNICORN_BACKLOG'] || 100)

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#385

Post by papsosouid »

themgt wrote: post #294
skrebbel wrote: post #286

Earlier quoted context omitted.

> it's just important to understand than Rails doesn't get any special treatment on a PaaS done correctly Why is it only "done correctly" if it does not account for specific properties of the technology used by a particular customer?

Because PaaS is a generic technology for running and scaling applications with a multitude of different language/framework/stacks, and many/most of those apps do not share the specific properties of single-threaded Rails (including many Ruby/Rails apps!) And Rails 4 is going to bake-in "live streaming", making single-threaded app servers even more of an edge case.

The 'P' stands for platform. Providing the platform as a service absolutely means catering to the specific needs of the platform. There is no generic platform. If you want to support multiple platforms, then you support multiple platforms. You don't stop supporting any platform at all.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#386

Post by DigitalJack »

vidarh wrote: post #379
mononcqc wrote: post #353

Earlier quoted context omitted.

I'm sure OP could prefer the extra 10ms, but then everyone else who can deal with random dispatching right now has to pay a 10ms penalty because OP built his stuff on a technology that can deal with only one request at a time on a server, which boggles the mind to begin with.

Why? The system could easily be built so that it by default only aggregates those services where the configuration indicates they can handle a concurrency below a certain level, and does random balancing of everything else. The "everyone else who can deal with random dispatching right now" is a much smaller group than you think. Anyone who has long running requests that grind the CPU or disk when running, will be at…

"The system could easily be built so that it by default only aggregates those services where the configuration indicates they can handle a concurrency below a certain level, and does random balancing of everything else."

Let me know when you are done with that.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#387

Post by vicks711 »

humbledrone wrote: post #318
vicks711 wrote: post #287

Earlier quoted context omitted.

I cant stop myself from saying this. You wrote all this instead of doing what?

Normally I would just downvote you and move on, but in this case your comment is frustrating enough that I have to say something. I found the comment you responded to (by nsrivast) quite fascinating. A well-written but brief analysis of the problem, with sources attached for further reading -- what's not to like? In-depth and thoughtful comments like that are what keep me coming back to this site, and are what make t…

your handle is humbledrone and your opinions are humbled. Boy am i impressed!

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#388

Post by vidarh »

joseph_cooney wrote: post #244
teraflop wrote: post #197

Earlier quoted context omitted.

They will if the request is CPU-bound. In that case, throwing more concurrent requests at a server than it has cores just slows all of them down.

How many web requests are typically CPU bound though? Not many.

Doesn't matter.

At some point you hit memory limits, disk IO limits, or simply a connection limit. It doesn't matter what limit:

If you have some requests that are longer running than others, random load balancing will make them start piling up once you reach some traffic threshold.

You can increase the threshold by adding more backends or increasing the capacity of each backend (by optimizing, or picking beefier hardware if you're on a platform that will let you), and maybe you can increase it enough that it won't affect you.

But no matter what you do, you end up having to allocate more spare resources at it than what you would need with more intelligent routing.

If you're lucky, the effect might be small enough to not cost you much money, and you might be able to ignore it, but it's still there.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#389

Post by vidarh »

DigitalJack wrote: post #386
vidarh wrote: post #379

Earlier quoted context omitted.

Why? The system could easily be built so that it by default only aggregates those services where the configuration indicates they can handle a concurrency below a certain level, and does random balancing of everything else. The "everyone else who can deal with random dispatching right now" is a much smaller group than you think. Anyone who has long running requests that grind the CPU or disk when running, will be at…

"The system could easily be built so that it by default only aggregates those services where the configuration indicates they can handle a concurrency below a certain level, and does random balancing of everything else." Let me know when you are done with that.

I've built fairly large haproxy based infrastructures, thank you very much. Doing this is not particularly challenging.

Actually what I'd probably do for a setup like this would be to balance by the Host: header, and simply have the second layer be a suitable set of haproxy instances balancing each by least connections.

Immediately vastly better than random.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#390

Post by pointful »

Just adding a top-level post to point out something buried in one of the threads here that is an important point on what is happening here:

The "queue at the dyno level" is coming from the Rails stack -- it's not something that Heroku is doing to/for the dynos.

Thin and Unicorn (and others, I imagine) will queue requests as socket connections on their listener. Both default to 1024 backlog requests. If you lower that number, Heroku will (according to the implications in the documentation on H21 errors) try multiple other dynos first before giving up.

See https://devcenter.heroku.com/articles/error-codes#h21-backen...

For a single-threaded process to be willing to backlog a thousand requests is problematic when combined with random load balancing. Dropping this number down significantly will lead to more sane load-balancing behavior by the overall stack, as long as there are other dynos available to take up the slack.

Also, the time the request spends on the dyno, including the time in the dyno's own backlog, is available in the heroku router log. It's the "service" time that you'll see as something like "... wait=0ms connect=1ms service=383ms ...". Definitely wish New Relic was graphing that somewhere...

Post reply on HN