Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Answering your questions about Heroku routing and web performance

blog.heroku.com

Post reply on HN
Print view View original

121–130 of 168 posts

Re: Answering your questions about Heroku routing and web performance

#121

Post by adamwiggins »

latchkey wrote: post #95
adamwiggins wrote: post #46

Earlier quoted context omitted.

Didn't intend to imply that. Number of dynos needed varies extremely widely, with the app's response time and language/framework being used as the main variables. There are apps on Heroku that serve 30k–50k reqs/min on 10–20 dynos, typically written in something like Scala/Akka or Node.js and serving incredibly short (~30ms) response times with very little variation. But these are unusual. The more common case of a w…

And if you get into any sort of slowness (like say mongo decides to pull something from disk instead of ram), it is instantly H12's all over the place and there is nothing you can do about it.

This comes back to visibility: knowing where the problem lies (especially when you're using a variety of add-on services or calling external APIs) and being able to understand what's happening, or what happened in retrospect.

Visibility is hard no matter where you run your app. But this is an area where Heroku can get a lot better, and we intend to.

Re: Answering your questions about Heroku routing and web performance

#122

Post by friism »

jules wrote: post #115
tptacek wrote: post #72

Earlier quoted context omitted.

I understand and don't disagree with everything you are saying, but the focus of my attention is on what you're talking about in your 3rd graf. When you talk about your example problems (1) and (2) with routing to concurrent systems on large number of dynos, what you're really discussing is an engineering flaw in the typical Rails stack. There's a tradeoff between: * a well-engineered request handler (a solved proble…

I'm not sure I agree. Yes, it's a Rails problem that it is using large amounts of memory (on the other hand (2) isn't Rails specific at all, it applies equally to e.g. Node). But it's a Heroku problem that it gives Dynos just 512MB of memory. It's a Heroku problem that it doesn't have a good load balancer. Heroku is in the business of providing painless app hosting, and part of that is painless request routing. These…

I work at Heroku. To address you concerns about memory limitations, know that we're fast-tracking 2X dynos (this is also mentioned in the FAQ blog post). Extra memory will make it easier to get more concurrency out of each dyno.

Re: Answering your questions about Heroku routing and web performance

#123

Post by adamwiggins »

latchkey wrote: post #114
adamwiggins wrote: post #111

Earlier quoted context omitted.

We're aware of the random H12s problems. Some apps are affected pretty badly, others not at all, and we're not sure why yet. Sorry that you've had such a bad experience with this. We're continuing to investigate. If we're not able to find a solution in a timely fashion, I'll completely understand if you no longer want to use our product for this app. Knowing how many dynos you need is definitely a problem. We have im…

Your response only re-enforces my hard won opinion that Heroku should never be used for a production environment for any business that is trying to be successful and popular. Admitting that you have no idea why critical areas of your infrastructure is causing issues, while at the same time charging people an arm and a leg for services (we pay ~$4k/mo) feels like theft to me. I've built solutions for a large porn comp…

Again, sorry to hear about your bad experience. Hard-to-diagnose errors, no autoscaling or other method to know proper dyno provisioning, and slow deploy times — these things suck.

Would love the chance to win back your trust and hang onto your business. Let me know your app name (in email if you prefer) and I can see if there's anything we can do for you in the near term.

Re: Answering your questions about Heroku routing and web performance

#124

Post by jholman »

I'd like to start by acknowledging that I'm one of the "non-customers who are watching from the sidelines". I think Adam's right that this is an important distinction.

Adam, there's something that confuses me about this. I'm no expert in routing theory, nor have I done the experiments, so forgive me if my reasoning misses something.

I understand why RapGenius took you up on your original promises of "intelligent routing", and I think I understand what you're saying about scaling, and how scaling "intelligent routing" is so far unsolvable, and the motivation for your transition from Bamboo to Cedar, especially in the context of concurrent clients. What I don't understand is this:

It seems to me that if you split into two (or more) tiers, and random-load-balance in the front tier (hit first by the customer), and then at the second tier only send requests to unloaded clients, that you eliminate RapGenius's problem for customers who followed your specific recommendations for good performance on Bamboo (to go single-threaded and trust the router).

Do you have reason to believe that this doesn't one-shot RapGenius's problem? Do you have strategic/architectural reasons for rejecting this even though it would work? Did you try it and it failed? What's the story there?

Maybe I'll write a simulator to (dis)prove my naive theory. :P

Re: Answering your questions about Heroku routing and web performance

#125

Post by latchkey »

adamwiggins wrote: post #121
latchkey wrote: post #95

Earlier quoted context omitted.

And if you get into any sort of slowness (like say mongo decides to pull something from disk instead of ram), it is instantly H12's all over the place and there is nothing you can do about it.

This comes back to visibility: knowing where the problem lies (especially when you're using a variety of add-on services or calling external APIs) and being able to understand what's happening, or what happened in retrospect. Visibility is hard no matter where you run your app. But this is an area where Heroku can get a lot better, and we intend to.

Visibility is one part of the problem. 50k requests/min is only ~833/s. The reality is that a single dyno should be able to more than handle that sort of load, especially if it is a simple app. People are doing 10k connections on a single laptop, 833s should be a piece of cake. So, yes, visibility is a big issue here because you have no idea if you need 10, 11, 12 or 20 dyno's to serve 50k requests/min. You just guess and when you guess wrong, it ends up with cascading failure of H12's and other issues. Never mind that very few apps have a steady stream of traffic and most have big dips depending on the time of day and HN popularity... and now we are back to the auto scaling discussion.

Another key part of your statement is 'with very little variation'. The code pretty much can't be doing anything other than serving up some static content because as soon as anything that requires any sort of IO or cpu will instantly throw the system into H12 hell. Yes, a CDN will take load off your Heroku dyno's because god forbid that your dyno actually do anything itself. Except that you forget that not all apps are webapps and in my case, there is no reason to add a CDN when I'm just serving requests and responses to an iphone app.

The other part of the problem is being able to actually do something about it. I've tried anywhere between 50 and 300 dynos (yes we got that number increased). If we could just throw money at the problem that would be one thing, but nothing was able to resolve the H12's that we see and our paid support contract was no help either.

"If you're running a blog that serves 600 rpm / 10 reqs/sec off of two dynos, you don't need to sweat it."

Once again, we are back at the same conclusion... don't use Heroku if you want to run a production system.

Re: Answering your questions about Heroku routing and web performance

#128

Post by jules »

tptacek wrote: post #117
jules wrote: post #115

Earlier quoted context omitted.

I'm not sure I agree. Yes, it's a Rails problem that it is using large amounts of memory (on the other hand (2) isn't Rails specific at all, it applies equally to e.g. Node). But it's a Heroku problem that it gives Dynos just 512MB of memory. It's a Heroku problem that it doesn't have a good load balancer. Heroku is in the business of providing painless app hosting, and part of that is painless request routing. These…

If there is another provider that seamlessly operates at Heroku's scale (ie, that can handle arbitrarily busy Rails apps) at a reasonable price that has better request dispatching, I think it's very easy to show that you're right. I'm not sure there are such providers, and if there aren't, I think it's safe to point the finger towards Rails. As a system for efficiently handling database-backed web requests, Rails is…

I certainly already agreed that Rails' architecture is bad (though the reason that it has this problem is its memory usage, and not any of the other reasons you mention). Herokus architecture is bad as well. It's the combination of these that causes the problem. But that does not mean that it's impossible, or even hard, to solve the problem at Herokus end.

> I'm not sure there are such providers, and if there aren't, I think it's safe to point the finger towards Rails.

This is not sound logic. I described above two methods for solving the problem: (1) increase the memory per Dyno (see below: they're doing this, going from 512MB to 1GB per Dyno IIRC, which although still low will be a great improvement if that means that your app can now run 2 concurrent processes per Dyno instead of 1), or (2) do intelligent routing for small groups of Dynos. Do you understand the problem with random routing, and why either of these two would solve it? If not you might find the paper I linked to previously very interesting:

"To motivate this survey, we begin with a simple problem that demonstrates a powerful fundamental idea. Suppose that n balls are thrown into n bins, with each ball choosing a bin independently and uniformly at random. Then the maximum load, or the largest number of balls in any bin, is approximately log n / log log n with high probability. Now suppose instead that the balls are placed sequentially, and each ball is placed in the least loaded of d >= 2 bins chosen independently and uniformly at random. Azar, Broder, Karlin, and Upfal showed that in this case, the maximum load is log log n / log d + Θ(1) with high probability [ABKU99].

The important implication of this result is that even a small amount of choice can lead to drastically different results in load balancing. Indeed, having just two random choices (i.e., d = 2) yields a large reduction in the maximum load over having one choice, while each additional choice beyond two decreases the maximum load by just a constant factor."

-- http://www.eecs.harvard.edu/~michaelm/postscripts/handbook20...

Re: Answering your questions about Heroku routing and web performance

#129

Post by perucoder »

adamwiggins wrote: post #110
perucoder wrote: post #83

Earlier quoted context omitted.

If its only a few customers that are bent out of shape, how come you haven't quickly offered them refunds?

We did.

When did this happen? As of March 2, Rap Genius was still seeking to get money refunded.

http://venturebeat.com/2013/03/02/rap-genius-responds/

Re: Answering your questions about Heroku routing and web performance

#130

Post by tptacek »

jules wrote: post #128
tptacek wrote: post #117

Earlier quoted context omitted.

If there is another provider that seamlessly operates at Heroku's scale (ie, that can handle arbitrarily busy Rails apps) at a reasonable price that has better request dispatching, I think it's very easy to show that you're right. I'm not sure there are such providers, and if there aren't, I think it's safe to point the finger towards Rails. As a system for efficiently handling database-backed web requests, Rails is…

I certainly already agreed that Rails' architecture is bad (though the reason that it has this problem is its memory usage, and not any of the other reasons you mention). Herokus architecture is bad as well. It's the combination of these that causes the problem. But that does not mean that it's impossible, or even hard, to solve the problem at Herokus end. > I'm not sure there are such providers, and if there aren't,…

I understand that one approach to dispatching requests at the load balancer is superior to the other, just as I understand that one way of absorbing requests at the app server is better than the other.

Most things are inferior to other substitutable things! :)

Post reply on HN