Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Answering your questions about Heroku routing and web performance

blog.heroku.com

Post reply on HN
Print view View original

81–90 of 168 posts

Re: Answering your questions about Heroku routing and web performance

#81

Post by josh2600 »

aphyr wrote: post #60
ebbv wrote: post #4

It seems to me that Heroku is still failing to understand (or at least cop to) the fact that the switch from intelligent to randomized routing was a loss of a major reason people chose Heroku in the first place. A lot of Heroku's apparent value came from the intelligent routing feature. Everybody knew that it was harder to implement than random routing, that's why they were willing to pay Heroku for it. Nobody's argu…

"the switch from intelligent to randomized routing" As I understand it, Heroku (on the Bamboo stack) didn't up and decide "Hey, we're gonna switch from intelligent to random routing." The routers are still (individually) intelligent. It's just that there are more of them now, and they were never designed to distribute their internal queue state across the cluster. The system as a whole behaves more and more like a ra…

I don't understand how the system behaves more intelligently when the edge routers increase. The core routers random behavior gets worse with the larger load, and increasing the number of intelligent routers doesn't help to solve that problem in any meaningful way.

Sorry, correct me if I missed something, but I believe that as the overall volume of system transactions increases (thus necessitating more "intelligent" nodes) the volume of random dispersal from the core routers increases as well, which can create situations like what we saw with RapGenius where some requests are 100ms and others are 6500ms (the reason being that random routing is not intelligent and can assign jobs to a node that's completely saturated). Adding more and more intelligent nodes doesn't solve the crux of the issue, which is the random assignment of jobs in the core routers to the "intelligent" routers/nodes.

This whole situation boils down to this: "Intelligent routing is hard, so fuck it" and that's why everyone is pissed off. Heroku could've said "hey Intelligent routing is hard so we're not doing that anymore" but instead they just silently deprecated the service. It's a textbook example of how to be a bad service provider.

Re: Answering your questions about Heroku routing and web performance

#82

Post by obilgic »

adamwiggins wrote: post #79
aphyr wrote: post #60

Earlier quoted context omitted.

"the switch from intelligent to randomized routing" As I understand it, Heroku (on the Bamboo stack) didn't up and decide "Hey, we're gonna switch from intelligent to random routing." The routers are still (individually) intelligent. It's just that there are more of them now, and they were never designed to distribute their internal queue state across the cluster. The system as a whole behaves more and more like a ra…

True, although it's more confusing than this. We didn't make an explicit decision for Bamboo, and thus the big problem — docs fell out of date, at a minimum. For Cedar, we did make an explicit decision. People on the leading edge of web development were running concurrent backends and long-running requests. Our experimental support for Node.js in 2010 was a big driver here, but also people who wanted to use concurren…

> true

that makes sense, i was wondering how intelligent routing was implemented in the first place.

Re: Answering your questions about Heroku routing and web performance

#83

Post by perucoder »

adamwiggins wrote: post #30
ebbv wrote: post #24

Earlier quoted context omitted.

I doubt anyone chose Heroku solely because of the routing algorithm, but the sales point of intelligent routing was certainly an appealing one. Developers don't want to pay for abstraction just for abstraction's sake, they want to pay for abstraction of difficult things. Intelligent routing is one of those difficult things. Random routing is easy, which is of course why it's also more reliable, but also why you're se…

Well said. It's interesting — very few customers are actually bent out of shape about this. (A few are, for sure.) It's more non-customers who are watching from the sidelines that seem to be upset. I do want to try to explain ourselves to the community in general, and that's what this post was for. But my first loyalty is to serving our current customers well.

If its only a few customers that are bent out of shape, how come you haven't quickly offered them refunds?

Re: Answering your questions about Heroku routing and web performance

#84

Post by prospero »

josh2600 wrote: post #81
aphyr wrote: post #60

Earlier quoted context omitted.

"the switch from intelligent to randomized routing" As I understand it, Heroku (on the Bamboo stack) didn't up and decide "Hey, we're gonna switch from intelligent to random routing." The routers are still (individually) intelligent. It's just that there are more of them now, and they were never designed to distribute their internal queue state across the cluster. The system as a whole behaves more and more like a ra…

I don't understand how the system behaves more intelligently when the edge routers increase. The core routers random behavior gets worse with the larger load, and increasing the number of intelligent routers doesn't help to solve that problem in any meaningful way. Sorry, correct me if I missed something, but I believe that as the overall volume of system transactions increases (thus necessitating more "intelligent"…

You seem to have misread the above comment. Here's what it actually said:

The system as a whole behaves more and more like a random router as the number of intelligent bamboo routing nodes increases

The point is that this process was gradual and implicit, so there's no point at which the intelligence was explicitly "deprecated". That doesn't excuse how things ended up, but it does explain it to some degree.

Re: Answering your questions about Heroku routing and web performance

#85

Post by aphyr »

josh2600 wrote: post #81
aphyr wrote: post #60

Earlier quoted context omitted.

"the switch from intelligent to randomized routing" As I understand it, Heroku (on the Bamboo stack) didn't up and decide "Hey, we're gonna switch from intelligent to random routing." The routers are still (individually) intelligent. It's just that there are more of them now, and they were never designed to distribute their internal queue state across the cluster. The system as a whole behaves more and more like a ra…

I don't understand how the system behaves more intelligently when the edge routers increase. The core routers random behavior gets worse with the larger load, and increasing the number of intelligent routers doesn't help to solve that problem in any meaningful way. Sorry, correct me if I missed something, but I believe that as the overall volume of system transactions increases (thus necessitating more "intelligent"…

Er, I wasn't trying to claim that the system behaves more intelligently. The performance degradation is just an emergent consequence of gradually adding more nodes to the routing tier. This post might be useful for some context: http://aphyr.com/posts/277-timelike-a-network-simulator

Re: Answering your questions about Heroku routing and web performance

#86

Post by adamwiggins »

josh2600 wrote: post #81
aphyr wrote: post #60

Earlier quoted context omitted.

"the switch from intelligent to randomized routing" As I understand it, Heroku (on the Bamboo stack) didn't up and decide "Hey, we're gonna switch from intelligent to random routing." The routers are still (individually) intelligent. It's just that there are more of them now, and they were never designed to distribute their internal queue state across the cluster. The system as a whole behaves more and more like a ra…

I don't understand how the system behaves more intelligently when the edge routers increase. The core routers random behavior gets worse with the larger load, and increasing the number of intelligent routers doesn't help to solve that problem in any meaningful way. Sorry, correct me if I missed something, but I believe that as the overall volume of system transactions increases (thus necessitating more "intelligent"…

> "Intelligent routing is hard, so fuck it"

Ok, let's really dig in on this. Is this truly a case of us being lazy? We just can't be bothered to implement something that would make our customers' lives better?

The answer to these questions is no.

Single global request queues have trade-offs. One of those tradeoffs is more latency per request. Another is availability on the app. Despite the sentiment here on Hacker News, most of our customers tell us that they're not willing to trade lower availability and higher latency per request for the efficiencies of a global request queue.

Are there other routing implementations that would be a happy medium between pure random (routers have no knowledge of what dynos are up to) and perfect, single global queue (routers have complete, 100% up-to-date knowledge of what dynos are up to)? Yes. We're experimenting with those; so far none have proven to be overwhelmingly good.

In the meantime, concurrent backends give the ability to run apps at scale on Heroku today; and offer other benefits, like cost efficiencies. That's why we're leaning on this area in the near term.

Re: Answering your questions about Heroku routing and web performance

#87

Post by streptomycin »

andrewem wrote: post #78
streptomycin wrote: post #53

Earlier quoted context omitted.

Even if that's true, you're just distracting from the main point. If I buy X from a company and they don't provide it, I expect a refund. Like if I order a laptop from Dell and, oops, it gets lost in the mail, it's not okay to just say "oh, we changed our shipping company, so that should happen less in the future."

Your example is extremely clear-cut, but doesn't match the situation with Heroku very closely. If Heroku promised an intelligent router, but then it was always down and never served you any requests, that would make your Dell example a good analogue. A closer comparison might be: What if Dell shipped you a laptop that they advertised as having an SSD with really good performance (and they thought that was true, or at…

I think most people would expect a refund in that situation. However, wouldn't a better analogy be like if they advertised an SSD and shipped an HDD (allegedly without knowing the performance difference), and also the diagnostic tools that they shipped just happened to not report the additional latency (allegedly without knowing that it would be a problem)? Then customers had to figure out on their own that it actually was an HDD despite their documentation and diagnostic tools saying otherwise, after wasting tons of time trying to figure out why performance sucked?

Re: Answering your questions about Heroku routing and web performance

#88

Post by adamwiggins »

obilgic wrote: post #82
adamwiggins wrote: post #79

Earlier quoted context omitted.

True, although it's more confusing than this. We didn't make an explicit decision for Bamboo, and thus the big problem — docs fell out of date, at a minimum. For Cedar, we did make an explicit decision. People on the leading edge of web development were running concurrent backends and long-running requests. Our experimental support for Node.js in 2010 was a big driver here, but also people who wanted to use concurren…

> true that makes sense, i was wondering how intelligent routing was implemented in the first place.

Oh, got it. How's this:

In the early days, Heroku only had a single routing node that sat out front. So it wasn't a distributed systems problem at that point. You could argue that Heroku circa 2009 was more of a prototype or a toy than a scalable piece of infrastructure. You couldn't run background workers, or large databases. We weren't even charging money yet.

Implementing a single global queue in a single node is trivial. In fact, this is what Unicorn (and other concurrent backends) do: put a queue within a single node, in this case a dyno. That's how we implemented it in the Heroku router (written in Erlang).

Later on, we scaled out to a few nodes, which meant a few queues. This was close enough to a single queue that it didn't matter much in terms of customer impact.

In late 2010 and early 2011 our growth started to really take off, and that's when we scaled out our router nodes to far more than a handful. And that's when the global queue effectively ceased to exist, even though we hadn't changed a line of code in the router.

The problem with this, of course, is that we didn't give it much attention because we had just launched a new product which made the explicit choice to leave out global queueing. It's this failure to continue full stewardship of our existing product that's the mistake that really hurt customers.

So to answer your question, there was never some crazy-awesome implementation of a distributed global queue that we got rid of. It was a single node's queue, a page of code in Erlang which is not too different from the code that you'll find in Unicorn, Puma, GUnicorn, Jetty, etc.

Re: Answering your questions about Heroku routing and web performance

#89

Post by adamwiggins »

nantes wrote: post #75
adamwiggins wrote: post #67

Earlier quoted context omitted.

That sucks. It may or may not be related to how the router works, but it's definitely about performance and visibility, which is what this is all about. Can you email me at adam at heroku dot com and I'll connect you with one of our Python experts? I can't promise we'll solve it, but I'd like to take a look.

I'd be really interested to see some public information resulting from debugging Python apps. We're holding pretty steady, but see a fairly constant stream of timeouts due, apparently, to variance in response times. To be sure, we're working on that. But, in the meantime, our experiments with gevent and Heroku have been less than inspiring.

I've connected Nathaniel (poster above) with one of our Python folks. Looking forward to seeing what they discover.

Would you be willing to pair up with one of our developers on your app's performance? If so email me (adam at heroku dot com).

Re: Answering your questions about Heroku routing and web performance

#90

Post by adamwiggins »

jcampbell1 wrote: post #71
ebbv wrote: post #4

It seems to me that Heroku is still failing to understand (or at least cop to) the fact that the switch from intelligent to randomized routing was a loss of a major reason people chose Heroku in the first place. A lot of Heroku's apparent value came from the intelligent routing feature. Everybody knew that it was harder to implement than random routing, that's why they were willing to pay Heroku for it. Nobody's argu…

Random routing will work fine so long as the operating system has the ability to intelligently schedule the work. The problem is that the dynos were setup to handle request sequentially. Unicorn will help mitigate the problem, but the best solution is to not try and serve webpages from hardware that is about as powerful as an old mobile phone. The 2x dynos are a step in the right direction, but I have no idea why the…

It's probably easy to guess that once we have 2X dynos, 4X dynos (and up) may be on the way. We'll drive this according to demand, so if you need/want dynos of a particular size, drop us a line.
Post reply on HN