Answering your questions about Heroku routing and web performance • Hacker News Board

Re: Answering your questions about Heroku routing and web performance

#21

Post by jfim » Wed, Apr 03, 2013, 5:53 PM UTC

hglaser wrote: post #3

This is a great response. I'm curious why something like this wasn't posted within 24 hours of RapGenius going public? I'd bet that a more thorough, technical reply would have mitigated a lot of the PR issues.

It's pretty much what they have been saying all along; Heroku switched to random routing to support other stacks than only Rails and they kept the routing infrastructure between Cedar and Bamboo the same, resulting in an undocumented performance degradation for large customers still on the old stack or running non-concurrent web frameworks on the new one. The last part was an oversight on Heroku's part, but the rest makes sense, especially considering Cedar supports other stacks than just Rails, including massively concurrent ones(Nodejs, Play, etc.).

The rest of the whole thing was people who were unsatisfied by the change and wanted the old product offering to return and/or felt shortchanged by Heroku. Luckily, Heroku doesn't lock you into their platform like GAE does, for example; it's essentially a bunch of shell scripts to deploy binaries on EC2 workers, a hosted instance of Postgres on EC2, some routing and ops as a service. Anyone who wasn't happy about the change could've contacted Heroku to say "you said we would be sold non-random routing, but you've sold us random routing" or moved their app away to another provider or even their own servers.

Does this suck if you wanted the old service? Sure. Witness the flak that the canning of Google Reader got and it's obvious that discontinuing old services isn't exactly a popular decision. On the other hand, you can self-host (it's massively cheaper than Heroku) or switch to another provider. And if there are no other providers that specialize in Rails hosting, isn't that a business opportunity?

Re: Answering your questions about Heroku routing and web performance

#22

Post by cynicalkane » Wed, Apr 03, 2013, 5:55 PM UTC

adamwiggins wrote: post #19

ebbv wrote: post #4

It seems to me that Heroku is still failing to understand (or at least cop to) the fact that the switch from intelligent to randomized routing was a loss of a major reason people chose Heroku in the first place. A lot of Heroku's apparent value came from the intelligent routing feature. Everybody knew that it was harder to implement than random routing, that's why they were willing to pay Heroku for it. Nobody's argu…

I hear you. Heroku's value proposition is that we abstract away infrastructure tasks and let you focus on your app. Keeping you from needing to deal with load balancers is one part of that. If you're worried about how routing works then you're in some way dealing with load balancing. However, if someone chose us purely because of a routing algorithm, we probably weren't a great fit for them to begin with. We're not s…

You sound like a politician talking to someone of the opposite party, in that you say "I hear you", but then completely fail to address anyone's concerns. Selling a "magic black box" that guarantees certain properties, changes them, then lies about having changed them presents a liability for users who want to do serious work.

A major selling point of Heroku is that scaling dynos wouldn't be a risk. This guarantee is now gone and it's not coming back soon even if routing behavior is reverted, because users prefer good communication and trust with their providers. The responses of Heroku are blithe non-acknowledgement acknowledgements of this problem.

Re: Answering your questions about Heroku routing and web performance

#23

Post by tvladeck » Wed, Apr 03, 2013, 5:56 PM UTC

All of the stuff that Heroku are doing now to mitigate the routing issues for their Bamboo customers are things they should have done when it first became an issue. They are not going above and beyond in any way to make up for the lost time and money their customers.

Again, this is not about

-how advanced Heroku's technology is on an absolute level

-how challenging routing is for their scale

-what competitors offer in this space and for what prices

This is only about the delta between what Heroku sold their customers and what their customers received. They are collapsing the delta now, by being honest about what they are selling (and improving their offering, it seems), but they are doing nothing to address the long time for which the delta was significant for a subset of their customers.

Re: Answering your questions about Heroku routing and web performance

#24

Post by ebbv » Wed, Apr 03, 2013, 5:58 PM UTC

adamwiggins wrote: post #19

ebbv wrote: post #4

It seems to me that Heroku is still failing to understand (or at least cop to) the fact that the switch from intelligent to randomized routing was a loss of a major reason people chose Heroku in the first place. A lot of Heroku's apparent value came from the intelligent routing feature. Everybody knew that it was harder to implement than random routing, that's why they were willing to pay Heroku for it. Nobody's argu…

I hear you. Heroku's value proposition is that we abstract away infrastructure tasks and let you focus on your app. Keeping you from needing to deal with load balancers is one part of that. If you're worried about how routing works then you're in some way dealing with load balancing. However, if someone chose us purely because of a routing algorithm, we probably weren't a great fit for them to begin with. We're not s…

I doubt anyone chose Heroku solely because of the routing algorithm, but the sales point of intelligent routing was certainly an appealing one.

Developers don't want to pay for abstraction just for abstraction's sake, they want to pay for abstraction of difficult things. Intelligent routing is one of those difficult things. Random routing is easy, which is of course why it's also more reliable, but also why you're seeing people feeling like they didn't get what they paid for.

I should be clear; this doesn't affect me personally but I am totally sympathetic with the customers who are bent out of shape about this and I still see a divide between your response and why people are upset, and I'm trying to help you bridge that divide.

Re: Answering your questions about Heroku routing and web performance

#25

Post by thinkbohemian » Wed, Apr 03, 2013, 5:59 PM UTC

boundlessdreamz wrote: post #16

adamwiggins wrote: post #13

Earlier quoted context omitted.

Yes, because that is the solution. Empirically. We've run many experiments over the past month to try other approaches to routing, including recreating the exact layout of the Bamboo routing layer (which would never scale to where we are today, but just as a point of reference). None have produced results that are anywhere near as good as using a concurrent backend. (I'd love to publish some of these results so that…

Unless the dyno signals to the router that it is busy, isn't this just postponing the problem? Per dyno, unicorn can handle more requests but requests will still get queued at dyno level if one of the requests is a slow one (say 2-3 seconds)

If only 1 request is slow and you have 7-8 unicorn workers, only one of them will stay busy. Unicorn knows which of it's workers are busy and does not queue jobs behind individual worker but rather behind the master who delegates the request to the first available worker.

Re: Answering your questions about Heroku routing and web performance

#26

Post by badgar » Wed, Apr 03, 2013, 5:59 PM UTC

> 1k req/min

Also known as trickle of traffic. Hooray for using bigger numbers and a nonstandard unit to hide inadequacy!

Does Heroku use req/min throughout their service? I can't understand why they would, unless they also can't build the infrastructure to measure on a per-second basis.

> After extensive research and experimentation, we have yet to find either a theoretical model or a practical implementation that beats the simplicity and robustness of random routing to web backends that can support multiple concurrent connections.

Does this CTO think companies like Google and Amazon route their HTTP traffic randomly? No... he knows there are scaleable routing solutions and random routing isn't the best. So he cites "simplicity and robustness." Here, this means "we can't be bothered."

Re: Answering your questions about Heroku routing and web performance

#27

Post by bhauer » Wed, Apr 03, 2013, 6:03 PM UTC

I am surprised they do not moderate the comments on their blog. They have one visible presently that is plain offensive.

Re: Answering your questions about Heroku routing and web performance

#28

Post by adamwiggins » Wed, Apr 03, 2013, 6:07 PM UTC

badgar wrote: post #26

> 1k req/min Also known as trickle of traffic. Hooray for using bigger numbers and a nonstandard unit to hide inadequacy! Does Heroku use req/min throughout their service? I can't understand why they would, unless they also can't build the infrastructure to measure on a per-second basis. > After extensive research and experimentation, we have yet to find either a theoretical model or a practical implementation that b…

Re: requests. RPM is the metric that New Relic reports, and it's the one most of our customers use when they talk about traffic. I try to speak in whatever terms are most familiar to our customers.

Re: I can't speak to Google and Amazon, and they aren't representative of the size of our customers anyway. We have discussed with many folks who run ops at many companies that are more on par with the size of our mid- and large-sized customers, and single global request queues are exceedingly rare.

The most common setup seems to be clusters of web backends (say, 4 clusters of 40 processes each) that each have their own request queue, but with random load balancing across those clusters. This is a reasonable happy medium between pure random and global request queue, and isn't too different from what you get running (say) 16 processes inside a 2X dyno and 8 web dynos.

Re: Answering your questions about Heroku routing and web performance

#29

Post by chc » Wed, Apr 03, 2013, 6:09 PM UTC

ebbv wrote: post #18

tptacek wrote: post #7

Earlier quoted context omitted.

Is anybody operating at Heroku's scale offering centralized request routing queues? At what price?

Not that I know of but that's why I'm saying it would be a premium product. Likely pricing would have to scale with the number of dynos running behind the router. But that's the service people thought they were getting and what they wanted. If Heroku prices out the intelligent routing and says; "Ok you can have intelligent routing with your current backend stack, but it's going to cost you $25/mo for evert 10 dynos,…

$25 a month? Did you forget a few zeroes?

Re: Answering your questions about Heroku routing and web performance

#30

Post by adamwiggins » Wed, Apr 03, 2013, 6:09 PM UTC

ebbv wrote: post #24

adamwiggins wrote: post #19

Earlier quoted context omitted.

I hear you. Heroku's value proposition is that we abstract away infrastructure tasks and let you focus on your app. Keeping you from needing to deal with load balancers is one part of that. If you're worried about how routing works then you're in some way dealing with load balancing. However, if someone chose us purely because of a routing algorithm, we probably weren't a great fit for them to begin with. We're not s…

I doubt anyone chose Heroku solely because of the routing algorithm, but the sales point of intelligent routing was certainly an appealing one. Developers don't want to pay for abstraction just for abstraction's sake, they want to pay for abstraction of difficult things. Intelligent routing is one of those difficult things. Random routing is easy, which is of course why it's also more reliable, but also why you're se…

Well said.

It's interesting — very few customers are actually bent out of shape about this. (A few are, for sure.) It's more non-customers who are watching from the sidelines that seem to be upset. I do want to try to explain ourselves to the community in general, and that's what this post was for. But my first loyalty is to serving our current customers well.