Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku - Bamboo Routing Performance

blog.heroku.com

Post reply on HN
Print view View original

31–40 of 152 posts

Re: Heroku - Bamboo Routing Performance

#31

Post by apinstein »

kevinfat wrote: post #19

Can someone explain, to people who know nothing about scaling infrastructure, why routing to idle dynos is a hard problem?

Because to be intelligent, you have to have the router talk to all the dynos to calculate load. Doing that in a performant way can get tricky, especially since people can hit a button and get 100 workers. The bigger the n, the more resources are required to track everything and the more things can go wrong.

It's not an intractable problem, but it's not trivial, affects only a small percentage of customers, and introduces complexity for everyone.

I feel pretty confident that there is a reasonable solution, and as someone that just spent the last 3 weeks building a custom buildpack and a new heroku app for an auto-scaling worker farm, I am happy to see such a quick, hopeful response.

Re: Heroku - Bamboo Routing Performance

#32

Post by tomlemon »

seivan wrote: post #6

Wait, so those guys were on Bamboo, and complaining? Fuck, that is so not cool. We've been on cedar ever since it launched, and been running puma threads or unicorn workers. The idea of one dyno per request is bullshit, and I wasn't sure if they were on cedar or not. A dyno is an allocated resource (512mb, not counting db, k/v store etc) How ballsy of them to complain when they are doing it wrong.

Rap Genius cofounder:

Random routing is still wayyyyy worse than intelligent routing even if you're processing multiple requests simultaneously on a single dyno (see http://rapgenius.com/1504222)

Requests will still queue, but since time spent queuing at the dyno level doesn't appear in your logs or on New Relic, you'll never know it.

Re: Heroku - Bamboo Routing Performance

#33

Post by gojomo »

kevinfat wrote: post #19

Can someone explain, to people who know nothing about scaling infrastructure, why routing to idle dynos is a hard problem?

It requires statefulness and decisionmaking at the routing layer, and that's another thing that adds overhead and can go wrong at scale. (For example, there may be no one place with knowledge of all in-process requests. Traffic surges may lead to an arbitrary growth of state in the routing layer, rather than at the dynos.)

There are probably some simple techniques whereby dynos can themselves approximate the throughput of routing-to-idle, while Heroku's load-balancers continue to route randomly. For example, if a 'busy' dyno could shed a request, simply throwing it back to get another random assignment, most jam-ups could be alleviated until most dynos are busy. (And even then, the load could be spread more evenly, even in the case of some long requests and unlucky randomization.) Heroku may just need to coach their customers in that direction.

Re: Heroku - Bamboo Routing Performance

#34

Post by benatkin »

tyler_grady wrote: post #26

Is it me not understanding disqus, or did Heroku's moderator just deleted my comment?

I think I saw your comment and that they must have deleted it. Apparently their idea of keeping it civil means keeping out links to the blog post that it was a response to.

Re: Heroku - Bamboo Routing Performance

#35

Post by ibrahima »

I think this is really a fine response considering the pretty terrible way the original post was written and the community responded. The simulation was a bit of a stretch because the supposed number of servers you need to achieve "equivalent" performance is highly dependent on how slow your worst case performance is, and if your worst case isn't that bad the numbers look a lot better. Don't remember the precise math, but back when I studied random processes we studied this problem and the conclusion was that randomly routing requests is generally not that much worse than doing the intelligent thing, and doing the intelligent thing is nowhere near as trivial as Rapgenius and random HN posters would have you believe. Given generally well behaved requests he random solution should be maybe 2-3x worse but nothing near 50x worse.

And besides, I really don't see why someone who needs that many dynos is still on Heroku.

Re: Heroku - Bamboo Routing Performance

#36

Post by gojomo »

ibdknox wrote: post #18

It's a good response in that they are taking responsibility, but it is pretty obvious that they are reluctant to say anything about a fix. In my mind, "it's hard" isn't a valid excuse in this case, especially when there are relatively straightforward solutions that will solve this at a practical level. For example, you could imagine a naive form of intelligent routing that would work simply by keeping a counter per d…

What if their inbound routing is hundreds of machines, each of which may get a request for any of their thousands of apps, spread across tens of thousands of web dynos?

Do you have a distributed sufficiently-consistent counter strategy that won't itself become a source of latency or bottlenecks or miscounts under traffic surges?

Re: Heroku - Bamboo Routing Performance

#37

Post by tomlemon »

ibrahima wrote: post #35

I think this is really a fine response considering the pretty terrible way the original post was written and the community responded. The simulation was a bit of a stretch because the supposed number of servers you need to achieve "equivalent" performance is highly dependent on how slow your worst case performance is, and if your worst case isn't that bad the numbers look a lot better. Don't remember the precise math…

Rap Genius cofounder:

> The simulation was a bit of a stretch because the supposed number of servers you need to achieve "equivalent" performance is highly dependent on how slow your worst case performance is, and if your worst case isn't that bad the numbers look a lot better

It's still pretty bad. Here's a graph of the relative performances of the different routing strategies when your response times are much better (50%: 50ms, 99%: 537ms, 99.9%: 898ms)

http://s3.amazonaws.com/rapgenius/1360871196_routerstyles_fa...

See http://rapgenius.com/1504222 for more

Re: Heroku - Bamboo Routing Performance

#38

Post by res0nat0r »

twog wrote: post #23
salman89 wrote: post #5

Most people are going to come here and mention how they are not planning on fixing the problem. Put it into context. Heroku made this change 3 years ago, and also has had no issues admitting the change to users. Their documentation has lagged far behind and I believe they will be more transparent in the future. This is an engineering decision they made a long time ago that happened to get a lot of PR in the past 24 h…

I think this PR has already hurt Heroku & caused them to lose customers.

How many customers are running on a stack that old that are at the scale being complained about? It is probably cheaper to tell them to upgrade if they want better performance, or if they dont want to/ can't then they unfortunately can't be officially supported.

Re: Heroku - Bamboo Routing Performance

#39

Post by stonewhite »

kevinfat wrote: post #19

Can someone explain, to people who know nothing about scaling infrastructure, why routing to idle dynos is a hard problem?

Assuming you keep your prices roughly the same, the computation needed for assigning an available dyno becomes exponentially harder(costlier) as more dynos are added to the ecosystem. Thats why they changed the intelligent routing to random routing; to save cpu cycles.

The decision made by heroku was not an engineering decision, it was a business decision. While it is quite a bit frustrating, it is understandable and I don't think it will change. Since reverting a whole infrastructure to its original no-longer-profitable position is generally not a smart move.

They'll just abandon rails devs and move on.

Re: Heroku - Bamboo Routing Performance

#40

Post by ibdknox »

gojomo wrote: post #36
ibdknox wrote: post #18

It's a good response in that they are taking responsibility, but it is pretty obvious that they are reluctant to say anything about a fix. In my mind, "it's hard" isn't a valid excuse in this case, especially when there are relatively straightforward solutions that will solve this at a practical level. For example, you could imagine a naive form of intelligent routing that would work simply by keeping a counter per d…

What if their inbound routing is hundreds of machines, each of which may get a request for any of their thousands of apps, spread across tens of thousands of web dynos? Do you have a distributed sufficiently-consistent counter strategy that won't itself become a source of latency or bottlenecks or miscounts under traffic surges?

Atomic counters are pretty fast. Redis, for example, should be able to handle it without breaking a sweat: http://redis.io/topics/benchmarks
Post reply on HN