Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku - Bamboo Routing Performance

blog.heroku.com

Post reply on HN
Print view View original

131–140 of 152 posts

Re: Heroku - Bamboo Routing Performance

#131

Post by gingerlime »

tomlemon wrote: post #37
ibrahima wrote: post #35

I think this is really a fine response considering the pretty terrible way the original post was written and the community responded. The simulation was a bit of a stretch because the supposed number of servers you need to achieve "equivalent" performance is highly dependent on how slow your worst case performance is, and if your worst case isn't that bad the numbers look a lot better. Don't remember the precise math…

Rap Genius cofounder: > The simulation was a bit of a stretch because the supposed number of servers you need to achieve "equivalent" performance is highly dependent on how slow your worst case performance is, and if your worst case isn't that bad the numbers look a lot better It's still pretty bad. Here's a graph of the relative performances of the different routing strategies when your response times are much bette…

Disclaimer: I've never used heroku :)

if I understand the chart correctly, using unicorn with two workers gets you pretty close to intelligent routing with no intelligence. I imagine adding up to three or four would make things even better... I don't know about puma/thin etc where you can perhaps crank it even further without too much memory tax(?)

To me this seems like the easiest approach for everybody concerned. Heroku can keep using random routing without adding complexity, and most users will not get affected if they are able to split the workload on the dyno-level.

On a slight tangent: On the rails app I'm working on I'm trying to religiously offload anything that might block or take too long to a resque task. It's not always feasible, but I think it's a good common-sense approach to try to avoid bottlenecks.

Re: Heroku - Bamboo Routing Performance

#132

Post by kami8845 »

antoko wrote: post #122
kami8845 wrote: post #107

Earlier quoted context omitted.

No this is not impressive. This is them fucking up and misleading customers for 3 years, enjoying a great reputation and now FINALLY getting called out for their BS. They're about to lose that great reputation that they've spent the past years building up, so of course they're in major crisis mode and doing everything they can to fix this.

Your response doesn't really seem to be directed at my comment. In your first sentence you're using "this" as if you're talking about the blog post which is what I praise in my comment(the tone of it), but then in your second sentence "this" refers to the routing issue raised by rapgenius, I wasn't commenting on that at all - I'm not a Heroku customer I was merely commenting on a GM's response to negative press.

Yeah, it's still not impressive. The only thing that I'd say is impressive in response to the situation would be:

"Hi, CEO of Heroku here.

Sorry. We've been misleading customers and only telling the truth when pressed-hard for years. We've created this financial model for all of our customers who have been overpaying on dynos because of our shitty routing and will be reimbursing them based on that.

We've also rolled out a second dyno-tier, called "dyno with non-shitty routing". It's 10x as expensive, but at least we're being honest about it. All current customers will enjoy our "dyno with non-shitty routing" for the price they're currently paying for the next 2 years. Enough time for them to migrate away, like any reasonable person would expect them to after this."

Re: Heroku - Bamboo Routing Performance

#133

Post by podperson »

Seems to me like a classy response to a real problem from Heroku.

We all need to remember that there are no magic bullets. The fact that Heroku can get a startup to, say, 5M uniques per day by dragging some sliders on a web panel and running up a bill on a corporate AMEX is pretty impressive.

At some point scaling a web business becomes a core competency and one needs to deal with it. I'm guessing by the time scaling an app on Heroku becomes an issue, if better understanding your scaling needs and handling them directly isn't going to save you a TON of money, your business model is probably broken.

Re: Heroku - Bamboo Routing Performance

#134

Post by callum85 »

gojomo wrote: post #12
RaphiePS wrote: post #2

Interesting -- he seems to be saying that they'll explain all about the problem, but not do anything about it.

That's not fair. What do you think "tools to understand and improve the performance of your apps" and "develop long-term solutions" from his bullets mean? But, I'm surprised they didn't wait until the "in-depth technical review" was available to apologize. And the idea that they were informed of a problem "yesterday" doesn't quite match the impression RapGenius gave, that they'd been discussing this with Heroku suppo…

I think RaphiePS's comment is fair.

"tools to understand and improve the performance of your apps" only commits them to updating their docs and tools to reflect how their system really works. It doesn't indicate any intention to fix the actual problem (the fact that requests can be routed to busy dynos), nor that they will make any kind of reimbursement to people who made business decisions based on incorrect docs.

"develop long-term solutions" doesn't really mean anything.

Re: Heroku - Bamboo Routing Performance

#136

Post by saraid216 »

redguava wrote: post #74
redguava wrote: post #70

Earlier quoted context omitted.

In answer to "who cares if the explanation comes today or tomorrow", I care if the explanation comes today or tomorrow. I use Heroku and have hit scaling issues in the last few weeks very similar to this. More information on what is going on behind the scenes will help me immediately. As for discrepancy in documentation, this is one of the most major parts of their infrastructure and directly relates to how well appl…

@bitcartel (not sure why I couldn't reply to your comment)... I believe from my discussions with Heroku that the "random selection algorithm" is also not true.

> (not sure why I couldn't reply to your comment)

FYI, this is probably because you saw his comment at around 0-1 minutes. If you click on "link", you'll get the Reply form... or refresh after a minute or two and the "reply" link will be available.

Re: Heroku - Bamboo Routing Performance

#137

Post by camus »

WestCoastJustin wrote: post #45
twog wrote: post #23

Earlier quoted context omitted.

I think this PR has already hurt Heroku & caused them to lose customers.

The only thing worse than being talked about is not being talked about - OW You are almost investing in heroku by using their stack and tool chain, it isn't easy for well established customers to just up and move. This is probably a PR win for them, rather than a loss. Truth be told, it will be how they handle this in the coming months that will make them win/lose customers.

If you are locked in the trunk with any cloud solution , then you are a bad programmer/syst admin/whatever taking bad decisions, period.

You should be able to move your project infrastructure quickly from a service to another ,if you cant to that, well too bad when your infrastructure fails...

Re: Heroku - Bamboo Routing Performance

#138

Post by dragonwriter »

alberth wrote: post #123

It seems strange for me to read in Heroku's response how forthcoming they are to accept blame and responsibility for the "a degradation in performance over the past 3 years". Yet they state their action plan to "fix" this issue is to update their DOCUMENTATION and no mention of fixing the DEGRADATION issues itself. Just bizarre.

> Yet they state their action plan to "fix" this issue is to update their DOCUMENTATION and no mention of fixing the DEGRADATION issues itself.

This is flat out untrue. The third bullet point in their action plan is to update their documentation, and the fifth is "Working closely with our customers to develop long-term solutions".

Updating the documentation to accurately reflect what the platform does is obviously critical to allow people to make decisions and manage applications on the platform as it is, so is an important and immediate part of the action plan.

Long-term fixes to the problem are also important, and are explicitly part of the action plan. Its clear that they haven't identified what those solutions are, but its not at all true that they haven't mentioned them as part of the action plan.

Re: Heroku - Bamboo Routing Performance

#139

Post by dragonwriter »

raylu wrote: post #27
timothya wrote: post #11

Working closely with our customers to develop long-term solutions Of the five action items they listed, it seems that only the last of them is about actually solving the problem. I hope they are committed to it - better visibility of the problem can help, but I'd rather not have the problem in the first place.

Actually, that's the only one that smells like BS to me. The others have clear meaning and goals.

The other ones are things that are obvious and immediate responses to the problem on description that don't take any deep analysis of alternatives.

Long-term fixes actually do require deep analysis of alternatives (and even what the appropriate parameters are for a solution that will deal with customers problems while maintaining Heroku's scalability), and aren't something you can do much more than make vague references to off the cuff.

The key question on that point will be follow-through.

Re: Heroku - Bamboo Routing Performance

#140

Post by dragonwriter »

RaphiePS wrote: post #2

Interesting -- he seems to be saying that they'll explain all about the problem, but not do anything about it.

Actually he is, quite explicitly, saying that they will do a short-term description of the current behavior and its impact, provide tools to assist customers in understanding and managing the effect on their specific apps, and work on long-term fixes.
Post reply on HN