Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku - Bamboo Routing Performance

blog.heroku.com

Post reply on HN
Print view View original

121–130 of 152 posts

Re: Heroku - Bamboo Routing Performance

#121

Post by antoko »

fauigerzigerk wrote: post #89
antoko wrote: post #9

That's actually a pretty impressive response as far as it goes. Obviously there's no details at this point, but he absolutely takes responsibility, doesn't try to deflect or sugar coat it, and manages to find a tone that is both professional/serious, yet also down-to-earth and earnest. I guess the real impact will be how they go about "making it right" but in terms of a first response to the situation the tone is nea…

I would reserve judgement on this response until we learn the truth. Whether or not a response is adequate depends on what they could have said and should have said, which obviously depends on the facts. So let's just wait for more information until we assign praise or blame.

My comment is 3 sentence long and has 3 caveats. I am definitely reserving judgement, my praise was mostly about the tone, which I maintain is near perfect. I'm really curious if it was a case of him just writing it because a response hadn't been forthcoming and there needed to be one or if there was high-level meetings dissecting every word and phrase to make it just so - or where on that continuum it fell.

Re: Heroku - Bamboo Routing Performance

#122

Post by antoko »

kami8845 wrote: post #107
antoko wrote: post #9

That's actually a pretty impressive response as far as it goes. Obviously there's no details at this point, but he absolutely takes responsibility, doesn't try to deflect or sugar coat it, and manages to find a tone that is both professional/serious, yet also down-to-earth and earnest. I guess the real impact will be how they go about "making it right" but in terms of a first response to the situation the tone is nea…

No this is not impressive. This is them fucking up and misleading customers for 3 years, enjoying a great reputation and now FINALLY getting called out for their BS. They're about to lose that great reputation that they've spent the past years building up, so of course they're in major crisis mode and doing everything they can to fix this.

Your response doesn't really seem to be directed at my comment. In your first sentence you're using "this" as if you're talking about the blog post which is what I praise in my comment(the tone of it), but then in your second sentence "this" refers to the routing issue raised by rapgenius, I wasn't commenting on that at all - I'm not a Heroku customer I was merely commenting on a GM's response to negative press.

Re: Heroku - Bamboo Routing Performance

#123

Post by alberth »

It seems strange for me to read in Heroku's response how forthcoming they are to accept blame and responsibility for the "a degradation in performance over the past 3 years".

Yet they state their action plan to "fix" this issue is to update their DOCUMENTATION and no mention of fixing the DEGRADATION issues itself.

Just bizarre.

Re: Heroku - Bamboo Routing Performance

#124

Post by mononcqc »

oillio wrote: post #69
apinstein wrote: post #31

Earlier quoted context omitted.

Because to be intelligent, you have to have the router talk to all the dynos to calculate load. Doing that in a performant way can get tricky, especially since people can hit a button and get 100 workers. The bigger the n, the more resources are required to track everything and the more things can go wrong. It's not an intractable problem, but it's not trivial, affects only a small percentage of customers, and introd…

I don't think this only affects a small percentage of customers. Maybe only a small percentage will notice. This will affect any user with a high standard deviation in their application's response time. Lets take an extreme case as an example: An application that has an average response time of 100ms, however 1% of the responses have a 3s response time. They have relatively small load and they only have 2 web dynos r…

If you can add even one worker per dyno, things get much better, faster. See http://rapgenius.com/1504221/James-somers-herokus-ugly-secre... for details.

The really big problem there is having dynos that can handle only one request at a time.

Re: Heroku - Bamboo Routing Performance

#125

Post by mononcqc »

oillio wrote: post #76
stonewhite wrote: post #39

Earlier quoted context omitted.

Assuming you keep your prices roughly the same, the computation needed for assigning an available dyno becomes exponentially harder(costlier) as more dynos are added to the ecosystem. Thats why they changed the intelligent routing to random routing; to save cpu cycles. The decision made by heroku was not an engineering decision, it was a business decision. While it is quite a bit frustrating, it is understandable and…

A simple intelligent router may have 2^n performance, but I cannot imagine there are not other solutions that would bring performance closer to linear. I am not a queue theory expert, but I know they are out there. This is a well studied area of CS. This is not only a problem for non-concurrent applications. It will become a problem as dyno usage increases for any application. The major factors will be the standard d…

What I see in that graph's purple line is that with the most basic concurrency you can have that is not nil, you get better times than some statistically optimal load-distributing solution (choice of two). I'd like to see similar graphs with 4, 8, or more workers to compare them.

Re: Heroku - Bamboo Routing Performance

#126

Post by sophacles »

GhotiFish wrote: post #52
ibrahima wrote: post #35

I think this is really a fine response considering the pretty terrible way the original post was written and the community responded. The simulation was a bit of a stretch because the supposed number of servers you need to achieve "equivalent" performance is highly dependent on how slow your worst case performance is, and if your worst case isn't that bad the numbers look a lot better. Don't remember the precise math…

I am very interested in hearing the intricacies and gotchas this problem presents. Do you know where I can read up on this topic?

Generally, this is a fall out from queueing theory. I don't have access to my library right now, but any decent process simulation text will give you insight into the math of modeling this sort of thing.

Re: Heroku - Bamboo Routing Performance

#127

Post by chrishoney »

spiralganglion wrote: post #66
redguava wrote: post #62

I don't understand why people think this is a great response. They know how their routing works, just say so. It can't be that hard to give a basic overview of it before they release a more comprehensive post. As for the comment "Improving our documentation and website to accurately reflect our product". That is a very round about way of saying "our website indicates our service does things that it does not" which is…

Documentation discrepancies happen. I've seen them with pretty much every platform I've worked on. Just yesterday, I found a critical discrepancy between the ActionScript documentation and the actual behaviour of the ActionScript compiler, costing my team a day of work. (I tried to report the issue to Adobe, but the Adobe Bug Reporting System was down. Perhaps they need a Bug Reporting System for the Bug Reporting Sy…

The thing that makes me nervous is Heroku obviously knew about this issue, yet they chose to do nothing about it until they were called out on it by a high-visibility customer.

Re: Heroku - Bamboo Routing Performance

#128

Post by ceejayoz »

7952 wrote: post #119
programminggeek wrote: post #114

Wow, I feel like Heroku is really dropping the ball here. Like, they are acting punch drunk or something. Basically all this says is "we hear you and we are sorry". They could have posted that a day ago. This still says nothing about what is wrong and what they are doing to fix it. Also, I'm not sure at what point this is, but at some point around say $3-5k a month, (100+ dynos) you really should rethink using Heroku…

At $3-5k a month Heroku may as well start offering a consultancy service rather than hosting. Wanting unlimited scaling without needing local talent is a reasonable thing to want, but its unrealistic to expect if from one single platform.

A lot of their success stories at http://success.heroku.com/ are sites you'd expect to be spending the $3-5k/month.

If their platform can't handle higher amounts of load, they really should indicate as such.

Re: Heroku - Bamboo Routing Performance

#129

Post by jrochkind1 »

badgar wrote: post #30
seivan wrote: post #6

Wait, so those guys were on Bamboo, and complaining? Fuck, that is so not cool. We've been on cedar ever since it launched, and been running puma threads or unicorn workers. The idea of one dyno per request is bullshit, and I wasn't sure if they were on cedar or not. A dyno is an allocated resource (512mb, not counting db, k/v store etc) How ballsy of them to complain when they are doing it wrong.

> We've been on cedar ever since it launched, and been running puma threads or unicorn workers. The idea of one dyno per request is bullshit, and I wasn't sure if they were on cedar or not. A dyno is an allocated resource (512mb, not counting db, k/v store etc) It doesn't matter if you think one dyno per request is "bullshit" or not, Rails isn't multithreaded, so what do you propose they do? Using unicorn_rails on Ce…

> It doesn't matter if you think one dyno per request is "bullshit" or not, Rails isn't multithreaded

Sure it is. You just have to set `config.threadsafe!`, which right now is slated to be on by default in rails4.

But then you need a server stack that supports multi-threaded requests, such as puma which the parent comment mentioned using.

And then if you're using MRI, your multiple threads in a process still can't use more than one cpu at a time -- but this _probably_ doesn't matter in the scenario we're talking about, because a single heroku dyno isn't going to have access to more than one cpu at a time anyway, right?

How this all shakes out is indeed confusing. I am _not_ saying "If you just use puma and config.threadsafe!" all is well with their scheduler. I do not know one way or the other, it is hella confusing.

But it is incorrect to say "Rails isn't multithreaded, so what do you propose they do"

Re: Heroku - Bamboo Routing Performance

#130

Post by thenduks »

damian2000 wrote: post #4

So the issue only affects Bamboo? that's what it seems to be saying

It does seem to be what they are saying but, unfortunately, no :(

Random request routing is also present on Cedar [1]. The difference is that, on Cedar, you can easily run multi-threaded or even multi-process apps (the latter being harder due to a 512mb memory limit) which can mitigate the problem, but does not solve it. Modifying your app so all of your requests are handled extremely quickly also mitigates the problem, but does not solve it.

Seems to me the obvious solution is to do these things (multi-threaded app server, serve only/mostly short requests) and use at least a somewhat intelligent routing algorithm (perhaps 'least connections' would make sense).

[1] - https://devcenter.heroku.com/articles/http-routing#request-d...

Post reply on HN