Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku - Bamboo Routing Performance

blog.heroku.com

Post reply on HN
Print view View original

41–50 of 152 posts

Re: Heroku - Bamboo Routing Performance

#41

Post by btilly »

ibdknox wrote: post #25
benatkin wrote: post #21

Earlier quoted context omitted.

The naive approach would fail badly if they didn't have some way of supporting a healthcheck. A customizable url path (/healthcheck is often used) where the app would return 200 if things look good would work. Otherwise you may wind up with dyno that's quickly sending back 500's where not appropriate, and since it's handling lots of requests the router would keep giving it more.

I'm not sure that should be a concern at the routing layer or even necessarily a concern of heroku. It's not their job to ensure that your code isn't blowing up. That being said, health checks are nice for other reasons and could be used outside of the routing layer (which you need to sail along as quickly as possible).

The problem with that is when your code is failing to run properly due to their hardware problem - now whose fault is it?

I don't know how big they are. 50k machines? Could be off by an order of magnitude either way but I'll go with that. Suppose that your servers have, let's be generous, a 5 year mean time between failure. That's 10k machines dying every year. About 27 per day. A bit over 1 per hour.

Machines don't necessarily die cleanly. They get flaky. Bits flip. Memory gets corrupted. Network interfaces claim to have sent data they didn't. Every kind of thing that can go wrong, will go wrong, regularly. And every one of them opens you up to following "impossible" code paths where the machine still looks pretty good, but your software did something that should not be possible, and is now in a state that makes no sense. Eventually you figure it out and pull the machine.

Yeah, it doesn't happen to an individual customer too often. But it is always happening to someone, somewhere. And if you use least connections routing, many of those failures will be much, much bigger deals than they would be otherwise. And every time it happens, it was Heroku's fault.

Re: Heroku - Bamboo Routing Performance

#43

Post by btilly »

badgar wrote: post #30
seivan wrote: post #6

Wait, so those guys were on Bamboo, and complaining? Fuck, that is so not cool. We've been on cedar ever since it launched, and been running puma threads or unicorn workers. The idea of one dyno per request is bullshit, and I wasn't sure if they were on cedar or not. A dyno is an allocated resource (512mb, not counting db, k/v store etc) How ballsy of them to complain when they are doing it wrong.

> We've been on cedar ever since it launched, and been running puma threads or unicorn workers. The idea of one dyno per request is bullshit, and I wasn't sure if they were on cedar or not. A dyno is an allocated resource (512mb, not counting db, k/v store etc) It doesn't matter if you think one dyno per request is "bullshit" or not, Rails isn't multithreaded, so what do you propose they do? Using unicorn_rails on Ce…

If you pay attention to queueing theory, you'd know that even a modest amount of parallelism per worker will let you run much closer to capacity while still having very few bad request pileups.

Another way to put that is that using Cedar lets you get acceptable end user performance with far fewer dynos.

Re: Heroku - Bamboo Routing Performance

#44

Post by badgar »

richcollins wrote: post #13
seivan wrote: post #6

Wait, so those guys were on Bamboo, and complaining? Fuck, that is so not cool. We've been on cedar ever since it launched, and been running puma threads or unicorn workers. The idea of one dyno per request is bullshit, and I wasn't sure if they were on cedar or not. A dyno is an allocated resource (512mb, not counting db, k/v store etc) How ballsy of them to complain when they are doing it wrong.

Does Cedar solve the problem? Random routing can still end up overloading a dyno. What's the solution?

No, you are correct, and move to AWS.

Re: Heroku - Bamboo Routing Performance

#45

Post by WestCoastJustin »

twog wrote: post #23
salman89 wrote: post #5

Most people are going to come here and mention how they are not planning on fixing the problem. Put it into context. Heroku made this change 3 years ago, and also has had no issues admitting the change to users. Their documentation has lagged far behind and I believe they will be more transparent in the future. This is an engineering decision they made a long time ago that happened to get a lot of PR in the past 24 h…

I think this PR has already hurt Heroku & caused them to lose customers.

The only thing worse than being talked about is not being talked about - OW

You are almost investing in heroku by using their stack and tool chain, it isn't easy for well established customers to just up and move. This is probably a PR win for them, rather than a loss. Truth be told, it will be how they handle this in the coming months that will make them win/lose customers.

Re: Heroku - Bamboo Routing Performance

#46

Post by brown9-2 »

ibdknox wrote: post #18

It's a good response in that they are taking responsibility, but it is pretty obvious that they are reluctant to say anything about a fix. In my mind, "it's hard" isn't a valid excuse in this case, especially when there are relatively straightforward solutions that will solve this at a practical level. For example, you could imagine a naive form of intelligent routing that would work simply by keeping a counter per d…

Seems obvious that a naive solution wouldn't be as easy at large scale. You have to imagine that Heroku would have considered a whole lot of options before deciding on random distribution. Give them some credit at least.

Re: Heroku - Bamboo Routing Performance

#47

Post by gojomo »

ibdknox wrote: post #40
gojomo wrote: post #36

Earlier quoted context omitted.

What if their inbound routing is hundreds of machines, each of which may get a request for any of their thousands of apps, spread across tens of thousands of web dynos? Do you have a distributed sufficiently-consistent counter strategy that won't itself become a source of latency or bottlenecks or miscounts under traffic surges?

Atomic counters are pretty fast. Redis, for example, should be able to handle it without breaking a sweat: http://redis.io/topics/benchmarks

I doubt they want every inbound request to require:

• query remote redis for lowest-connection-count dyno(s) (from among potentially hundreds): 1 network roundtrip

• increment count at remote redis for chosen dyno: 1 network roundtrip (maybe can be coalesced with above?)

• when connection ends, decrement count at remote redis for chosen dyno: 1 network roundtrip

That's 2-3 extra roundtrips each inbound request, and new potential failure modes and bottlenecks around the redis instance(s). And the redis instance(s) might need retuning as operations scale and more state is needed.

Random routing lets a single loosely-consistent (perhaps distributed) table of 'up' dynos, with no other counter state, drive an arbitrarily large plant of simple, low-state routers.

Re: Heroku - Bamboo Routing Performance

#48

Post by ibdknox »

brown9-2 wrote: post #46
ibdknox wrote: post #18

It's a good response in that they are taking responsibility, but it is pretty obvious that they are reluctant to say anything about a fix. In my mind, "it's hard" isn't a valid excuse in this case, especially when there are relatively straightforward solutions that will solve this at a practical level. For example, you could imagine a naive form of intelligent routing that would work simply by keeping a counter per d…

Seems obvious that a naive solution wouldn't be as easy at large scale. You have to imagine that Heroku would have considered a whole lot of options before deciding on random distribution. Give them some credit at least.

To be clear, I'm not saying the naive solution I proposed would be "easy". Just suggesting that better solutions do exist and given that solving this very problem is one of their biggest value propositions, taking the easier way out isn't acceptable.

Re: Heroku - Bamboo Routing Performance

#49

Post by auggierose »

Well, let's put it like this. Those of us who know our programming shit and aren't afraid of a little math know exactly what has being going on here and that this answer is pretty much BS (what else is he supposed to say? basically he makes minimal concessions given the facts).

Re: Heroku - Bamboo Routing Performance

#50

Post by ibdknox »

gojomo wrote: post #47
ibdknox wrote: post #40

Earlier quoted context omitted.

Atomic counters are pretty fast. Redis, for example, should be able to handle it without breaking a sweat: http://redis.io/topics/benchmarks

I doubt they want every inbound request to require: • query remote redis for lowest-connection-count dyno(s) (from among potentially hundreds): 1 network roundtrip • increment count at remote redis for chosen dyno: 1 network roundtrip (maybe can be coalesced with above?) • when connection ends, decrement count at remote redis for chosen dyno: 1 network roundtrip That's 2-3 extra roundtrips each inbound request, and n…

My argument was built on the premise that random routing isn't acceptable given the potential slow downs it can cause (as pointed out in the Rap Genius post). If you believe otherwise, then there's no real argument for me to make :)

With that said, in your example, you could do one and two together and the response doesn't need to wait on the completion of #3. So it's one network roundtrip, which I would imagine is a tiny fraction of what they're having to do already. It is certainly another moving piece, but again my argument is that they have to have a solution and this doesn't seem infeasible.

Post reply on HN