Heroku Blog: Routing Performance Update • Hacker News Board

Re: Heroku Blog: Routing Performance Update

#11

Post by trafficlight » Sat, Feb 16, 2013, 6:49 AM UTC

seldo wrote: post #5

Not to be too harsh, but I'm not sure whether "we had no idea it was so bad" is better or worse than "we knew it was bad, but didn't tell anybody" for a platform company. The tone of the post is appropriately apologetic, but this does make you wonder what other problems they're missing.

As a long time NetAdmin for a small WISP I understand where they are coming from. Even with monitoring and health checks, there are a lot of unknowns on the network. Sometimes I don't know there's a problem until a customer calls to complain. And even when they do complain, it can be tough to identify a root cause even with all the data we collect.

We try to do our best to be proactive, but sometimes you don't even know to look for something until it's a problem. Heroku has quite a few more moving parts than our modest network with a large percentage of that in the form of code written by thousands of other people. As long as Heroku is learning lessons from their failures, which I believe is the case, they are doing a good job.

Re: Heroku Blog: Routing Performance Update

#12

Post by csense » Sat, Feb 16, 2013, 6:49 AM UTC

It sounds like they have one routing cluster for all of Heroku. If this is the case, and large routing clusters are the problem's root cause, they should just shard the cluster.

I.e., give their bigger customers like RapGenius (who said they pay Heroku $20k/month and whose HN post spurred this debate) their own dedicated routing cluster with 2-5 nodes. Once a single customer exceeds that size, they're probably paying Heroku enough that Heroku can afford to devote an engineer to working specifically with them to implement large-scale architecture choices like running the app's DB on a cluster, etc.

Pile several smaller customers on a shared routing cluster to cut costs and keep the cluster utilization high, but once the cluster gets to be a certain size (reaches a fixed number of backend dynos or metrics get bad enough), start putting customers on a new cluster.

It should be fairly trivial to use DNS or router rules to dynamically move existing customers from one routing cluster to another.

Re: Heroku Blog: Routing Performance Update

#13

Post by bunkat » Sat, Feb 16, 2013, 6:49 AM UTC

jkat wrote: post #7

Depending on what side of Hanlon's razor you fall, the only conclusion I get from this is that they are either incompetent or dishonest. I have a very hard time believing that this issue remained unknown to them for years. As for the post, it's pretty much just documentation. I didn't see any apology. And the only promise of a better tomorrow is a vague "Working to better support concurrent-request Rails apps on Ceda…

I also didn't see any mention of refunds for all of the extra dynos that were needed due to the degrading performance of their service - or all the extra support hours where they told everyone 'not our problem!'.

Re: Heroku Blog: Routing Performance Update

#14

Post by jlgaddis » Sat, Feb 16, 2013, 6:50 AM UTC

jkat wrote: post #7

Depending on what side of Hanlon's razor you fall, the only conclusion I get from this is that they are either incompetent or dishonest. I have a very hard time believing that this issue remained unknown to them for years. As for the post, it's pretty much just documentation. I didn't see any apology. And the only promise of a better tomorrow is a vague "Working to better support concurrent-request Rails apps on Ceda…

> ... they are either incompetent or dishonest ...

Exactly. If they didn't know, they should have. If they did know, well, ...

Re: Heroku Blog: Routing Performance Update

#15

Post by Shank » Sat, Feb 16, 2013, 6:53 AM UTC

From the perspective of someone who might be looking at Heroku as a host in the future, this is a bit scary. Their response appears to be mostly apologetic in that they're sorry that it happened - but does nothing to address the issue. It's more of a "we screwed up, oh well" than anything else.

They would have warranted a better response if they said they were actively looking into how to improve the routing system, but by the looks of things they're going to sit by and hope developers switch practices so they don't have to solve their problem.

Re: Heroku Blog: Routing Performance Update

#16

Post by jeswin » Sat, Feb 16, 2013, 6:54 AM UTC

I'm not able to follow parts of the post. Our routing cluster remained small for most of Bamboo’s history, which masked this inefficiency.

If you went from 1 router to 2, 50% of routers can't optimally route a request. If you went from 2 to 3, you would have 66% which can't route. 3 to 4, 75%.

Once you get to say 10 routers, you are already at 90% sub-optimal routing. So are they saying, the had only 1 or 2 routers earlier?

Re: Heroku Blog: Routing Performance Update

#17

Post by beambot » Sat, Feb 16, 2013, 6:56 AM UTC

This is still not ideal. Even if you're running unicorn, you're still susceptible to queueing spikes due to random load balancing. The concurrency just gives you a small buffer and/or some smoothing on 95th percentile responses. Right?

At least there's a commitment to update the reporting tools... getting bad data in New Relic was (IMHO) the worst -- even worse than out-of-date docs.

Re: Heroku Blog: Routing Performance Update

#18

Post by fatbird » Sat, Feb 16, 2013, 7:04 AM UTC

trafficlight wrote: post #11

seldo wrote: post #5

Not to be too harsh, but I'm not sure whether "we had no idea it was so bad" is better or worse than "we knew it was bad, but didn't tell anybody" for a platform company. The tone of the post is appropriately apologetic, but this does make you wonder what other problems they're missing.

As a long time NetAdmin for a small WISP I understand where they are coming from. Even with monitoring and health checks, there are a lot of unknowns on the network. Sometimes I don't know there's a problem until a customer calls to complain. And even when they do complain, it can be tough to identify a root cause even with all the data we collect. We try to do our best to be proactive, but sometimes you don't even k…

In this whole sad, sorry tale, the problem is that the Heroku support engineers clearly knew, and communicated to the customer, that the unexpected/unreported lag was caused by the random queuing. They knew what the problem was for three years; it was only when a big customer went very public that they adopted the right tone and action plan.

I imagine there's great discomfort inside Heroku these days because somewhere between the line engineers who knew about this issue, and the CEO, the fact that customers were complaining was swallowed up.

Re: Heroku Blog: Routing Performance Update

#19

Post by fatbird » Sat, Feb 16, 2013, 7:07 AM UTC

csense wrote: post #12

It sounds like they have one routing cluster for all of Heroku. If this is the case, and large routing clusters are the problem's root cause, they should just shard the cluster. I.e., give their bigger customers like RapGenius (who said they pay Heroku $20k/month and whose HN post spurred this debate) their own dedicated routing cluster with 2-5 nodes. Once a single customer exceeds that size, they're probably paying…

The problem, as documented by the customer who went public with this issue, is that their request distribution scheme went from intelligent (i.e., load-based) to random, and a random distribution of requests is almost guaranteed to cause significant queuing for some non-trivial number of requests unless one has an absurd amount of extra capacity in place already, with ruinous financial aspects.

Re: Heroku Blog: Routing Performance Update

#20

Post by chubot » Sat, Feb 16, 2013, 7:07 AM UTC

Well, this certainly calls into question their competence. They're a PaaS company that doesn't understand or measure their load balancing performance.

If you are a PaaS company, and you only have 5 metrics you can record, then 99% percentile latency across all apps should be one of them.

On another note: why is Rails single-threaded??? That seems unbelievable. So if you have a 2 second database query, your Rails process does nothing else for that 2 seconds? I mean people complain about the GIL in Python, which actually has reasons behind it, but this is just crazy.