Answering your questions about Heroku routing and web performance • Hacker News Board

Re: Answering your questions about Heroku routing and web performance

#151

Post by marshray » Thu, Apr 04, 2013, 3:00 PM UTC

twic wrote: post #150

marshray wrote: post #139

Earlier quoted context omitted.

> So to answer your question, there was never some crazy-awesome implementation of a distributed global queue that we got rid of. So it was an oversimplified system that worked great but wasn't scalable and was at some point going to completely fall over under increasing load. IMExp, this is not a wrong thing to build initially and it's not wrong to replace it either. But the replacement is going to have a hard time…

Not a wrong thing to build initially, but perhaps a wrong thing to advertise a feature based on, unless you have a plan for how to continue to deliver that feature as you scale up.

Everybody who scales rapidly has some growing pains, so I'm sympathetic. But I agree that by advertising that as a feature they're specifically asking customers to outsource this hard problem to them.

Re: Answering your questions about Heroku routing and web performance

#152

Post by twic » Thu, Apr 04, 2013, 4:24 PM UTC

adamwiggins wrote: post #54

wmf wrote: post #45

Earlier quoted context omitted.

Amazon ELB? It does cost significantly more than Heroku AFAIK.

My understanding is that ELBs are HAProxy, and they may be set to use the leastconn algorithm (a global request queue that is friendly to concurrent backends). However, once you get any amount of traffic they start to scale out the nodes in the ELB, which produces essentially the same results as the degradation of the Bamboo router that we've documented. The difference, of course, is that ELBs are single-tenant. So a…

I hope you'll be able to share your findings with us, even if they're negative. As someone who has no stake in Heroku, i have the luxury of finding this problem simply interesting!

My hypothesis is that tenant-specific intelligent load balancers would be plausible; i would guess that you would never need more than a handful of HAProxy or nginx-type balancers to front even a large application. Your main challenge would then be routing requests to the right load balancer cluster. If you had your own hardware, LVS could handle that (i believe that Wikipedia in 2010 ran all page text requests through a single LVS in each datacentre), but i'm not sure what you do on EC2.

However, "hypothesis" is just a fancy way of saying "guess", which is why your findings from actual experiments would be so interesting.

Re: Answering your questions about Heroku routing and web performance

#153

Post by adamwiggins » Thu, Apr 04, 2013, 6:02 PM UTC

perucoder wrote: post #129

adamwiggins wrote: post #110

Earlier quoted context omitted.

We did.

When did this happen? As of March 2, Rap Genius was still seeking to get money refunded. http://venturebeat.com/2013/03/02/rap-genius-responds/

Just in the last few weeks. I won't disclose details for any particular customer, but I assume that Tom @ RG will make a public statement about it at some point.

Re: Answering your questions about Heroku routing and web performance

#154

Post by adamwiggins » Thu, Apr 04, 2013, 6:18 PM UTC

moe wrote: post #131

adamwiggins wrote: post #86

Earlier quoted context omitted.

> "Intelligent routing is hard, so fuck it" Ok, let's really dig in on this. Is this truly a case of us being lazy? We just can't be bothered to implement something that would make our customers' lives better? The answer to these questions is no. Single global request queues have trade-offs. One of those tradeoffs is more latency per request. Another is availability on the app. Despite the sentiment here on Hacker Ne…

most of our customers tell us that they're not willing to trade lower availability and higher latency per request What's the constraint that prevents you from having your dynos register with the loadbalancer cluster and then having the latter perform leastconn balancing per application? Also why would that mean "lower availability" or "higher latency"? Did you look into zookeeper?

> What's the constraint that prevents you from having your dynos register with the loadbalancer cluster and then having the latter perform leastconn balancing per application?

This is how it works. Dynos register their presence into a dyno manager which publishes the results into a feed, and then all the routing nodes subscribe to that feed.

But dyno presence is not the rapidly-changing data which is subject to CAP constraints; it's dyno activity, which changes every few milliseconds (e.g. whenever a request begins or ends). Any implementation that tracks that data will be subject to CAP, and this is where you make your choice on tradeoffs.

> why would that mean "lower availability" or "higher latency"?

I'll direct you back to the same resources we've referenced before:

http://aphyr.com/posts/278-timelike-2-everything-fails-all-t... http://ksat.me/a-plain-english-introduction-to-cap-theorem/

> Did you look into zookeeper?

This is the best question ever. Not only did we look into it, we actually invested several man-years of engineering into building our own Zookeeper-like datastore:

https://github.com/ha/doozerd

Zookeeper and Doozerd make almost the opposite trade-off as what's needed in the router: they are both really slow, in exchange for high availability and perfect consistency. Useful for many things but not tracking fast-changing data like web requests.

Re: Answering your questions about Heroku routing and web performance

#155

Post by adamwiggins » Thu, Apr 04, 2013, 6:43 PM UTC

jholman wrote: post #124

I'd like to start by acknowledging that I'm one of the "non-customers who are watching from the sidelines". I think Adam's right that this is an important distinction. Adam, there's something that confuses me about this. I'm no expert in routing theory, nor have I done the experiments, so forgive me if my reasoning misses something. I understand why RapGenius took you up on your original promises of "intelligent rout…

> It seems to me that if you split into two (or more) tiers, and random-load-balance in the front tier (hit first by the customer), and then at the second tier only send requests to unloaded clients [...]

I'm unclear how you'd think introducing a second tier changes things. That tier would need to track dyno availability and then you're right back to the same distributed state problem.

Perhaps you mean if the second tier was smaller, or even a single node? In that case, yes, we did try a variation of that. It had some benefits but also some downsides, one being that the extra network hop added latency overhead. We're continuing to explore this and variations of it, but so far we have no evidence that it would provide a major short-term benefit for RG or anyone else.

> Do you have reason to believe that this doesn't one-shot RapGenius's problem?

As a rule of thumb, I find it's best to avoid one-shots (or "specials"). It's appealing in the short term, but in the medium and long term it creates huge technical debt and almost always results in an upset customer. Products made for, and used by, many people have a level of polish and reliability that will never be matched by one-offs.

So if we're going to invest a bunch of energy into trying to solve one (or a handful) of customer's problems, a better investment is to get those customers onto the most recent product, and using all the best practices (e.g. concurrent backend, CDN, asset compilation at build time). That's a more sustainable and long-term solution.

Re: Answering your questions about Heroku routing and web performance

#156

Post by adamwiggins » Thu, Apr 04, 2013, 6:50 PM UTC

habosa wrote: post #15

adamwiggins wrote: post #13

Earlier quoted context omitted.

Yes, because that is the solution. Empirically. We've run many experiments over the past month to try other approaches to routing, including recreating the exact layout of the Bamboo routing layer (which would never scale to where we are today, but just as a point of reference). None have produced results that are anywhere near as good as using a concurrent backend. (I'd love to publish some of these results so that…

Please publish these results. I think a chart showing that Unicorn + Random routing is better than Thin + Intelligent routing would go a long way to ending this whole thing. That's assuming that you can make deploying a Unicorn app as easy as it was with Thin ('git push heroku')

We might. But what does this actually get us? It helps clear Heroku's name, but it doesn't help our customers at all. I'd prefer to spend our time and energy making customer's lives better.

Given the choice between continuing the theoretical debate over routing algorithms vs working on real customer problems (like the H12 visibility problem mentioned elsewhere in this thread), I much prefer the latter.

Re: Answering your questions about Heroku routing and web performance

#157

Post by adamwiggins » Thu, Apr 04, 2013, 6:55 PM UTC

jordanthoms wrote: post #136

My biggest issue with heroku is the general slowness with the API - maybe I'm just impatient, but most simple commands like listing releases, viewing logs etc take at least a second, sometimes five before anything happens. Pushes also take quite a while, even the Git push part is much slower than pushing to github. It's just a general sluggishness which gets annoying after a while. If they could get all the API reque…

Yeah, the developer-facing control surfaces on the platform (API calls and git push mainly) have gotten slower over the past year or two. This is on my list of of personal pet peeves, but so far has not made it onto our list of priorities.

We try to drive priorities based on what customers want, not what we want: and what we've heard in the last year or so is all about app uptime, security, and now performance and visibility.

I'm very much hoping that bringing back "fast is a feature" on the developer-facing portions of the product is something we can work on this year.

Re: Answering your questions about Heroku routing and web performance

#158

Post by adamwiggins » Thu, Apr 04, 2013, 6:57 PM UTC

randall wrote: post #140

I think the most annoying thing is they still don't answer Rap Genius's questions about being owed money for paying megabucks for newrelic. I mean If you offer a service that provides incorrect data for two years and you don't offer any sort of framework for reimbursement, that still seems, at best annoying, at worst, dishonest.

We spent quite a bit of time trying to find a one-size-fits-all framework. There just wasn't one, so we've done credits on a case-by-case basis.

Sorry you find it annoying. It's what was best for our customers.

Re: Answering your questions about Heroku routing and web performance

#159

Post by adamwiggins » Thu, Apr 04, 2013, 7:07 PM UTC

j-kidd wrote: post #137

From the article: > Q. Did the Bamboo router degrade? > A. Yes. Our older router was built and designed during the early years of Heroku to support the Aspen and later the Bamboo stack. These stacks did not support concurrent backends, and thus the router was designed with a per-app global request queue. This worked as designed originally, but then degraded slowly over the course of the next two years. From Adam's me…

There are two separate issues here, and it's easy to get them confused. One is the slow degradation on Bamboo without any change to the routing algorithm code, and the other was the explicit product choice for Cedar with a different code path in the router. Both are described fully here: https://blog.heroku.com/archives/2013/2/16/routing_performan...

The reason it's easy to confuse these two is also part of what confused us at the time. The slow degradation of the Bamboo routing behavior was causing it to gradually become more and more like the explicit choice we had made for our new product.

But of course it's up to you (and everyone else observing) to judge whether this was some kind of malicious intent to mislead, versus that we made a series of oversights that added up to some serious problems for our customers. And that we are now doing everything in our power to be fully transparent about, to rectify, and to make sure never happen again.

Re: Answering your questions about Heroku routing and web performance

#160

Post by habosa » Thu, Apr 04, 2013, 7:09 PM UTC

adamwiggins wrote: post #156

habosa wrote: post #15

Earlier quoted context omitted.

Please publish these results. I think a chart showing that Unicorn + Random routing is better than Thin + Intelligent routing would go a long way to ending this whole thing. That's assuming that you can make deploying a Unicorn app as easy as it was with Thin ('git push heroku')

We might. But what does this actually get us? It helps clear Heroku's name, but it doesn't help our customers at all. I'd prefer to spend our time and energy making customer's lives better. Given the choice between continuing the theoretical debate over routing algorithms vs working on real customer problems (like the H12 visibility problem mentioned elsewhere in this thread), I much prefer the latter.

I respect that mindset, I just don't think it would hurt. Maybe a middle ground would be a full-scale tutorial on how to switch from Thin on Bamboo/Cedar to Unicorn on Cedar for Rails users. It's a non-trivial process and I know I'd like some help with it. And in this same tutorial/article you could throw down the benchmarks you ran as motivation/justification.