Answering your questions about Heroku routing and web performance • Hacker News Board

Re: Answering your questions about Heroku routing and web performance

#131

Post by moe » Thu, Apr 04, 2013, 2:22 AM UTC

adamwiggins wrote: post #86

josh2600 wrote: post #81

Earlier quoted context omitted.

I don't understand how the system behaves more intelligently when the edge routers increase. The core routers random behavior gets worse with the larger load, and increasing the number of intelligent routers doesn't help to solve that problem in any meaningful way. Sorry, correct me if I missed something, but I believe that as the overall volume of system transactions increases (thus necessitating more "intelligent"…

> "Intelligent routing is hard, so fuck it" Ok, let's really dig in on this. Is this truly a case of us being lazy? We just can't be bothered to implement something that would make our customers' lives better? The answer to these questions is no. Single global request queues have trade-offs. One of those tradeoffs is more latency per request. Another is availability on the app. Despite the sentiment here on Hacker Ne…

most of our customers tell us that they're not willing to trade lower availability and higher latency per request

What's the constraint that prevents you from having your dynos register with the loadbalancer cluster and then having the latter perform leastconn balancing per application?

Also why would that mean "lower availability" or "higher latency"? Did you look into zookeeper?

Re: Answering your questions about Heroku routing and web performance

#132

Post by marshray » Thu, Apr 04, 2013, 2:52 AM UTC

moe wrote: post #131

adamwiggins wrote: post #86

Earlier quoted context omitted.

> "Intelligent routing is hard, so fuck it" Ok, let's really dig in on this. Is this truly a case of us being lazy? We just can't be bothered to implement something that would make our customers' lives better? The answer to these questions is no. Single global request queues have trade-offs. One of those tradeoffs is more latency per request. Another is availability on the app. Despite the sentiment here on Hacker Ne…

most of our customers tell us that they're not willing to trade lower availability and higher latency per request What's the constraint that prevents you from having your dynos register with the loadbalancer cluster and then having the latter perform leastconn balancing per application? Also why would that mean "lower availability" or "higher latency"? Did you look into zookeeper?

Note: Just a bystander here

> What's the constraint that prevents you from having your dynos register with the loadbalancer cluster and then having the latter perform leastconn balancing per application

I suspect this is a consequence of the CAP theorem. You'll end up with every loadbalancer needing a near-instantaneous perception of every server's queue state and then updating that state atomically when routing a request. Now consider the failure modes that such a system can enter and how they affect latency. Best not to go there.

My understanding is that Apache Zookeeper is designed for slowly-changing data.

Re: Answering your questions about Heroku routing and web performance

#133

Post by moe » Thu, Apr 04, 2013, 3:26 AM UTC

marshray wrote: post #132

moe wrote: post #131

Earlier quoted context omitted.

most of our customers tell us that they're not willing to trade lower availability and higher latency per request What's the constraint that prevents you from having your dynos register with the loadbalancer cluster and then having the latter perform leastconn balancing per application? Also why would that mean "lower availability" or "higher latency"? Did you look into zookeeper?

Note: Just a bystander here > What's the constraint that prevents you from having your dynos register with the loadbalancer cluster and then having the latter perform leastconn balancing per application I suspect this is a consequence of the CAP theorem. You'll end up with every loadbalancer needing a near-instantaneous perception of every server's queue state and then updating that state atomically when routing a re…

You'll end up with every loadbalancer needing a near-instantaneous perception of every server's queue

But that's not true. Only the loadbalancers concerned with a given application need to share that state amongst one another. And the number of loadbalancers per application is usually very small. I.e. the number is 99% of sites and you need quite a popular site to push it into the double digits (a single haproxy instance can sustain >5k connect/sec).

Assigning pooled loadbalancers to apps while ensuring HA is not trivial, but it's also not rocket science. I'm a little surprised by the heroku response here, hence my question which constraint I might have missed.

My understanding is that Apache Zookeeper is designed for slowly-changing data.

Dyno-presence per application is very slowly-changing data by zookeeper standards.

Re: Answering your questions about Heroku routing and web performance

#134

Post by stickfigure » Thu, Apr 04, 2013, 3:49 AM UTC

deedubaya wrote: post #76

Haters gonna hate. I wonder how many people bitching here are actual customers who are having problems that haven't been address with a solution. I'm guessing that number is low. Oh, you're a potential customer? That's why you're bitching? About a problem you may or may not have if you actually choose the product? Think about that argument for a second. I've never seen such a transparent response and follow up as I h…

I'm a real customer with real problems. Grep this page for latchkey's description of it.

The funny thing is, I don't have much sympathy for Rails users. Scaling problems with a single-threaded, serial request-processing architecture? No surprise there. But we have inexplicable H12 problems with Node.js. There's something broken in the system and it isn't random routing.

Re: Answering your questions about Heroku routing and web performance

#135

Post by dandelany » Thu, Apr 04, 2013, 4:39 AM UTC

adamwiggins wrote: post #99

dandelany wrote: post #68

Earlier quoted context omitted.

I think you're right, and defraud is the wrong word, because fraud implies malicious intent. However, that's almost beside the point - many customers paid for the product that was advertised/documented and received something completely different. Whether or not it was intentional, it should be remediated rather than painted over. Let's completely ignore the random vs. intelligent routing question for the moment and j…

Fair enough, and good analogy. As you said it's hard to make comparisons between physical goods and a metered software service. The New Relic question is tricky. The free version of NR includes queue time, so that implies that the incremental value you're getting from the paid service does not include this. I'm also not sure how "this product you've gotten for free has a bug" fits into this equation. But overall, yes…

Thanks for the thoughtful reply, Adam. FWIW, I think you guys provide a very valuable service & I wish you the best as you work through these issues.

Re: Answering your questions about Heroku routing and web performance

#136

Post by jordanthoms » Thu, Apr 04, 2013, 4:40 AM UTC

My biggest issue with heroku is the general slowness with the API - maybe I'm just impatient, but most simple commands like listing releases, viewing logs etc take at least a second, sometimes five before anything happens. Pushes also take quite a while, even the Git push part is much slower than pushing to github. It's just a general sluggishness which gets annoying after a while.

If they could get all the API requests down under 500ms I'd be much happier.

Re: Answering your questions about Heroku routing and web performance

#137

Post by j-kidd » Thu, Apr 04, 2013, 4:42 AM UTC

From the article:

> Q. Did the Bamboo router degrade?

> A. Yes. Our older router was built and designed during the early years of Heroku to support the Aspen and later the Bamboo stack. These stacks did not support concurrent backends, and thus the router was designed with a per-app global request queue. This worked as designed originally, but then degraded slowly over the course of the next two years.

From Adam's message on Feb 17th, 2011 (https://groups.google.com/forum/?fromgroups=#!topic/heroku/8...):

> You're correct, the routing mesh does not behave in quite the way described by the docs. We're working on evolving away from the global backlog concept in order to provide better support for different concurrency models, and the docs are no longer accurate. The current behavior is not ideal, but we're on our way to a new model which we'll document fully once it's done.

It looks like random load balancing was already the expected behavior 2 years ago? The "slow degradation" part seems a bit dishonest to me.

Re: Answering your questions about Heroku routing and web performance

#138

Post by marshray » Thu, Apr 04, 2013, 4:49 AM UTC

moe wrote: post #133

marshray wrote: post #132

Earlier quoted context omitted.

Note: Just a bystander here > What's the constraint that prevents you from having your dynos register with the loadbalancer cluster and then having the latter perform leastconn balancing per application I suspect this is a consequence of the CAP theorem. You'll end up with every loadbalancer needing a near-instantaneous perception of every server's queue state and then updating that state atomically when routing a re…

You'll end up with every loadbalancer needing a near-instantaneous perception of every server's queue But that's not true. Only the loadbalancers concerned with a given application need to share that state amongst one another. And the number of loadbalancers per application is usually very small. I.e. the number is 99% of sites and you need quite a popular site to push it into the double digits (a single haproxy inst…

Again, I'm no expert on Heroku's architectre. Just thinking out loud here, and feel free to tell me to RTFA. :-)

> the number of loadbalancers per application is usually very small. I.e. the number is 99% of sites and you need quite a popular site to push it into the double digits (a single haproxy instance can sustain >5k connect/sec).

So most Heroku sites have only a single frontend loadbalancer doing their routing, and even these cases are getting random routed with suboptimal results?

Or is the latency issue mainly with respect to exactly those popular sites that end up using a distributed array of loadbalancers?

> Assigning pooled loadbalancers to apps while ensuring HA is not trivial, but it's also not rocket science.

To me the short history of "cloud-scale" (sorry) app proxy load balancing shows that very well-resourced and well-engineered systems often work great and scale great, that is until some weird failure mode unbalances the whole system and response time goes all hockey stick.

> Dyno-presence per application is very slowly-changing data by zookeeper standards.

OK, but instantaneous queue depth for each and every server? (within a given app)

Re: Answering your questions about Heroku routing and web performance

#139

Post by marshray » Thu, Apr 04, 2013, 4:52 AM UTC

adamwiggins wrote: post #88

obilgic wrote: post #82

Earlier quoted context omitted.

> true that makes sense, i was wondering how intelligent routing was implemented in the first place.

Oh, got it. How's this: In the early days, Heroku only had a single routing node that sat out front. So it wasn't a distributed systems problem at that point. You could argue that Heroku circa 2009 was more of a prototype or a toy than a scalable piece of infrastructure. You couldn't run background workers, or large databases. We weren't even charging money yet. Implementing a single global queue in a single node is…

> So to answer your question, there was never some crazy-awesome implementation of a distributed global queue that we got rid of.

So it was an oversimplified system that worked great but wasn't scalable and was at some point going to completely fall over under increasing load.

IMExp, this is not a wrong thing to build initially and it's not wrong to replace it either. But the replacement is going to have a hard time being as simple or predictable. :-)

Re: Answering your questions about Heroku routing and web performance

#140

Post by randall » Thu, Apr 04, 2013, 5:01 AM UTC

I think the most annoying thing is they still don't answer Rap Genius's questions about being owed money for paying megabucks for newrelic. I mean If you offer a service that provides incorrect data for two years and you don't offer any sort of framework for reimbursement, that still seems, at best annoying, at worst, dishonest.