Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku - Bamboo Routing Performance

blog.heroku.com

Post reply on HN
Print view View original

91–100 of 152 posts

Re: Heroku - Bamboo Routing Performance

#91

Post by philwelch »

bad_user wrote: post #68
redguava wrote: post #62

I don't understand why people think this is a great response. They know how their routing works, just say so. It can't be that hard to give a basic overview of it before they release a more comprehensive post. As for the comment "Improving our documentation and website to accurately reflect our product". That is a very round about way of saying "our website indicates our service does things that it does not" which is…

Developers are smart and capable people, but boy, what spoiled brats we are. > It can't be that hard to give a basic overview of it before they release a more comprehensive post Well, actually it is very hard to give a basic overview of anything complicated, without leaving your readership with their heads scratching. The person that wrote the blog post might not even understand the whole stack to be in the position…

> Dude, this is a service used by developers and devops.

Developers, sure. Devops? Heroku's entire purpose is so you don't have to care about ops.

> You aren't meant to trust what they say.

Evidently not! I'd rather not do business with liars.

> They acknowledge sincerely that they have a problem, with a promise that they are going to fix it.

That's an optimistic interpretation of their blog post.

Re: Heroku - Bamboo Routing Performance

#92

Post by grey-area »

brown9-2 wrote: post #46
ibdknox wrote: post #18

It's a good response in that they are taking responsibility, but it is pretty obvious that they are reluctant to say anything about a fix. In my mind, "it's hard" isn't a valid excuse in this case, especially when there are relatively straightforward solutions that will solve this at a practical level. For example, you could imagine a naive form of intelligent routing that would work simply by keeping a counter per d…

Seems obvious that a naive solution wouldn't be as easy at large scale. You have to imagine that Heroku would have considered a whole lot of options before deciding on random distribution. Give them some credit at least.

What's worse is that they are not dealing with the behaviour of just one app. Random routing probably works best for a certain subset of their customers, maybe even the majority. If they switched to it it's unlikely to have happened without measuring a few different approaches and deciding this gave the best results overall. Unfortunately they have to deal with all of their customers apps, some of whom might have a huge variation in response time, which seems to trigger these issues - it's a complicated topic and one solution might not work for everyone.

Re: Heroku - Bamboo Routing Performance

#93

Post by nikcub »

kawsper wrote: post #87
nikcub wrote: post #72

Earlier quoted context omitted.

This has all been solved previously. In Google Appengine the scheduler is aware of, for each instance: * the type of instance it is * the amount of memory currently being used * the amount of CPU currently being used * the last request time handled by that instance It also tracks the profile of your application, and applies a scheduling algorithm based on what it has learned. For eg. the url /import may take 170MB an…

> It also tracks the profile of your application, and applies a scheduling algorithm based on what it has learned. For eg. the url /import may take 170MB and 800ms to run, on average, so it would schedule it with an instance that has more resources available. That is very awesome technology, but it something like that available for non-google people?

Expensive commercial appliances like the popular f5 big ip's can, and that is what a lot of large-scale websites use:

http://www.f5.com/glossary/load-balancer/

In terms of open source, HAProxy has layer 7 algorithms but they are much simpler:

http://cbonte.github.com/haproxy-dconv/configuration-1.5.htm...

If you were inclined, you could write an algorithm to implement something similar in one of the open source routers.

Re: Heroku - Bamboo Routing Performance

#94

Post by nikcub »

gojomo wrote: post #82
nikcub wrote: post #72

Earlier quoted context omitted.

This has all been solved previously. In Google Appengine the scheduler is aware of, for each instance: * the type of instance it is * the amount of memory currently being used * the amount of CPU currently being used * the last request time handled by that instance It also tracks the profile of your application, and applies a scheduling algorithm based on what it has learned. For eg. the url /import may take 170MB an…

Sounds nice, but I'm not sure it's the only way -- that Heroku 'essentially needs to build all of that'. It'd be interesting to see whose routing-to-instance is faster in the non-contended case, between Heroku and GAE. Do you know of any benchmarks?

Don't know of any benchmarks, but I have/had a number of projects on AppEngine and it is very good (but expensive). I would be looking to include Elastic Beanstalk in a comparison as well, as it is gaining popularity since it launched (it doesn't have the lockin and supports any environment).

Re: Heroku - Bamboo Routing Performance

#95

Post by xwowsersx »

What the hell? It's good he owned up...I guess. But the response basically sounds like "yeah, we've been charging the same prices over the last few years for increasingly degraded performance and we would have continued to do so, but someone finally caught on so I guess we have to now do something about this, right?"

Re: Heroku - Bamboo Routing Performance

#96

Post by DeepDuh »

gojomo wrote: post #79
DeepDuh wrote: post #57

Earlier quoted context omitted.

Really? Is that so hard? All you need is a table on the router that tells it the best information it can currently have about the number of requests processed on each dyno - without doing a roundtrip. This requires exactly one additional (one-way) package: A message from the dyno to the router, telling it that it has finished the current request. Now, to avoid dead dynos (because the finished message might have been…

A problem with this proposal is the assumption that there is one 'the router' with this info, updated for tens of thousands of dynos and millions of requests per second.

Well, if this is the case that would be a pretty deep architectural problem, I'd say, for a PaaS like Heroku.

I think it's pretty obvious that you need at least two layers of hierarchy for the routing here: One (or more) router forwarding requests to virtualized routers (per Heroku customer 'instance' or whatever that's called), which in turn provide the functionality I described in software. I'd probably use VMs running a specialized minimal linux distro for the per-instance-routers.

Re: Heroku - Bamboo Routing Performance

#97

Post by larsmak »

bad_user wrote: post #68
redguava wrote: post #62

I don't understand why people think this is a great response. They know how their routing works, just say so. It can't be that hard to give a basic overview of it before they release a more comprehensive post. As for the comment "Improving our documentation and website to accurately reflect our product". That is a very round about way of saying "our website indicates our service does things that it does not" which is…

Developers are smart and capable people, but boy, what spoiled brats we are. > It can't be that hard to give a basic overview of it before they release a more comprehensive post Well, actually it is very hard to give a basic overview of anything complicated, without leaving your readership with their heads scratching. The person that wrote the blog post might not even understand the whole stack to be in the position…

I agree, some people tend to think that Heroku is some kind of silver bullet that magically scales anything into the sky. But the underlying platform is nothing we haven not used before, and scaling linearly as the userbase grows is no easy task indeed. Give them some slack..

Re: Heroku - Bamboo Routing Performance

#98

Post by vidarh »

gojomo wrote: post #36
ibdknox wrote: post #18

It's a good response in that they are taking responsibility, but it is pretty obvious that they are reluctant to say anything about a fix. In my mind, "it's hard" isn't a valid excuse in this case, especially when there are relatively straightforward solutions that will solve this at a practical level. For example, you could imagine a naive form of intelligent routing that would work simply by keeping a counter per d…

What if their inbound routing is hundreds of machines, each of which may get a request for any of their thousands of apps, spread across tens of thousands of web dynos? Do you have a distributed sufficiently-consistent counter strategy that won't itself become a source of latency or bottlenecks or miscounts under traffic surges?

You don't need a distributed sufficiently-consistent counter strategy. You can do just fine with a two layer routing mechanism where the first layer "sort" the incoming requests (e.g. by host header) so that the second layer is grouped by customer and can apply any number of simple alternatives like least-connections.

Re: Heroku - Bamboo Routing Performance

#99

Post by gojomo »

jacques_chester wrote: post #90
gojomo wrote: post #85

Earlier quoted context omitted.

They could partition the routing, and maybe they do. But then (a) there's one extra hop mapping to the specialist routing group; and (b) it's still nice to have super-thin minimal-state routers, for example with just a list of up dynos updated once every few seconds, as opposed to live dyno load state updated thousands of times per second. I too hope their full response givss more insight into their architecture... I…

> there's one extra hop mapping I thought about mentioning this. Because 1 small hop is still less than random blowouts in response time. You can even cheat by pushing the router IP into DNS. Hop eliminated. > it's still nice to have super-thin minimal-state routers I imagine Heroku's customers are not interested in what is nice for Heroku, they want Heroku to do the icky difficult stuff for them . That was the whole…

...pushing the router IP into DNS...

Maybe, but they don't currently give each app its own IP, and might not want the complications of volatile IP reassignments, DNS TTLs, and so on. (Though, their current "CNAME-to-yourapp.herokuapp.com" recommendation would allow for this.)

...want Heroku to do the icky difficult stuff...

Yes, but to a point. Customers also want Heroku to provide a simple model that allows scaling as easy as twisting a knob to deploy more dynos, or move to a higher-resourced plan. Customers accept some limitations to fit that model.

Maybe Heroku has a good reason for thin, fast, stateless routing -- and that works well for most customers, perhaps with some app adjustments. Then, coaxing customers to fit that model, rather than rely on any sort of 'smart' routing that would be overkill for most, is the right path.

We'll know a lot more when they post their "in-depth technical review" Friday.

Re: Heroku - Bamboo Routing Performance

#100

Post by vidarh »

jacques_chester wrote: post #56
gojomo wrote: post #33

Earlier quoted context omitted.

It requires statefulness and decisionmaking at the routing layer, and that's another thing that adds overhead and can go wrong at scale. (For example, there may be no one place with knowledge of all in-process requests. Traffic surges may lead to an arbitrary growth of state in the routing layer, rather than at the dynos.) There are probably some simple techniques whereby dynos can themselves approximate the throughp…

I must be stupid, because surely it can't be that hard to partition the routing groups? For example, use a hashing algorithm that switches to 1 of N intelligent routers based on domain name. If you pick the right algo you can pretty much add routers whenever you like. (It would be nice to know what Heroku have tried so far, at the very least to drive off know-it-all blowhards like me.)

You are right. In the previous thread I pointed out one way of doing this with haproxy (there are many other tools that could do the job): "balance hdr(host)" on the first layer, and least connections balancing on the second layer. Or to know exactly which hosts are handled at the second layer (to minimize configuration size for each second layer server), create acl's for either specific hosts, or suitably short substrings, and your second layer servers can keep only the configuration for that subset.

You can also easily enough use "just" iptables at the second layer (supports weighted least connections and many others and you can plug in your own modules), which makes it very easy to do dynamic reconfiguration (to e.g. add/remove dynos), as well as traffic accounting (setting iptables to count bytes routed per rule is easy) and other fun stuff.

Post reply on HN