Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku - Bamboo Routing Performance

blog.heroku.com

Post reply on HN
Print view View original

81–90 of 152 posts

Re: Heroku - Bamboo Routing Performance

#81

Post by nostrademons »

benatkin wrote: post #58
btilly wrote: post #41

Earlier quoted context omitted.

The problem with that is when your code is failing to run properly due to their hardware problem - now whose fault is it? I don't know how big they are. 50k machines? Could be off by an order of magnitude either way but I'll go with that. Suppose that your servers have, let's be generous, a 5 year mean time between failure. That's 10k machines dying every year. About 27 per day. A bit over 1 per hour. Machines don't…

I'm having trouble finding information about this. It's disturbing how little attention seems to be given to load balancing relative to how important it is.

Yeah, it's interesting what sort of emergent properties come out of massively-scalable, massively-distributed systems. For example, when you write software in school or for single-machine deployment, you're taught to assume that when there's a bug it's your fault, a defect in your software. That's no longer the case when you get into massive (10K+ machine) clusters, where when your program fails, it might be your software, or it might be the hardware, or it might be a random event like a cosmic ray (seriously...in early Google history, there were several failed crawls that happened because cosmic rays caused random single-bit errors in the software).

And so all the defect-prevention approaches you learn for writing single-machine software - testing, assertions, invariants, static-typing, code reviews, linters, coding standards - need to be supplemented with architectural approaches. Retries, canaries, phased rollouts, supervisors, restarts, checksums, timeouts, distributed transactions, replicas, Paxos, quorums, recovery logic, etc. A typical C++ or Java programmer thinks of reliability in terms of "How many bugs are in my program?" The Erlang guys figured out a while ago that this is insufficient for reliability, because the hardware might (and will, in a sufficiently large system) fail, and so to build reliable systems you need at least two computers, and it's better to let errors kill the process and trigger a fallback than to try to never have errors.

Re: Heroku - Bamboo Routing Performance

#82

Post by gojomo »

nikcub wrote: post #72
gojomo wrote: post #47

Earlier quoted context omitted.

I doubt they want every inbound request to require: • query remote redis for lowest-connection-count dyno(s) (from among potentially hundreds): 1 network roundtrip • increment count at remote redis for chosen dyno: 1 network roundtrip (maybe can be coalesced with above?) • when connection ends, decrement count at remote redis for chosen dyno: 1 network roundtrip That's 2-3 extra roundtrips each inbound request, and n…

This has all been solved previously. In Google Appengine the scheduler is aware of, for each instance: * the type of instance it is * the amount of memory currently being used * the amount of CPU currently being used * the last request time handled by that instance It also tracks the profile of your application, and applies a scheduling algorithm based on what it has learned. For eg. the url /import may take 170MB an…

Sounds nice, but I'm not sure it's the only way -- that Heroku 'essentially needs to build all of that'. It'd be interesting to see whose routing-to-instance is faster in the non-contended case, between Heroku and GAE. Do you know of any benchmarks?

Re: Heroku - Bamboo Routing Performance

#83

Post by aneth4 »

This is a horribly inadequate response. Prices for hardware have dropped 30% over the last 3 years and heroku is admitting their performance has degraded by many orders of magnitude. It's completely unacceptable to simply say, "yeah there's a problem, we'll give you some metrics to understand it better."

Sure, it's great they responded. The response should be "you're right, we are fixing it and issue credits" for revenue gained from fraudulent claims about the performance of their product and a credibility straining bait-and-switch.

Re: Heroku - Bamboo Routing Performance

#84

Post by cbs »

twog wrote: post #24

Honest question, why would Rapgenuis still be on Heroku if the y needed 100 dynos? Why not go directly to AWS at that scale? The cost savings would be pretty significant. Am I missing something?

Before you undertake the more significant time and cost of migrating platforms, you see what you can wring out of what you've got. It is possible that after improving their performance enough on Heroku (or hell, even without improving it), they can not justify the up-front money and resources to migrate platforms.

Or, for all we know, they could already be in the process of migrating away from Heroku, but that doesn't happen overnight and doesn't help their performance in the meantime.

Re: Heroku - Bamboo Routing Performance

#85

Post by gojomo »

jacques_chester wrote: post #56
gojomo wrote: post #33

Earlier quoted context omitted.

It requires statefulness and decisionmaking at the routing layer, and that's another thing that adds overhead and can go wrong at scale. (For example, there may be no one place with knowledge of all in-process requests. Traffic surges may lead to an arbitrary growth of state in the routing layer, rather than at the dynos.) There are probably some simple techniques whereby dynos can themselves approximate the throughp…

I must be stupid, because surely it can't be that hard to partition the routing groups? For example, use a hashing algorithm that switches to 1 of N intelligent routers based on domain name. If you pick the right algo you can pretty much add routers whenever you like. (It would be nice to know what Heroku have tried so far, at the very least to drive off know-it-all blowhards like me.)

They could partition the routing, and maybe they do. But then (a) there's one extra hop mapping to the specialist routing group; and (b) it's still nice to have super-thin minimal-state routers, for example with just a list of up dynos updated once every few seconds, as opposed to live dyno load state updated thousands of times per second.

I too hope their full response givss more insight into their architecture... I have a couople of small projects at Herooku already and may use them. for several larger ones in the future.

Re: Heroku - Bamboo Routing Performance

#86

Post by rmason »

spiralganglion wrote: post #66
redguava wrote: post #62

I don't understand why people think this is a great response. They know how their routing works, just say so. It can't be that hard to give a basic overview of it before they release a more comprehensive post. As for the comment "Improving our documentation and website to accurately reflect our product". That is a very round about way of saying "our website indicates our service does things that it does not" which is…

Documentation discrepancies happen. I've seen them with pretty much every platform I've worked on. Just yesterday, I found a critical discrepancy between the ActionScript documentation and the actual behaviour of the ActionScript compiler, costing my team a day of work. (I tried to report the issue to Adobe, but the Adobe Bug Reporting System was down. Perhaps they need a Bug Reporting System for the Bug Reporting Sy…

On ActionScript your first source of information should be on Flex's new Apache site http://flex.apache.org/

I have friends in that community and while they're all volunteers I've seen first hand how hard they try and help people.

Re: Heroku - Bamboo Routing Performance

#87

Post by kawsper »

nikcub wrote: post #72
gojomo wrote: post #47

Earlier quoted context omitted.

I doubt they want every inbound request to require: • query remote redis for lowest-connection-count dyno(s) (from among potentially hundreds): 1 network roundtrip • increment count at remote redis for chosen dyno: 1 network roundtrip (maybe can be coalesced with above?) • when connection ends, decrement count at remote redis for chosen dyno: 1 network roundtrip That's 2-3 extra roundtrips each inbound request, and n…

This has all been solved previously. In Google Appengine the scheduler is aware of, for each instance: * the type of instance it is * the amount of memory currently being used * the amount of CPU currently being used * the last request time handled by that instance It also tracks the profile of your application, and applies a scheduling algorithm based on what it has learned. For eg. the url /import may take 170MB an…

> It also tracks the profile of your application, and applies a scheduling algorithm based on what it has learned. For eg. the url /import may take 170MB and 800ms to run, on average, so it would schedule it with an instance that has more resources available.

That is very awesome technology, but it something like that available for non-google people?

Re: Heroku - Bamboo Routing Performance

#88

Post by philwelch »

ibdknox wrote: post #29
twog wrote: post #24

Honest question, why would Rapgenuis still be on Heroku if the y needed 100 dynos? Why not go directly to AWS at that scale? The cost savings would be pretty significant. Am I missing something?

Ops guys cost a lot more than just using Heroku, not to mention the cost of simply having the responsibility of servers (even if they are virtual). Never underestimate the value of just not having to think about something, especially when you're small group of people.

I think the amount of time and energy they've invested in studying Heroku's routing and queueing strategy counts as having to think about something.

Re: Heroku - Bamboo Routing Performance

#89

Post by fauigerzigerk »

antoko wrote: post #9

That's actually a pretty impressive response as far as it goes. Obviously there's no details at this point, but he absolutely takes responsibility, doesn't try to deflect or sugar coat it, and manages to find a tone that is both professional/serious, yet also down-to-earth and earnest. I guess the real impact will be how they go about "making it right" but in terms of a first response to the situation the tone is nea…

I would reserve judgement on this response until we learn the truth. Whether or not a response is adequate depends on what they could have said and should have said, which obviously depends on the facts. So let's just wait for more information until we assign praise or blame.

Re: Heroku - Bamboo Routing Performance

#90

Post by jacques_chester »

gojomo wrote: post #85
jacques_chester wrote: post #56

Earlier quoted context omitted.

I must be stupid, because surely it can't be that hard to partition the routing groups? For example, use a hashing algorithm that switches to 1 of N intelligent routers based on domain name. If you pick the right algo you can pretty much add routers whenever you like. (It would be nice to know what Heroku have tried so far, at the very least to drive off know-it-all blowhards like me.)

They could partition the routing, and maybe they do. But then (a) there's one extra hop mapping to the specialist routing group; and (b) it's still nice to have super-thin minimal-state routers, for example with just a list of up dynos updated once every few seconds, as opposed to live dyno load state updated thousands of times per second. I too hope their full response givss more insight into their architecture... I…

> there's one extra hop mapping

I thought about mentioning this. Because 1 small hop is still less than random blowouts in response time.

You can even cheat by pushing the router IP into DNS. Hop eliminated.

> it's still nice to have super-thin minimal-state routers

I imagine Heroku's customers are not interested in what is nice for Heroku, they want Heroku to do the icky difficult stuff for them. That was the whole pitch.

Anyway, we're arguing about Star Wars vs Star Trek here because we have no earthly idea what they've tried.

Post reply on HN