Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku - Bamboo Routing Performance

blog.heroku.com

Post reply on HN
Print view View original

71–80 of 152 posts

Re: Heroku - Bamboo Routing Performance

#71

Post by redguava »

bad_user wrote: post #68
redguava wrote: post #62

I don't understand why people think this is a great response. They know how their routing works, just say so. It can't be that hard to give a basic overview of it before they release a more comprehensive post. As for the comment "Improving our documentation and website to accurately reflect our product". That is a very round about way of saying "our website indicates our service does things that it does not" which is…

Developers are smart and capable people, but boy, what spoiled brats we are. > It can't be that hard to give a basic overview of it before they release a more comprehensive post Well, actually it is very hard to give a basic overview of anything complicated, without leaving your readership with their heads scratching. The person that wrote the blog post might not even understand the whole stack to be in the position…

"Dude, this is a service used by developers and devops. You aren't meant to trust what they say." that is ridiculous. I use their service and I have trusted what they have said. They are the experts on their own service, why shouldn't I?

Also, the reason I chose Heroku is because DevOps is not my area of expertise, if I wanted to measure and analysis the inner workings I would have hosted elsewhere. Heroku is meant to alleviate that pain for me, I pay them excess over what I would pay elsewhere for that luxury.

As for the website not lying, yes it does. Go to "how it works" and click on routing. It's plain not true. Is it too much to ask that they update this in the last 3 years since the change?

I think the response is purely PR. Actually give something that will help their existing customers with this problem.

Re: Heroku - Bamboo Routing Performance

#72

Post by nikcub »

gojomo wrote: post #47
ibdknox wrote: post #40

Earlier quoted context omitted.

Atomic counters are pretty fast. Redis, for example, should be able to handle it without breaking a sweat: http://redis.io/topics/benchmarks

I doubt they want every inbound request to require: • query remote redis for lowest-connection-count dyno(s) (from among potentially hundreds): 1 network roundtrip • increment count at remote redis for chosen dyno: 1 network roundtrip (maybe can be coalesced with above?) • when connection ends, decrement count at remote redis for chosen dyno: 1 network roundtrip That's 2-3 extra roundtrips each inbound request, and n…

This has all been solved previously. In Google Appengine the scheduler is aware of, for each instance:

* the type of instance it is

* the amount of memory currently being used

* the amount of CPU currently being used

* the last request time handled by that instance

It also tracks the profile of your application, and applies a scheduling algorithm based on what it has learned. For eg. the url /import may take 170MB and 800ms to run, on average, so it would schedule it with an instance that has more resources available.

It does all this prior to the requests running.

You can find more docs on it here:

https://developers.google.com/appengine/docs/adminconsole/in...

For eg.

> Each instance has its own queue for incoming requests. App Engine monitors the number of requests waiting in each instance's queue. If App Engine detects that queues for an application are getting too long due to increased load, it automatically creates a new instance of the application to handle that load

This is what it looks like from a user point of view:

http://i.imgur.com/QFMXeT1.png

Heroku essentially need to build all of that. The way it is solved is that the network roundtrips to poll the instances run in parallel to the scheduler. You don't do:

* accept request

* poll scheduler

* poll instance/dyno

* serve request

* update scheduler

* update instance/dyno

This all happens asynchronously. At most your data is 10ms out of date. It would also use a very lightweight UDP based protocol and would broadcast (and not round-trip, since you send the data frequently enough with a checksum that a single failure doesn't really matter, at worst it delays a request or two).

Re: Heroku - Bamboo Routing Performance

#73

Post by bitcartel »

redguava wrote: post #70
spiralganglion wrote: post #66

Earlier quoted context omitted.

Documentation discrepancies happen. I've seen them with pretty much every platform I've worked on. Just yesterday, I found a critical discrepancy between the ActionScript documentation and the actual behaviour of the ActionScript compiler, costing my team a day of work. (I tried to report the issue to Adobe, but the Adobe Bug Reporting System was down. Perhaps they need a Bug Reporting System for the Bug Reporting Sy…

In answer to "who cares if the explanation comes today or tomorrow", I care if the explanation comes today or tomorrow. I use Heroku and have hit scaling issues in the last few weeks very similar to this. More information on what is going on behind the scenes will help me immediately. As for discrepancy in documentation, this is one of the most major parts of their infrastructure and directly relates to how well appl…

Well, for marketing purposes, random doesn't sound as impressive as intelligent. It's a discrepancy but it does appear to be disclosed.

On the page you link to, it says: "Incoming web traffic is automatically routed to web dynos, with intelligent distribution of load instantly as you scale."

When you click on "Read more about routing..." it says: "Request distribution - The routing mesh uses a random selection algorithm for HTTP request load balancing across web processes." https://devcenter.heroku.com/articles/http-routing

Re: Heroku - Bamboo Routing Performance

#74

Post by redguava »

redguava wrote: post #70
spiralganglion wrote: post #66

Earlier quoted context omitted.

Documentation discrepancies happen. I've seen them with pretty much every platform I've worked on. Just yesterday, I found a critical discrepancy between the ActionScript documentation and the actual behaviour of the ActionScript compiler, costing my team a day of work. (I tried to report the issue to Adobe, but the Adobe Bug Reporting System was down. Perhaps they need a Bug Reporting System for the Bug Reporting Sy…

In answer to "who cares if the explanation comes today or tomorrow", I care if the explanation comes today or tomorrow. I use Heroku and have hit scaling issues in the last few weeks very similar to this. More information on what is going on behind the scenes will help me immediately. As for discrepancy in documentation, this is one of the most major parts of their infrastructure and directly relates to how well appl…

@bitcartel (not sure why I couldn't reply to your comment)... I believe from my discussions with Heroku that the "random selection algorithm" is also not true.

Re: Heroku - Bamboo Routing Performance

#75

Post by nikcub »

There is a perverse conflict with platform service providers - the worse your scheduler performs the more profitable your service will be.

You replace intelligent request scheduling with more hardware and instances, which you charge the user for.

How much investment is there in platform service providers towards developing better schedulers that would reduce the number of instances required to serve an application? That answer, in this case, is "not a lot"

The incentives between provider and user are not aligned, which is why I am more inclined to buy and manage at a layer lower with virtual machines.

Edit: AppEngine went through a similar issue. Here is an interesting response from an engineer on their team:

https://groups.google.com/forum/#!msg/google-appengine/y-LnZ...

Re: Heroku - Bamboo Routing Performance

#76

Post by oillio »

stonewhite wrote: post #39
kevinfat wrote: post #19

Can someone explain, to people who know nothing about scaling infrastructure, why routing to idle dynos is a hard problem?

Assuming you keep your prices roughly the same, the computation needed for assigning an available dyno becomes exponentially harder(costlier) as more dynos are added to the ecosystem. Thats why they changed the intelligent routing to random routing; to save cpu cycles. The decision made by heroku was not an engineering decision, it was a business decision. While it is quite a bit frustrating, it is understandable and…

A simple intelligent router may have 2^n performance, but I cannot imagine there are not other solutions that would bring performance closer to linear. I am not a queue theory expert, but I know they are out there. This is a well studied area of CS.

This is not only a problem for non-concurrent applications. It will become a problem as dyno usage increases for any application. The major factors will be the standard deviation of your response times and the number of responses you can handle per dyno. The issue just manifests first in applications that can only handle one request per dyno.

Check out this graph released by rapgenius: http://s3.amazonaws.com/rapgenius/1360871196_routerstyles_fa...

The purple line is a multi-threaded application. No matter how concurrent your framework, it is still possible to overload the resources of a single dyno. When that happens, the router will continue to stack requests up in the queue.

Re: Heroku - Bamboo Routing Performance

#77

Post by nostrademons »

nikcub wrote: post #75

There is a perverse conflict with platform service providers - the worse your scheduler performs the more profitable your service will be. You replace intelligent request scheduling with more hardware and instances, which you charge the user for. How much investment is there in platform service providers towards developing better schedulers that would reduce the number of instances required to serve an application? T…

In theory, the service provider could just switch to a better scheduler and raise their rates to compensate, since the TCO for a customer would go down correspondingly. Their costs go down, their profits go up, and customers still benefit.

Prices are usually determined more by competition than by technical factors. Unfortunately, there's a fair bit of lock-in to PaaS vendors, which is the real reason I'd be skeptical about building a business on them. The nice thing about the VM layer is there's a well-documented, public API between your code and the platform, so when your service provider raises their rates, you can switch to another one or operate your own hardware.

Re: Heroku - Bamboo Routing Performance

#78

Post by gojomo »

ibdknox wrote: post #50
gojomo wrote: post #47

Earlier quoted context omitted.

I doubt they want every inbound request to require: • query remote redis for lowest-connection-count dyno(s) (from among potentially hundreds): 1 network roundtrip • increment count at remote redis for chosen dyno: 1 network roundtrip (maybe can be coalesced with above?) • when connection ends, decrement count at remote redis for chosen dyno: 1 network roundtrip That's 2-3 extra roundtrips each inbound request, and n…

My argument was built on the premise that random routing isn't acceptable given the potential slow downs it can cause (as pointed out in the Rap Genius post). If you believe otherwise, then there's no real argument for me to make :) With that said, in your example, you could do one and two together and the response doesn't need to wait on the completion of #3. So it's one network roundtrip, which I would imagine is a…

As I've written elsewhere, I think just having a way for dynos to refuse load (by not accepting a connection, or returning an error or redirect), such that the load tries another dyno, will probably achieve most of the benefits of 'intelligent' routing. And, preserve the stateless scalability of the 'routing mesh'.

Re: Heroku - Bamboo Routing Performance

#79

Post by gojomo »

DeepDuh wrote: post #57
gojomo wrote: post #47

Earlier quoted context omitted.

I doubt they want every inbound request to require: • query remote redis for lowest-connection-count dyno(s) (from among potentially hundreds): 1 network roundtrip • increment count at remote redis for chosen dyno: 1 network roundtrip (maybe can be coalesced with above?) • when connection ends, decrement count at remote redis for chosen dyno: 1 network roundtrip That's 2-3 extra roundtrips each inbound request, and n…

Really? Is that so hard? All you need is a table on the router that tells it the best information it can currently have about the number of requests processed on each dyno - without doing a roundtrip. This requires exactly one additional (one-way) package: A message from the dyno to the router, telling it that it has finished the current request. Now, to avoid dead dynos (because the finished message might have been…

A problem with this proposal is the assumption that there is one 'the router' with this info, updated for tens of thousands of dynos and millions of requests per second.

Re: Heroku - Bamboo Routing Performance

#80

Post by jacques_chester »

bad_user wrote: post #68
redguava wrote: post #62

I don't understand why people think this is a great response. They know how their routing works, just say so. It can't be that hard to give a basic overview of it before they release a more comprehensive post. As for the comment "Improving our documentation and website to accurately reflect our product". That is a very round about way of saying "our website indicates our service does things that it does not" which is…

Developers are smart and capable people, but boy, what spoiled brats we are. > It can't be that hard to give a basic overview of it before they release a more comprehensive post Well, actually it is very hard to give a basic overview of anything complicated, without leaving your readership with their heads scratching. The person that wrote the blog post might not even understand the whole stack to be in the position…

> Developers are smart and capable people, but boy, what spoiled brats we are.

For expecting providers to provide what they say they're providing?

The scoundrels.

Post reply on HN