Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku - Bamboo Routing Performance

blog.heroku.com

Post reply on HN
Print view View original

111–120 of 152 posts

Re: Heroku - Bamboo Routing Performance

#111

Post by praptak »

nikcub wrote: post #75

There is a perverse conflict with platform service providers - the worse your scheduler performs the more profitable your service will be. You replace intelligent request scheduling with more hardware and instances, which you charge the user for. How much investment is there in platform service providers towards developing better schedulers that would reduce the number of instances required to serve an application? T…

> There is a perverse conflict with platform service providers - the worse your scheduler performs the more profitable your service will be.

I think the practical significance of this kind of incentives is overrated. The company I work for does outsourcing work, paid by hour. Do they have incentives to make me produce less so that their customers pay for more hours? Theoretically. Do they act on it? Hell, no - there is competition and customer satisfaction matters.

Re: Heroku - Bamboo Routing Performance

#112

Post by cpressey »

tomlemon wrote: post #32
seivan wrote: post #6

Wait, so those guys were on Bamboo, and complaining? Fuck, that is so not cool. We've been on cedar ever since it launched, and been running puma threads or unicorn workers. The idea of one dyno per request is bullshit, and I wasn't sure if they were on cedar or not. A dyno is an allocated resource (512mb, not counting db, k/v store etc) How ballsy of them to complain when they are doing it wrong.

Rap Genius cofounder: Random routing is still wayyyyy worse than intelligent routing even if you're processing multiple requests simultaneously on a single dyno (see http://rapgenius.com/1504222 ) Requests will still queue, but since time spent queuing at the dyno level doesn't appear in your logs or on New Relic, you'll never know it.

I think that's one of the real tragedies under all this, and compounding the tragedy is that it's going under-discussed: the performance monitoring is incomplete. If the dyno level queue time was apparent, this issue would have become obvious much sooner.

Re: Heroku - Bamboo Routing Performance

#113

Post by drchiu »

What I find incredibly irritating about this blog response by Heroku is that it took a very visible post on Hackernews for them to act and reconsider their way of doing business.

They saw the potential loss in customers, and then acted. What this means is that they never had in mind to provide the best support and product they could for their customers before this news broke out.

Sad.

Re: Heroku - Bamboo Routing Performance

#114

Post by programminggeek »

Wow, I feel like Heroku is really dropping the ball here. Like, they are acting punch drunk or something. Basically all this says is "we hear you and we are sorry". They could have posted that a day ago. This still says nothing about what is wrong and what they are doing to fix it.

Also, I'm not sure at what point this is, but at some point around say $3-5k a month, (100+ dynos) you really should rethink using Heroku. At that point and higher, you really ought to know about your infrastructure enough to optimize for scale. The "just add more dynos" approach is stupid because adding more web fronts is often the lazy/expensive approach. Add a few queues or some smarter caching and you'll need fewer web servers. Throw in something like Varnish where you can and you need even fewer servers. Point being, at some point scaling is no longer "free", it takes work and Heroku isn't magic.

Re: Heroku - Bamboo Routing Performance

#115

Post by badgar »

philwelch wrote: post #91
bad_user wrote: post #68

Earlier quoted context omitted.

Developers are smart and capable people, but boy, what spoiled brats we are. > It can't be that hard to give a basic overview of it before they release a more comprehensive post Well, actually it is very hard to give a basic overview of anything complicated, without leaving your readership with their heads scratching. The person that wrote the blog post might not even understand the whole stack to be in the position…

> Dude, this is a service used by developers and devops. Developers, sure. Devops? Heroku's entire purpose is so you don't have to care about ops. > You aren't meant to trust what they say. Evidently not! I'd rather not do business with liars. > They acknowledge sincerely that they have a problem, with a promise that they are going to fix it. That's an optimistic interpretation of their blog post.

> Evidently not! I'd rather not do business with liars.

Is the entire startup world this naïve, or just this particular commenter?

Re: Heroku - Bamboo Routing Performance

#116

Post by imbriaco »

bitcartel wrote: post #73
redguava wrote: post #70

Earlier quoted context omitted.

In answer to "who cares if the explanation comes today or tomorrow", I care if the explanation comes today or tomorrow. I use Heroku and have hit scaling issues in the last few weeks very similar to this. More information on what is going on behind the scenes will help me immediately. As for discrepancy in documentation, this is one of the most major parts of their infrastructure and directly relates to how well appl…

Well, for marketing purposes, random doesn't sound as impressive as intelligent. It's a discrepancy but it does appear to be disclosed. On the page you link to, it says: "Incoming web traffic is automatically routed to web dynos, with intelligent distribution of load instantly as you scale." When you click on "Read more about routing..." it says: "Request distribution - The routing mesh uses a random selection algori…

I'm really confused where people are coming up with the idea that the word "intelligent" is somehow a synonym for "queueing". That doesn't make any sense to me.

The intelligent part of it is that it knows which servers and ports your application is running on as dynos come and go, and the routers is updated in near real time to reflect these changes. Since dynos move about frequently, and there are a great many of them starting and stopping per second, this requires much more intelligence than you probably think.

Re: Heroku - Bamboo Routing Performance

#118

Post by jules »

nostrademons wrote: post #77
nikcub wrote: post #75

There is a perverse conflict with platform service providers - the worse your scheduler performs the more profitable your service will be. You replace intelligent request scheduling with more hardware and instances, which you charge the user for. How much investment is there in platform service providers towards developing better schedulers that would reduce the number of instances required to serve an application? T…

In theory, the service provider could just switch to a better scheduler and raise their rates to compensate, since the TCO for a customer would go down correspondingly. Their costs go down, their profits go up, and customers still benefit. Prices are usually determined more by competition than by technical factors. Unfortunately, there's a fair bit of lock-in to PaaS vendors, which is the real reason I'd be skeptical…

What you could do is price based on the value you provide.

You could price per request that you serve for your customers. Of course that just puts the perverse incentive on your customers, since now the service provider is bearing the cost of a nonoptimized application. The client doesn't care much about the weight of a single request, since the cost to him is the same (well except for latency).

So what you could do is define a 'typical request' by running a suite of open source web application on the service provider. This way you see how much hardware is required for a given throughput of 'typical requests'. Put a price tag on that. Now pricing also becomes much clearer for the customers: they know that if their application's requests are roughly of the same weight as open source web application X, Y and Z, and if they serve N requests per month, then their cost will be roughly N*(price of 1 typical request) per month.

Now the service provider has the incentive to optimize its hardware and services, because then it can run more typical requests at the same cost. The client has the incentive to optimize his application, because then he will use fewer typical request units, and hence pay less.

You could also determine the cost of a typical request by averaging over all of your clients' applications, instead of a suite of open source applications.

Re: Heroku - Bamboo Routing Performance

#119

Post by 7952 »

programminggeek wrote: post #114

Wow, I feel like Heroku is really dropping the ball here. Like, they are acting punch drunk or something. Basically all this says is "we hear you and we are sorry". They could have posted that a day ago. This still says nothing about what is wrong and what they are doing to fix it. Also, I'm not sure at what point this is, but at some point around say $3-5k a month, (100+ dynos) you really should rethink using Heroku…

At $3-5k a month Heroku may as well start offering a consultancy service rather than hosting. Wanting unlimited scaling without needing local talent is a reasonable thing to want, but its unrealistic to expect if from one single platform.
Post reply on HN