Live data from Hacker News

Heroku Blog: Routing Performance Update

blog.heroku.com

61–70 of 197 posts

Re: Heroku Blog: Routing Performance Update

#61

What a scumbag move from Heroku. 1) Releasing a press release at 7 AM in the morning on a Saturday (CET) 2) The release looks mostly like the stuff a politicians spindoctor would ask the politician to say. Don't promis/admit too much. 3) They clearly state that they want to continue with this extremely inefficient way of routing. The right thing to do would be to make smaller clusters of Load Balancers who could then…

> 1) Releasing a press release at 7 AM in the morning on a Saturday (CET)

I think they aimed to put out a response ASAP.

Re: Heroku Blog: Routing Performance Update

#62

Rap Genius cofounder here. Below is the full unedited text of https://help.heroku.com/tickets/37665 , a Heroku support ticket I logged about 1 year ago. Sorry it is so long, but I think you'll find it interesting: Tom@Rapgenius| about 1 year ago I know this is a bit of a vague problem, but I've been getting a bunch of Error H12 (Request Timeout)s recently, and I'm not sure what to do about it. It's not like I have so…

I love how their first suggestion is upgrade your db $$$ .

Re: Heroku Blog: Routing Performance Update

#63

Rap Genius cofounder here. Below is the full unedited text of https://help.heroku.com/tickets/37665 , a Heroku support ticket I logged about 1 year ago. Sorry it is so long, but I think you'll find it interesting: Tom@Rapgenius| about 1 year ago I know this is a bit of a vague problem, but I've been getting a bunch of Error H12 (Request Timeout)s recently, and I'm not sure what to do about it. It's not like I have so…

Have you guys considered suing Heroku to get some of your money back? Given the nature of Heroku's deception and the resulting ill-gotten gains across its entire customer bases, it would seem like you could work with an enterprising attorney to form a class-action suit against the company and get money back not just for yourselves but for the entire effected customer bases. Just a thought. ;)

It'd be like suing an airline for a delayed flight.

Re: Heroku Blog: Routing Performance Update

#64

Earlier quoted context omitted.

Have you guys considered suing Heroku to get some of your money back? Given the nature of Heroku's deception and the resulting ill-gotten gains across its entire customer bases, it would seem like you could work with an enterprising attorney to form a class-action suit against the company and get money back not just for yourselves but for the entire effected customer bases. Just a thought. ;)

A Y-Combinator company suing another Y-Combinator company. Now that would be interesting.

Heroku was acquired by Salesforce, so I guess it's not really a YC company anymore.

Re: Heroku Blog: Routing Performance Update

#65
post #37

Hmm, no tangible solutions yet, but I expect that will be next. From the discussion I've seen they have roughly two minimal options: (1) Shard/tier the Bamboo routing nodes, so that a single router tends to handle any particular app, and thus the original behavior is restored. Consistent hashing on the app name could do the trick, or DNS tricks on the app names mapping to different routing subshards. (2) Enable dynos…

I would suggest a slight variation on the first. Have initial routers pick a random appropriate dyno then pick the router for that dyno. The second router picks a random appropriate dyno from the list it is in charge of.

If you make sure that the dynos for a given app are clumped behind few routers in the second layer, then you effectively get the old behavior. But you get it in a much more scaleable way. The cost is, of course, that you add an extra router to everything.

(I emailed this suggestion to them. I have no idea whether they will listen.)

Re: Heroku Blog: Routing Performance Update

#66
post #7

Depending on what side of Hanlon's razor you fall, the only conclusion I get from this is that they are either incompetent or dishonest. I have a very hard time believing that this issue remained unknown to them for years. As for the post, it's pretty much just documentation. I didn't see any apology. And the only promise of a better tomorrow is a vague "Working to better support concurrent-request Rails apps on Ceda…

I think apologies are over-demanded by our somewhat hysterical media that likes nothing better than to enhumble/humiliate a public figure (because it sells papers); and this flows through into expectations of private and corporate behaviour. But I've never had much use for apologies from other people. Years of abuse make "sorry" an entirely debased term in my lexicon. I've seen statements of regret that omit the word and are all the more sincere for it.

Much more useful than an apology is an acceptance of fault (which is not the same thing); an expression of desire to improve, and a sincere and demonstrable commitment to doing so.

[NB: don't mean to imply that Heroku have necessarily achieved all of that here]

Re: Heroku Blog: Routing Performance Update

#67
post #29

Welp, I was waiting for their official response to decide if I should deploy my app with Heroku or roll up my sleeves and rig up AWS servers (which I've done before but was looking forward to not having to deal with it.) Based upon this post, it sounds like there are really no concrete steps that they have planned to fix the underlying issue. So, AWS it is. I am still considering having Heroku manage my PostgreSQL in…

If your app is Railsian and will have requests that take a few seconds to be served, I'd suggest AWS. If it's ultimately a simple CRUD app then Heroku should be fine until you're at significant scale - and this issue will never be as severe.

Bit I wrote earlier about time-consuming requests - http://news.ycombinator.com/item?id=5216593

Re: Heroku Blog: Routing Performance Update

#68

Rap Genius cofounder here. Below is the full unedited text of https://help.heroku.com/tickets/37665 , a Heroku support ticket I logged about 1 year ago. Sorry it is so long, but I think you'll find it interesting: Tom@Rapgenius| about 1 year ago I know this is a bit of a vague problem, but I've been getting a bunch of Error H12 (Request Timeout)s recently, and I'm not sure what to do about it. It's not like I have so…

I love how their first suggestion is upgrade your db $$$ .

Whatever you do, get your database off Heroku. Heroku's postgres offering has sweet options, but performance is fucking awful. We went from a 20k+ month heroku bill to a fraction of that price on EC2 so we could get a slave that could keep up with the writes to our master.

Performance being shit would be fine if it wasn't a miserable experience trying to migrate off. Not being able to setup replication to slaves off of Heroku means you have to deal with things like Bucardo and all the problems that brings along.

Heroku is really awesome for little web apps that performance doesn't matter on. If you think you're building something that people will want to use, you don't want to be on Heroku.

Re: Heroku Blog: Routing Performance Update

#69

Earlier quoted context omitted.

Have you guys considered suing Heroku to get some of your money back? Given the nature of Heroku's deception and the resulting ill-gotten gains across its entire customer bases, it would seem like you could work with an enterprising attorney to form a class-action suit against the company and get money back not just for yourselves but for the entire effected customer bases. Just a thought. ;)

It'd be like suing an airline for a delayed flight.

In a way, yes––if the airline's product for sale wasn't a particular "flight," but instead a speedy "travel system" that guaranteed customers to fly on the first available (ergot fastest) flight.

To further extend the airline analogy, it's as if you had bought one of these "fast travel" products and were told to buy more plane tickets to avoid delays!

The tort (aka crime) committed is the deceitful way in which nature of the product was marketed to the clear financial benefit of Heroku and to the clear financial detriment of the customer. The fix offered to customers is always an up-sell effort to "buy more Heroku products."

The thing about Heroku's intelligent queing system is that it lowers customers' costs and thus Heroku's profits. Switching away from intelligent queuing to random queuing was a business decision with clear moneymaking advantages for Heroku.

In one fell swoop, the move made the value of a customer to Heroku skyrocket, potentially increasing its valuation upon acquisition and/or helping the founders more easily hit any earn-out revenue targets required by Salesforce as part of their acquisition.

Re: Heroku Blog: Routing Performance Update

#70
post #68

Earlier quoted context omitted.

I love how their first suggestion is upgrade your db $$$ .

Whatever you do, get your database off Heroku. Heroku's postgres offering has sweet options, but performance is fucking awful. We went from a 20k+ month heroku bill to a fraction of that price on EC2 so we could get a slave that could keep up with the writes to our master. Performance being shit would be fine if it wasn't a miserable experience trying to migrate off. Not being able to setup replication to slaves off…

Interesting -- can you go into more detail or provide references that you found helpful when trying to run Postgres on ec2?
Post reply on HN