Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku Blog: Routing Performance Update

blog.heroku.com

Post reply on HN
Print view View original

71–80 of 197 posts

Re: Heroku Blog: Routing Performance Update

#71

Post by orensol »

I think the discussion comes down to which option you ultimately prefer -- 1) be in full control of your stack, having only yourself to criticize, and having to deal with all the devops action that comes with it, or 2) let heroku deal with the underlying while you focus on your app, as their promise suggests, and suffer (also financially) from time to time due to faults like this.

Everyone has their own reasoning on this decision, and that's ok. I think that the voices here saying "I won't start my project on heroku because of this" are not thinking this through. I think that for any project starting up, focusing on app rather than infrastructure is much more important than routing issues.

Re: Heroku Blog: Routing Performance Update

#72

Post by ctide »

gfodor wrote: post #70
ctide wrote: post #68

Earlier quoted context omitted.

Whatever you do, get your database off Heroku. Heroku's postgres offering has sweet options, but performance is fucking awful. We went from a 20k+ month heroku bill to a fraction of that price on EC2 so we could get a slave that could keep up with the writes to our master. Performance being shit would be fine if it wasn't a miserable experience trying to migrate off. Not being able to setup replication to slaves off…

Interesting -- can you go into more detail or provide references that you found helpful when trying to run Postgres on ec2?

Yeah, if you run Ubuntu, don't try running any versions other than 11.04 if you're going to be under heavy i/o. We had all kinds of CPU soft lock issues with software raid on other versions. Otherwise? 2 m2.4xlarge running mostly standard postgresql installs absolutely dominate Mecha instances.

Obviously, you need to deal with backups and that sort of thing, but, you can hire a full time person for the price difference between Heroku and EC2 as soon as you get to the top end of Heroku pricing.

Re: Heroku Blog: Routing Performance Update

#73

Post by fdr »

ctide wrote: post #68
edouard1234567 wrote: post #62

Earlier quoted context omitted.

I love how their first suggestion is upgrade your db $$$ .

Whatever you do, get your database off Heroku. Heroku's postgres offering has sweet options, but performance is fucking awful. We went from a 20k+ month heroku bill to a fraction of that price on EC2 so we could get a slave that could keep up with the writes to our master. Performance being shit would be fine if it wasn't a miserable experience trying to migrate off. Not being able to setup replication to slaves off…

That's quite interesting; I don't see how one can really so soundly beat the disk configuration, but it certainly can be matched by something similar.

It was posted about quite some time ago: http://orion.herokuapp.com/past/2009/7/29/io_performance_on_.... If this has become bad advice, inquiring minds wish to know. EBS Priops seem promising and different.

Replication out of Heroku would be neat, but the WAL as-is is architecture and operating system dependent. Luck and tenaciousness willing, logical replication in Postgres may yet serve that use case, some day -- we'll have to see as to the details. A lot of effort has been expended attempting to get it in some form into 9.3, but it's far from sure thing.

Also, don't forget to set up archiving, if you continue to retain fast interconnect with S3: https://github.com/wal-e/wal-e, which I principally wrote on Heroku's behalf, but now is a project of multiple users and contributors. It has a focus on ease of use and reliability, although it could suffer improvement.

Re: Heroku Blog: Routing Performance Update

#74

Post by edouard1234567 »

MediaSquirrel wrote: post #69
edouard1234567 wrote: post #63

Earlier quoted context omitted.

It'd be like suing an airline for a delayed flight.

In a way, yes––if the airline's product for sale wasn't a particular "flight," but instead a speedy "travel system" that guaranteed customers to fly on the first available (ergot fastest) flight. To further extend the airline analogy, it's as if you had bought one of these "fast travel" products and were told to buy more plane tickets to avoid delays! The tort (aka crime) committed is the deceitful way in which natur…

They don't really claim to be the most reliable or performant, just the easiest to deploy and maintain (arguably scale). Keep in mind they don't even have any meaningful SLA.

Re: Heroku Blog: Routing Performance Update

#75

Post by tomjen3 »

clicks wrote: post #64
minimaxir wrote: post #58

Earlier quoted context omitted.

A Y-Combinator company suing another Y-Combinator company. Now that would be interesting.

Heroku was acquired by Salesforce, so I guess it's not really a YC company anymore.

Ah that explains why they have turned evil. Usually once a company has been acquired they are no longer worth using.

Re: Heroku Blog: Routing Performance Update

#76

Post by azarias »

It seems to me that Heroku has chosen to be dishonest:

Heroku's blog response: "but until this week, we failed to see a common thread among these reports."

vs.

Adam's response to Tim Watson, a year ago:

"You're correct, the routing mesh does not behave in quite the way described by the docs. We're working on evolving away from the global backlog concept in order to provide better support for different concurrency models, and the docs are no longer accurate. The current behavior is not ideal, but we're on our way to a new model which we'll document fully once it's done."

https://groups.google.com/forum/?fromgroups=#!msg/heroku/8eO...

Re: Heroku Blog: Routing Performance Update

#77

Post by boundlessdreamz »

msoad wrote: post #34

I am not a Rails developer nor a sys admin. I didn't really understand what was the problem the reported via Rap Genius. My question is does it affect Node.js apps?

It will. If X is the max number of requests node can handle per dyno, the router can send >X requests to a single dyno even if you have other dynos which are processing Essentially the router sends requests by random. This results in request pileups. In the original rap genius post, there is an animated image which shows this in action

Re: Heroku Blog: Routing Performance Update

#78

Post by neebz »

latchkey wrote: post #49

I'm watching all this debate with great interest. The company I'm consulting for has ~150 dynos, a paid support contract, etc. We're on Cedar... NodeJS latest (which just got upgraded to 0.8.19 after being ignored since October), with Mongo, Rabbit and Redis hosted by third party addon companies. After a ton of H12 errors, they helped us find out some slow points and optimize some things that were relatively slow. On…

150 dynos with Node.js and you are still getting H12 errors?

The issue so far reported is that if we ever get multiple requests on a single dyno then we'll have a queuing delay because Rails is essentially single-threaded. But with Node.js I think it would be a fairly large amount of requests on a single dyno I suspect before we get any queuing delays

Re: Heroku Blog: Routing Performance Update

#79

Post by ctide »

fdr wrote: post #73
ctide wrote: post #68

Earlier quoted context omitted.

Whatever you do, get your database off Heroku. Heroku's postgres offering has sweet options, but performance is fucking awful. We went from a 20k+ month heroku bill to a fraction of that price on EC2 so we could get a slave that could keep up with the writes to our master. Performance being shit would be fine if it wasn't a miserable experience trying to migrate off. Not being able to setup replication to slaves off…

That's quite interesting; I don't see how one can really so soundly beat the disk configuration, but it certainly can be matched by something similar. It was posted about quite some time ago: http://orion.herokuapp.com/past/2009/7/29/io_performance_on_... . If this has become bad advice, inquiring minds wish to know. EBS Priops seem promising and different. Replication out of Heroku would be neat, but the WAL as-is i…

Yeah, we're using wal-e in conjunction with streaming replication.

Either of those options would have been awesome (anything but Bucardo, really. Bucardo's triggers overwhelmed an already overwhelmed database)

Re: Heroku Blog: Routing Performance Update

#80

Post by [deleted] »

latchkey wrote: post #49

I'm watching all this debate with great interest. The company I'm consulting for has ~150 dynos, a paid support contract, etc. We're on Cedar... NodeJS latest (which just got upgraded to 0.8.19 after being ignored since October), with Mongo, Rabbit and Redis hosted by third party addon companies. After a ton of H12 errors, they helped us find out some slow points and optimize some things that were relatively slow. On…

[deleted]
Post reply on HN