Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku Blog: Routing Performance Update

blog.heroku.com

Post reply on HN
Print view View original

111–120 of 197 posts

Re: Heroku Blog: Routing Performance Update

#111

Post by matt2000 »

homosaur wrote: post #42
gfodor wrote: post #29

Welp, I was waiting for their official response to decide if I should deploy my app with Heroku or roll up my sleeves and rig up AWS servers (which I've done before but was looking forward to not having to deal with it.) Based upon this post, it sounds like there are really no concrete steps that they have planned to fix the underlying issue. So, AWS it is. I am still considering having Heroku manage my PostgreSQL in…

If you're considering Heroku, don't automatically dismiss it because of all this. For one, it's unlikely your site/app will ever be as big as RapGenius. That's not a shot, just reality. They ran into problems as an edge case. Is Heroku and their architecture at fault? Hell yeah it is, but I have faith that they will fix it. Why? Because I really think that when push comes to shove, Heroku was actually trying to do th…

Fwiw our site isn't as big as rap genius and we see highly variable performance and timeouts even on requests that on average only take 100ms. I'm not ready to blame heroku entirely since there's a lot of moving parts that still could be at fault, but after optimizing as much as we could we still see the behavior. It feels like the routing layer could be the most likely problem. Summary: you might see highly variable request time regardless of traffic levels.

Re: Heroku Blog: Routing Performance Update

#112

Post by matt2000 »

jacques_chester wrote: post #61
oellegaard wrote: post #47

What a scumbag move from Heroku. 1) Releasing a press release at 7 AM in the morning on a Saturday (CET) 2) The release looks mostly like the stuff a politicians spindoctor would ask the politician to say. Don't promis/admit too much. 3) They clearly state that they want to continue with this extremely inefficient way of routing. The right thing to do would be to make smaller clusters of Load Balancers who could then…

> 1) Releasing a press release at 7 AM in the morning on a Saturday (CET) I think they aimed to put out a response ASAP.

I agree, the other way to look at this is that they're working through the weekend to figure it out.

Re: Heroku Blog: Routing Performance Update

#113

Post by tolas »

ig1 wrote: post #107
raverbashing wrote: post #96

Earlier quoted context omitted.

Kind of depends There's the "two guys in a dorm eating ramen" and there's the YC backed "startup" that already has a couple of hires So if your monthly budget is of hundreds of dollars, using a VPS instead of Heroku makes sense

Let say your average http request takes 200ms to serve, then with 5 heroku worker dynos you can serve about 10 million requests a day which should comfortably cover the requirements of most startups. How much do those dynos cost ? - $143/month. That's less than three hours salary for a developer. If your monthly budget is hundreds of dollars I'm guessing your traffic is low enough you can just use the free tiers of h…

Well the point of all this is that simple scaling math like that doesn't add up because of the queuing latency introduced by heroku's routing layer. But still at that scale your point is valid.

Re: Heroku Blog: Routing Performance Update

#114

Post by raverbashing »

ig1 wrote: post #107
raverbashing wrote: post #96

Earlier quoted context omitted.

Kind of depends There's the "two guys in a dorm eating ramen" and there's the YC backed "startup" that already has a couple of hires So if your monthly budget is of hundreds of dollars, using a VPS instead of Heroku makes sense

Let say your average http request takes 200ms to serve, then with 5 heroku worker dynos you can serve about 10 million requests a day which should comfortably cover the requirements of most startups. How much do those dynos cost ? - $143/month. That's less than three hours salary for a developer. If your monthly budget is hundreds of dollars I'm guessing your traffic is low enough you can just use the free tiers of h…

Yes, but there's a step before

$143 is money that is missed by a college student, for example.

Better go with something cheaper or the free tiers of Heroku, AWS, etc

You're probably still developing the system, or having only a small amount of requests.

Re: Heroku Blog: Routing Performance Update

#115

Post by derefr »

nthj wrote: post #103
antirez wrote: post #90

Come on startups, you should be technically skilled and able to optimize in order to spend little money . If you sum EC2 and Heroku you are going to pay like 10x what it takes to run the same machines power in a dedicated server, all this because you can't handle the operations? This is absurd IMHO. Also people that want to start a business, there is a huge opportunity here , create software that makes managing Apach…

> Also people that want to start a business, there is a huge opportunity here, create software that makes managing Apache, Redis, PostgreSQL, ..., in dedicated servers very easy and robust. Traget a popular and robust non-commercial distribution like Ubuntu LTE, and provide all is needed to deploy web nodes, database nodes, with backups, monitoring, and everything else trivial. So the pitch is: clone Heroku, which ha…

No, presumably the pitch is: clone Heroku's software stack as an OSS project, then get the server/VPS sellers to let people opt to install it as a package when they rent out a node, the same way things like Wordpress can automatically be installed.

Re: Heroku Blog: Routing Performance Update

#116

Post by nthj »

derefr wrote: post #115
nthj wrote: post #103

Earlier quoted context omitted.

> Also people that want to start a business, there is a huge opportunity here, create software that makes managing Apache, Redis, PostgreSQL, ..., in dedicated servers very easy and robust. Traget a popular and robust non-commercial distribution like Ubuntu LTE, and provide all is needed to deploy web nodes, database nodes, with backups, monitoring, and everything else trivial. So the pitch is: clone Heroku, which ha…

No, presumably the pitch is: clone Heroku's software stack as an OSS project, then get the server/VPS sellers to let people opt to install it as a package when they rent out a node, the same way things like Wordpress can automatically be installed.

Sure, see OpenStack: http://www.openstack.org/

[that said, comparing automated scaling of a Ruby on Rails SaaS app with 100K+ users to a one-click WordPress install is kinda humorous. Even WordPress has a market for expert DevOps guys to manage your site: http://wpengine.com]

Re: Heroku Blog: Routing Performance Update

#117

Post by nahname »

antirez wrote: post #90

Come on startups, you should be technically skilled and able to optimize in order to spend little money . If you sum EC2 and Heroku you are going to pay like 10x what it takes to run the same machines power in a dedicated server, all this because you can't handle the operations? This is absurd IMHO. Also people that want to start a business, there is a huge opportunity here , create software that makes managing Apach…

>"I can only write my Ruby code but can't handle operations" is not the right attitude, grow up.

How is such a condescending post at the top?

Everyone running a startup is an idiot because they choose not to waste their time on your priority?

Heroku isn't 10 times more and it wouldn't matter even if it was. Talented people are hard to find and spending time on operations when you might not be around in 4 months may not be the most important thing to focus on. On a 6 to 8 month time scale, heroku would be 10 times cheaper that taking the hit of setting up everything to emulate it. Deployments, backups, monitoring. Those things take a lot of time to setup correctly.

Re: Heroku Blog: Routing Performance Update

#118

Post by jlouis »

WillieBKevin wrote: post #110
jlouis wrote: post #87

So to recap: Ruby on Rails is using a default configuration where each process can serve one request at a time. There is no cooperative switch (as in Node.js) or (near) preemptive switch (as in Erlang, Haskell, Go, ...). The routing infrastructure at Heroku is distributed . There are several routers and one router will queue at most one message per back-end dyno in the Bamboo stack and route randomly in the Cedar sta…

"Forgetting who is right and wrong" The right vs wrong issue here is not the proper way to architect a router. The issue is that Heroku glossed over an extremely important aspect of their engineering documentation, because it painted their platform in a bad light. This is particularly damning, because as an engineer working on their platform, I could design around their shortcomings as long as they don't hide them fr…

The reason I want to stay out of that discussion is that it often just amounts to some mud-slinging from one side upon the other. Intentionally mislead is quite an accusation and I don't think Heroku would indulge in it.

Personally, I think it is incredibly naive to build an application around a framework where you have no built-in concurrency. The main reason is that the queue you will build up in front of it is outside your reach so you have to sustain it.

It is also naive to think that your cooked up statistical model resembles reality in any way. Routing is a hard problem so it is entirely plausible that your model does not hold up in reality. Besides, the time it takes to construct those R models is a missed opportunity for improving the backend you have. And the R models doesn't say a lot, sorry. At best they just stir up the storm --- and boy did they succeed.

I agree it is unfortunate that Heroku's documentation isn't better and that New Relic doesn't provide the accurate latency statistics. But to claim that this is entirely Heroku's fault is, frankly, naive as well.

Re: Heroku Blog: Routing Performance Update

#119

Post by jlouis »

rst wrote: post #105
jlouis wrote: post #87

So to recap: Ruby on Rails is using a default configuration where each process can serve one request at a time. There is no cooperative switch (as in Node.js) or (near) preemptive switch (as in Erlang, Haskell, Go, ...). The routing infrastructure at Heroku is distributed . There are several routers and one router will queue at most one message per back-end dyno in the Bamboo stack and route randomly in the Cedar sta…

True for Rails 3, though you can enable threading (multiple requests per process). Rails 4 will likely default new apps threaded, but that option would still have to be manually enabled for older ones. (I say likely because there are no formal releases yet, and it's still possible stuff will get pulled back.)

It was news to me that Rails operated in this way. It is a very weak concurrency model (to be precise: it is nonexistent). I knew that Rails was naive, but I had not dawned on me just how naive it is. It is interesting because then a system like node.js is a giant leap in the right direction - even if there are better languages out there.

(disclaimer: I do Erlang for a living and often operate in highly concurrent settings.)

Re: Heroku Blog: Routing Performance Update

#120

Post by Tloewald »

wging wrote: post #4
jlgaddis wrote: post #2

> "... but until this week, we failed to see a common thread among these reports." So Rap Genius, a customer , was able to figure out the issues (from the outside looking in) but Heroku, "on the inside" wasn't able to figure them out? Or they're playing the "we didn't know, we're going to fix it right away" angle? EDIT: also, s/failed to/did not/ makes more sense. "failed" implies they tried.

If you want a press release to be ignored, release it at EOD on a Friday.

...after a meteor explodes over Russia.
Post reply on HN