Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku Blog: Routing Performance Update

blog.heroku.com

Post reply on HN
Print view View original

121–130 of 197 posts

Re: Heroku Blog: Routing Performance Update

#121

Post by antirez »

ig1 wrote: post #107
raverbashing wrote: post #96

Earlier quoted context omitted.

Kind of depends There's the "two guys in a dorm eating ramen" and there's the YC backed "startup" that already has a couple of hires So if your monthly budget is of hundreds of dollars, using a VPS instead of Heroku makes sense

Let say your average http request takes 200ms to serve, then with 5 heroku worker dynos you can serve about 10 million requests a day which should comfortably cover the requirements of most startups. How much do those dynos cost ? - $143/month. That's less than three hours salary for a developer. If your monthly budget is hundreds of dollars I'm guessing your traffic is low enough you can just use the free tiers of h…

10,000,000 requests each day are 115 requests per second. I don't follow your math here with a 200ms CPU time per request.

Re: Heroku Blog: Routing Performance Update

#122

Post by antirez »

nthj wrote: post #103
antirez wrote: post #90

Come on startups, you should be technically skilled and able to optimize in order to spend little money . If you sum EC2 and Heroku you are going to pay like 10x what it takes to run the same machines power in a dedicated server, all this because you can't handle the operations? This is absurd IMHO. Also people that want to start a business, there is a huge opportunity here , create software that makes managing Apach…

> Also people that want to start a business, there is a huge opportunity here, create software that makes managing Apache, Redis, PostgreSQL, ..., in dedicated servers very easy and robust. Traget a popular and robust non-commercial distribution like Ubuntu LTE, and provide all is needed to deploy web nodes, database nodes, with backups, monitoring, and everything else trivial. So the pitch is: clone Heroku, which ha…

That's what I think about your reasoning.

1) Once you develop a culture of PaaS the reality is, if you check the history of most startups, you'll continue along this way, and the money you'll burn will be massive. I suspect that if you build a sysop culture earlier, later you'll be able to just hire a full time sysop instead of spending a lot more.

2) If you don't understand very well your platform from the point of view of operations, likely you'll build a non scalable system (scalable not just from the point of view, let's distribute the load among N nodes, but about the constant times / CPU / energy it takes to serve a request). You'll end spending all the time again porting your app to another framework / language. This happens all the times.

3) The first months of a startup are usually not so critical from the point of view of operations, it is unlikely you'll spend a lot of time managing servers, unless...

Unless you selected a platform that is too complex to start with. I think that many of the PaaS companies I see today are here mainly for two reasons: Rails & Java frameworks.

So first of all, select your platform wisely.

Re: Heroku Blog: Routing Performance Update

#123

Post by oellegaard »

matt2000 wrote: post #112
jacques_chester wrote: post #61

Earlier quoted context omitted.

> 1) Releasing a press release at 7 AM in the morning on a Saturday (CET) I think they aimed to put out a response ASAP.

I agree, the other way to look at this is that they're working through the weekend to figure it out.

That would be great - more specifically if they would release a new one on Monday stating that they found a solution. I still don't like the concept of using a PaaS and having to discover these kind of things yourself.

Re: Heroku Blog: Routing Performance Update

#124

Post by jonny_eh »

nahname wrote: post #117
antirez wrote: post #90

Come on startups, you should be technically skilled and able to optimize in order to spend little money . If you sum EC2 and Heroku you are going to pay like 10x what it takes to run the same machines power in a dedicated server, all this because you can't handle the operations? This is absurd IMHO. Also people that want to start a business, there is a huge opportunity here , create software that makes managing Apach…

>"I can only write my Ruby code but can't handle operations" is not the right attitude, grow up. How is such a condescending post at the top? Everyone running a startup is an idiot because they choose not to waste their time on your priority? Heroku isn't 10 times more and it wouldn't matter even if it was. Talented people are hard to find and spending time on operations when you might not be around in 4 months may n…

The annoying thing about that post is that it derails the conversation. The top post should be analyzing Heroku's response, whether it was sufficient or not. Or how developers can mitigate Heroku's dumb-by-design routing issue.

Re: Heroku Blog: Routing Performance Update

#125

Post by steveklabnik »

> On the Cedar stack, the root cause is the fact that Cedar is optimized for concurrent request routing, while some frameworks, like Rails, are not concurrent in their default configurations.

First of all, Ruby doesn't exactly block on I/O [1], so there's that. Secondly, while Rails 3 has config.threadsafe! turned off, Rails 4 will have it turned on. [2]

1: https://github.com/tenderlove/fibur#synopsis 2: http://tenderlovemaking.com/2012/06/18/removing-config-threa...

EDIT: I just realized non-Rubyists might not get the Fibur joke. Here's the code: https://github.com/tenderlove/fibur/blob/master/lib/fibur.rb...

Re: Heroku Blog: Routing Performance Update

#126

Post by WillieBKevin »

jlouis wrote: post #118
WillieBKevin wrote: post #110

Earlier quoted context omitted.

"Forgetting who is right and wrong" The right vs wrong issue here is not the proper way to architect a router. The issue is that Heroku glossed over an extremely important aspect of their engineering documentation, because it painted their platform in a bad light. This is particularly damning, because as an engineer working on their platform, I could design around their shortcomings as long as they don't hide them fr…

The reason I want to stay out of that discussion is that it often just amounts to some mud-slinging from one side upon the other. Intentionally mislead is quite an accusation and I don't think Heroku would indulge in it. Personally, I think it is incredibly naive to build an application around a framework where you have no built-in concurrency. The main reason is that the queue you will build up in front of it is out…

It is quite an accusation, but I'm far from a bystander in this issue. I have extensive, documented communication with Heroku engineers over the course of 1.5 years (Feb 2011 - June 2012).

I'm not discussing any sort of cooked up statistical models. I'm discussing the real-world experience I had scaling an application on Heroku.

"mud-slinging from one side upon the other"

You imply that I was uninvolved before Rap Genius's expose. I assure you that is not the case. I've chosen a side in this argument well before Rap Genius went public.

Re: Heroku Blog: Routing Performance Update

#127

Post by dylanvee »

latchkey wrote: post #60
dylanvee wrote: post #32

For what it's worth, Google App Engine uses a so-called "intelligent" global request queue/scheduler. In most circumstances it's quite effective.

It is effective, but not without problems. http://code.google.com/p/googleappengine/issues/detail?id=78... http://code.google.com/p/googleappengine/issues/detail?id=57... http://code.google.com/p/googleappengine/issues/detail?id=78... Needless to say, it is VASTLY better than Heroku's router. You'll never see an H12 error with GAE.

Indeed, while having a global request queue is good, the scheduler's one-size-fits-all behavior can sometimes be suboptimal. Thankfully there are several ways in which it can be tuned: https://developers.google.com/appengine/docs/adminconsole/pe...

Re: Heroku Blog: Routing Performance Update

#128

Post by dylanvee »

vetrom wrote: post #46
dylanvee wrote: post #32

For what it's worth, Google App Engine uses a so-called "intelligent" global request queue/scheduler. In most circumstances it's quite effective.

Source please?

"Pending request latency arises when all of your application's available instances are too busy to serve new requests. When this happens, incoming requests go to a pending request queue." https://developers.google.com/appengine/docs/adminconsole/pe...

Pending latency is exposed as a metric on the App Engine Dashboard.

Re: Heroku Blog: Routing Performance Update

#129

Post by jonny_eh »

jlgaddis wrote: post #2

> "... but until this week, we failed to see a common thread among these reports." So Rap Genius, a customer , was able to figure out the issues (from the outside looking in) but Heroku, "on the inside" wasn't able to figure them out? Or they're playing the "we didn't know, we're going to fix it right away" angle? EDIT: also, s/failed to/did not/ makes more sense. "failed" implies they tried.

To me it read as "we didn't know, we're not going to fix it right away (if ever), we'll just document it better"

Re: Heroku Blog: Routing Performance Update

#130

Post by btilly »

azarias wrote: post #76

It seems to me that Heroku has chosen to be dishonest: Heroku's blog response: "but until this week, we failed to see a common thread among these reports." vs. Adam's response to Tim Watson, a year ago: "You're correct, the routing mesh does not behave in quite the way described by the docs. We're working on evolving away from the global backlog concept in order to provide better support for different concurrency mod…

It is quite possible to know that the mesh is not exactly as documented without having realized that the difference has a severe performance impact.
Post reply on HN