Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku Blog: Routing Performance Update

blog.heroku.com

Post reply on HN
Print view View original

181–190 of 197 posts

Re: Heroku Blog: Routing Performance Update

#181

Post by jlouis »

wmf wrote: post #149
jlouis wrote: post #119

Earlier quoted context omitted.

It was news to me that Rails operated in this way. It is a very weak concurrency model (to be precise: it is nonexistent ). I knew that Rails was naive, but I had not dawned on me just how naive it is. It is interesting because then a system like node.js is a giant leap in the right direction - even if there are better languages out there. ( disclaimer: I do Erlang for a living and often operate in highly concurrent…

Rails has a pure shared-nothing scale-out philosophy; it achieves concurrency with multiple processes. The fact that Heroku only runs one process per dyno by default is a shame though.

It won't help much for several reasons:

Context switches are expensive. Very expensive. So your system runs overhead in the operating system. And you need to spawn a process per request.

Pooling processes is a problem as well. If we only have 4 workers in the pool, then we have to queue requests on one of the 4 workers. But we don't know how expensive those requests are to serve, a priori. Even knowing the queue length or the queue sojourn time won't be able to divulge this information to us, only help a little. More workers just push the problem out further.

If you want to be fast, you need:

The ability to switch between work quickly, in the same process.

The ability to interleave expensive work with cheap work.

The two main solutions are evented servers: Node.js, Twisted, Tornado (both python); and preemptive runtimes: Go, Haskell, Erlang (of which Erlang is the only truly preemptive). I much prefer the preempted solution because it is automatic and you don't have to code anything yourself.

There is a strong similarity to cooperative and preemptive multitasking in operating systems by the way. Events are cooperative. Do note there are no more cooperative operating systems around which you use on a daily basis :)

Re: Heroku Blog: Routing Performance Update

#182

Post by ansgri »

ig1 wrote: post #107
raverbashing wrote: post #96

Earlier quoted context omitted.

Kind of depends There's the "two guys in a dorm eating ramen" and there's the YC backed "startup" that already has a couple of hires So if your monthly budget is of hundreds of dollars, using a VPS instead of Heroku makes sense

Let say your average http request takes 200ms to serve, then with 5 heroku worker dynos you can serve about 10 million requests a day which should comfortably cover the requirements of most startups. How much do those dynos cost ? - $143/month. That's less than three hours salary for a developer. If your monthly budget is hundreds of dollars I'm guessing your traffic is low enough you can just use the free tiers of h…

And the real problem is the non-uniformity of the load. If your 10M/mo request start concentrating around weekends, you've got a problem... Except you haven't, as your infrastructure is elastic, and this is the real bonus.

Also, I'd be very afraid to get accidentally hit by HN traffic if I'd host my own server.

Re: Heroku Blog: Routing Performance Update

#183

Post by thenomad »

nthj wrote: post #116
derefr wrote: post #115

Earlier quoted context omitted.

No, presumably the pitch is: clone Heroku's software stack as an OSS project, then get the server/VPS sellers to let people opt to install it as a package when they rent out a node, the same way things like Wordpress can automatically be installed.

Sure, see OpenStack: http://www.openstack.org/ [that said, comparing automated scaling of a Ruby on Rails SaaS app with 100K+ users to a one-click WordPress install is kinda humorous. Even WordPress has a market for expert DevOps guys to manage your site: http://wpengine.com ]

And there are some VERY good reasons to invest in DevOps even for Wordpress.

I'd write a lengthy post on this subject right here, but I already did so elsewhere: http://www.mmomeltingpot.com/2012/03/wpengine-review-after-1...

(And Patio11 wrote on the same topic a while ago, too: http://www.kalzumeus.com/2012/02/09/why-i-dont-host-my-own-b... )

Re: Heroku Blog: Routing Performance Update

#184

Post by jlouis »

WillieBKevin wrote: post #126
jlouis wrote: post #118

Earlier quoted context omitted.

The reason I want to stay out of that discussion is that it often just amounts to some mud-slinging from one side upon the other. Intentionally mislead is quite an accusation and I don't think Heroku would indulge in it. Personally, I think it is incredibly naive to build an application around a framework where you have no built-in concurrency. The main reason is that the queue you will build up in front of it is out…

It is quite an accusation, but I'm far from a bystander in this issue. I have extensive, documented communication with Heroku engineers over the course of 1.5 years (Feb 2011 - June 2012). I'm not discussing any sort of cooked up statistical models. I'm discussing the real-world experience I had scaling an application on Heroku. "mud-slinging from one side upon the other" You imply that I was uninvolved before Rap Ge…

The "you" were addressed to RG, not your experience. I completely agree that scaling is one of the harder problems in computer science. Most people just run a stateless system and hope for the best, but this is a delicate matter and it is a hard problem.

The hard part being a customer or Heroku is that the problem might be on the other side of the fence. And how do you communicate that in a diplomatic way?

Personally my opinion is something along the lines of "If you use Ruby +rails in that configuration, then you deserve the problem".

Re: Heroku Blog: Routing Performance Update

#185

Post by ansgri »

pytrin wrote: post #138
antirez wrote: post #90

Come on startups, you should be technically skilled and able to optimize in order to spend little money . If you sum EC2 and Heroku you are going to pay like 10x what it takes to run the same machines power in a dedicated server, all this because you can't handle the operations? This is absurd IMHO. Also people that want to start a business, there is a huge opportunity here , create software that makes managing Apach…

My thoughts exactly. Like you don't outsource dev, you can't outsource sysops, it's a core competency to a tech startup.

Is it not more like you can't outsource customer support and product development, but you'd better outsource everything else if it's economically viable?

Thus if your product is some kind of hosting (i.e. blog or image hosting), the reliability is your product, so you don't outsource sysops, but if you offer some project management software... Well it's almost certainly hosted so reliability is your product again and you better have your own sysops.

So the only viable conclusion seems like you limit your trust to the PaaS provider and have some plan for the rare case when they get screwed. Or you could bet on technology not requiring such monstrous resources as Rails and host your entire MegaCorp on a single server rack.

Re: Heroku Blog: Routing Performance Update

#186

Post by thenomad »

antirez wrote: post #90

Come on startups, you should be technically skilled and able to optimize in order to spend little money . If you sum EC2 and Heroku you are going to pay like 10x what it takes to run the same machines power in a dedicated server, all this because you can't handle the operations? This is absurd IMHO. Also people that want to start a business, there is a huge opportunity here , create software that makes managing Apach…

Fail fast.

I can perfectly well install, set up and maintain my own Ruby servers - but it takes time.

Alternatively, I can pay someone else to do that, and remove that timesink from the elapsed time between "start developing" and "find out how well we've achieved market fit".

I can always optimise later - move off Heroku, develop our own load balancing, all that stuff. Once I've got a working product/market fit, I probably will.

But doing that before I know if I'm going to chuck the entire infrastructure in the garbage and move on to idea #2 (and #3, and #4...), or indeed pivot so wildly that we'll have to reorganise all our server stuff anyway, is a waste of time. And time is valuable.

Re: Heroku Blog: Routing Performance Update

#187

Post by namidark »

thenomad wrote: post #186
antirez wrote: post #90

Come on startups, you should be technically skilled and able to optimize in order to spend little money . If you sum EC2 and Heroku you are going to pay like 10x what it takes to run the same machines power in a dedicated server, all this because you can't handle the operations? This is absurd IMHO. Also people that want to start a business, there is a huge opportunity here , create software that makes managing Apach…

Fail fast. I can perfectly well install, set up and maintain my own Ruby servers - but it takes time. Alternatively, I can pay someone else to do that, and remove that timesink from the elapsed time between "start developing" and "find out how well we've achieved market fit". I can always optimise later - move off Heroku, develop our own load balancing, all that stuff. Once I've got a working product/market fit, I pr…

Chef/Puppet/Stackscripts/Salt/etc, and develop your own load balancing? Why on earth would you do that instead of just using HAProxy?

Re: Heroku Blog: Routing Performance Update

#188

Post by Jare »

latchkey wrote: post #174
Jare wrote: post #168

Earlier quoted context omitted.

I'm still confused: since you're using NodeJS, I imagine your dynos are effectively handling a large amount of concurrent requests. This should in turn negate any impact of long-running requests, since they don't cause further requests to be queued in any way. So, where are requests (or responses) being queued (or lost) in your app? In Rabbit? Are you getting errors and slowness as streaks rather than isolated random…

Node handles one request at a time. It isn't multithreaded. It will receive a request, process that request and return a response. If another request comes in at the same time another request is in process, it is queued until the currently processing request is finished. I googled around, here is a good explanation for you. http://howtonode.org/understanding-process-next-tick The way my application worked is that we…

> Node handles one request at a time. It isn't multithreaded. It will receive a request, process that request and return a response. If another request comes in at the same time another request is in process, it is queued until the currently processing request is finished.

I'm missing something here. Node does not multithread requests, but it surely can process many requests simultaneously if these requests are waiting for async operations: database, external APIs or other types of I/O usually. That's the very core idea of evented servers.

So, my model is that i.e. if a node process receives 100 requests over the period of 1 second, and each request takes 3 seconds to process but most of that time is spent waiting for async, then the 100 responses will be sent back essentially 3 seconds after they arrived, no queuing to speak of.

From your description, routers do not send multiple requests to the same dyno even if dynos could handle them, and only have a limited amount of dynos they talk to. So queuing is happening in the routers, while dynos idle away waiting for async.

This would be complementary to the problem described by Rapgenius, and mean that the Heroku architecture does not play well with any type of server, neither evented (Node, yours) nor sequential (Rails, as shown by Rapgenius) nor presumably multithreaded or multiprocess (which effectively behaves like evented to the outside world). A huge mess indeed!

Re: Heroku Blog: Routing Performance Update

#189

Post by latchkey »

Jare wrote: post #188
latchkey wrote: post #174

Earlier quoted context omitted.

Node handles one request at a time. It isn't multithreaded. It will receive a request, process that request and return a response. If another request comes in at the same time another request is in process, it is queued until the currently processing request is finished. I googled around, here is a good explanation for you. http://howtonode.org/understanding-process-next-tick The way my application worked is that we…

> Node handles one request at a time. It isn't multithreaded. It will receive a request, process that request and return a response. If another request comes in at the same time another request is in process, it is queued until the currently processing request is finished. I'm missing something here. Node does not multithread requests, but it surely can process many requests simultaneously if these requests are waiti…

> Node does not multithread requests, but it surely can process many requests simultaneously if these requests are waiting for async operations: database, external APIs or other types of I/O usually. That's the very core idea of evented servers.

Within a single request, Node can async its dealing with outside services (databases, api's, etc), but it is still only processing one request at a time. There is no 'synchronized' keyword in javascript. ;-)

Check this out:

https://devcenter.heroku.com/articles/http-routing#heroku-he...

There is an interesting header in there: X-Heroku-Dynos-In-Use. From what I understand, this header is the number of dynos that a router is communicating with. For us, this is always around 2-3.

I suspect that the router is just a dumb nginx process sitting in front of my app. It is setup to communicate with 2-3 of my dyno's in a round robin fashion. If any one of those dyno's doesn't process the request fast enough, then requests start to back up. Once requests start to back up past 30s worth of execution, the router starts just killing those queued requests instead of just leaving them in a queue or sending the requests to another set of dyno's. Even worse is if you have a dyno that crashes (nodejs likes to crash at the first sign of an exception). I suspect that is why we see 2 or 3 in that header.

I think that part of the problem is that the routers don't just start talking to more dyno's if you have them available. So, it doesn't matter if you have 50, 200, 500 dyno's because the router is always only talking to a small subset of them. Even if you add more dyno's in the middle of heavy requests, you are still stuck with H12 for the existing dyno's. A full system restart is necessary then.

Re: Heroku Blog: Routing Performance Update

#190

Post by Jare »

latchkey wrote: post #189
Jare wrote: post #188

Earlier quoted context omitted.

> Node handles one request at a time. It isn't multithreaded. It will receive a request, process that request and return a response. If another request comes in at the same time another request is in process, it is queued until the currently processing request is finished. I'm missing something here. Node does not multithread requests, but it surely can process many requests simultaneously if these requests are waiti…

> Node does not multithread requests, but it surely can process many requests simultaneously if these requests are waiting for async operations: database, external APIs or other types of I/O usually. That's the very core idea of evented servers. Within a single request, Node can async its dealing with outside services (databases, api's, etc), but it is still only processing one request at a time. There is no 'synchro…

By 'processing' I mean the Node application has received the request and has not yet sent the response, i.e. the connection is still alive. 'synchronized' has no bearing here. If the request processing is purely CPU-bound with no async operations then only one request will be processed at any time, otherwise Node will happily process up to thousands of requests simultaneously. This is the ideal use case for Node. It should be trivial to log the amount of simultaneous requests being processed.

According to Heroku docs, Cedar routers do not do any queuing and just serve requests immediately to any random dyno. They are pretty clear on this in multiple places, specifically talking about concurrent requests in Node. They also mention a 'routing mesh', which suggests there are many routers doing their thing. But that header you see maybe not be relevant to Cedar, just like the other header 'X-Heroku-Queue-Depth' should not apply to Cedar either.

Post reply on HN