Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku Blog: Routing Performance Update

blog.heroku.com

Post reply on HN
Print view View original

141–150 of 197 posts

Re: Heroku Blog: Routing Performance Update

#141

Post by malandrew »

tomjen3 wrote: post #75
clicks wrote: post #64

Earlier quoted context omitted.

Heroku was acquired by Salesforce, so I guess it's not really a YC company anymore.

Ah that explains why they have turned evil. Usually once a company has been acquired they are no longer worth using.

Normally I'd say yes, but one of the core Cedar stack engineers is a close friend of mine and I've been over at Heroku a few times to nerd out over beers. Given my conversations with him and other engineers about the designs I can guarantee to you that (1) Heroku is extremely autonomous relative to Salesforce (I bet you most of SF's influence is on the BD side of things and not on the engineering side. In fact I doubt they have much influence on engineering) and (2) there was no malice involved. Like any engineering first organization, they probably spotted an opportunity to innovate on efficiency and optimize a key process. Problem is that in this case v1.0 of their solution turned out to be a dud and they probably didn't have the instrumentation needed for this new approach in place to detect and prevent these issues. They may not have been measuring the all the right things so a modification detrimental to performance slipped through the cracks;

Re: Heroku Blog: Routing Performance Update

#142

Post by spitfire »

gojomo wrote: post #51
jkat wrote: post #7

Depending on what side of Hanlon's razor you fall, the only conclusion I get from this is that they are either incompetent or dishonest. I have a very hard time believing that this issue remained unknown to them for years. As for the post, it's pretty much just documentation. I didn't see any apology. And the only promise of a better tomorrow is a vague "Working to better support concurrent-request Rails apps on Ceda…

They apologized in the last post. Also, self-critical language like "fallen short of [our] promise" and "we failed to..." is a de facto apology and acceptance of responsibility even when the word 'sorry' only appeared earlier. I can understand how this developed. Things worked well for most customers. Many of those with problems got them under control with more dynos or multi-worker setups. Heroku's Rails roots biase…

In short, Heroku needs to hire someone with some operations research experience. This is a mathematical modelling problem, not really a code problem.

Break out Mathematica, Matlab or R and model the damn problem. Then go research the solutions already available (Hint: look at many grocery stores, queuing problems).

Re: Heroku Blog: Routing Performance Update

#143

Post by latchkey »

jharsman wrote: post #91
neebz wrote: post #78

Earlier quoted context omitted.

150 dynos with Node.js and you are still getting H12 errors? The issue so far reported is that if we ever get multiple requests on a single dyno then we'll have a queuing delay because Rails is essentially single-threaded. But with Node.js I think it would be a fairly large amount of requests on a single dyno I suspect before we get any queuing delays

If your requests are CPU intensive, Node.js won't help since it doesn't support preemption. And even if you're primarily IO-limited, a single request that consumes too much CPU will cause queuing.

At this point we've done so much optimization of our app that our requests are not CPU or IO bound (those things have been offloaded to backend processes through a Rabbit message queue) and we still get H12 errors and random slowness. At the time of this writing, in the last 10 minutes, we've had 2 H12 errors. It should be zero.

Re: Heroku Blog: Routing Performance Update

#144

Post by chubot »

trustfundbaby wrote: post #24
chubot wrote: post #20

Well, this certainly calls into question their competence. They're a PaaS company that doesn't understand or measure their load balancing performance. If you are a PaaS company, and you only have 5 metrics you can record, then 99% percentile latency across all apps should be one of them. On another note: why is Rails single-threaded??? That seems unbelievable. So if you have a 2 second database query, your Rails proc…

On another note: why is Rails single-threaded??? That seems unbelievable. So if you have a 2 second database query, your Rails process does nothing else for that 2 seconds? --------------------- Thats not what happens, Rails does not block on IO, it will switch to another thread/process another request while it waits for IO to complete in the first request. http://yehudakatz.com/2010/08/14/threads-in-ruby-enough-alre…

I think you only read the beginning of that article, not the end. What the article seems to imply is that you have to use a web server that spins off Rails in multiple processes. Ruby does concurrent I/O (obviously) but Rails itself does not appear to do it by default, causing the problems discussed here.

"Rails, in fact, does not yet reliably support concurrent request handling. This leaves Rails developers unable to leverage the additional concurrency capabilities offered by the Cedar stack, unless they move to a concurrent web server like Puma or Unicorn."

"Most obviously, early versions of Rails were not threadsafe. As a result, all Rails users were operating with a mutex around the entire request, forcing Rails to behave like the first “Imagined” diagram above. Annoyingly, Mongrel, the most common Ruby web server for a few years, hardcoded this mutex into its Rails handler. As a result, if you spun up Rails in “threadsafe” mode a year ago using Mongrel, you would have gotten exactly zero concurrency."

Re: Heroku Blog: Routing Performance Update

#145

Post by jstanley »

antirez wrote: post #121
ig1 wrote: post #107

Earlier quoted context omitted.

Let say your average http request takes 200ms to serve, then with 5 heroku worker dynos you can serve about 10 million requests a day which should comfortably cover the requirements of most startups. How much do those dynos cost ? - $143/month. That's less than three hours salary for a developer. If your monthly budget is hundreds of dollars I'm guessing your traffic is low enough you can just use the free tiers of h…

10,000,000 requests each day are 115 requests per second. I don't follow your math here with a 200ms CPU time per request.

115/30 is almost 4, which is almost 5; 5*200ms = 1 second;

I guess he meant 10M/mo, not 10M/day.

Re: Heroku Blog: Routing Performance Update

#146

Post by doktrin »

antirez wrote: post #90

Come on startups, you should be technically skilled and able to optimize in order to spend little money . If you sum EC2 and Heroku you are going to pay like 10x what it takes to run the same machines power in a dedicated server, all this because you can't handle the operations? This is absurd IMHO. Also people that want to start a business, there is a huge opportunity here , create software that makes managing Apach…

This feels like a distraction. How about instead of pontificating on how someone else chooses to do business, we discuss the actual merits (or lack thereof) of Heroku's routing issues.

Re: Heroku Blog: Routing Performance Update

#147

Post by bitcartel »

jharsman wrote: post #91
neebz wrote: post #78

Earlier quoted context omitted.

150 dynos with Node.js and you are still getting H12 errors? The issue so far reported is that if we ever get multiple requests on a single dyno then we'll have a queuing delay because Rails is essentially single-threaded. But with Node.js I think it would be a fairly large amount of requests on a single dyno I suspect before we get any queuing delays

If your requests are CPU intensive, Node.js won't help since it doesn't support preemption. And even if you're primarily IO-limited, a single request that consumes too much CPU will cause queuing.

Ah, the issue of CPU intensive code in a single-threaded event loop of a system designed to handle I/O bound applications... let's not go there again!

Re: Heroku Blog: Routing Performance Update

#148

Post by Strshps1MoreTim »

jonny_eh wrote: post #124
nahname wrote: post #117

Earlier quoted context omitted.

>"I can only write my Ruby code but can't handle operations" is not the right attitude, grow up. How is such a condescending post at the top? Everyone running a startup is an idiot because they choose not to waste their time on your priority? Heroku isn't 10 times more and it wouldn't matter even if it was. Talented people are hard to find and spending time on operations when you might not be around in 4 months may n…

The annoying thing about that post is that it derails the conversation. The top post should be analyzing Heroku's response, whether it was sufficient or not. Or how developers can mitigate Heroku's dumb-by-design routing issue.

Right, I'm interested in XYZ, so that is what the top post should be ...

Re: Heroku Blog: Routing Performance Update

#149

Post by wmf »

jlouis wrote: post #119
rst wrote: post #105

Earlier quoted context omitted.

True for Rails 3, though you can enable threading (multiple requests per process). Rails 4 will likely default new apps threaded, but that option would still have to be manually enabled for older ones. (I say likely because there are no formal releases yet, and it's still possible stuff will get pulled back.)

It was news to me that Rails operated in this way. It is a very weak concurrency model (to be precise: it is nonexistent ). I knew that Rails was naive, but I had not dawned on me just how naive it is. It is interesting because then a system like node.js is a giant leap in the right direction - even if there are better languages out there. ( disclaimer: I do Erlang for a living and often operate in highly concurrent…

Rails has a pure shared-nothing scale-out philosophy; it achieves concurrency with multiple processes. The fact that Heroku only runs one process per dyno by default is a shame though.

Re: Heroku Blog: Routing Performance Update

#150

Post by beambot »

bgentry wrote: post #133
beambot wrote: post #17

This is still not ideal. Even if you're running unicorn, you're still susceptible to queueing spikes due to random load balancing. The concurrency just gives you a small buffer and/or some smoothing on 95th percentile responses. Right? At least there's a commitment to update the reporting tools... getting bad data in New Relic was (IMHO) the worst -- even worse than out-of-date docs.

Actually, no. Using unicorn with only 2 workers makes a tremendous difference, not just incremental. RapGenius' own statistical model demonstrates this. Picture each individual dyno in that case as its own "intelligent router". Since it's not distributed and this requires no network coordination, the job of knowing which workers are available becomes trivial. If you're inclined to read up on queuing theory, you'll se…

Fantastic, that's good to hear. I'm able to run ~5 workers on Heroku with gunicorn (Django), so I imagine that means I'm outta the woods for a while at least.

I would love to read up a bit on queuing theory. Any good pointers?

Post reply on HN