Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

rapgenius.com

Post reply on HN
Print view View original

321–330 of 437 posts

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#321

Post by donavanm »

CoffeeDregs wrote: post #217
lkrubner wrote: post #143

Good lord!!!!! Percentage of the requests served within a certain time (ms) 50% 844 66% 2977 75% 5032 80% 7575 90% 16052 95% 20069 98% 29282 99% 30029 100% 30029 (longest request) Those numbers are amazingly awful. If I ever run ab and see 4 digits I assume I need to optimize my software or server. But 5 digits? Why in the world would a company spend $20,000 a month for service this awful?

Worse than that: * 89/100 requests failed (according to https://gist.github.com/a-warner/c8cc02565dc214d5f77d ). * Heroku times out requests after 30 seconds, so the 30000ms numbers may be timeouts (I've forgotten if *ab* includes those in the summary). * That said, the *ab* stats could be biased by using overly large concurrency settings (not probably if you're running 50 dynos...), but still... But still WTF. 89/10…

> I've never understood using round-robin or random algorithms for load-balancers...

LeastConns/FastestConn selection is very dangerous when a backend host fails. Imagine a host has a partial failure, allowing health checks to pass. This host now fast fails and returns a 500 faster than other hosts in the pool generate 200s. This poison host will have less active connections and your LB will route more requests to it. A single host failure just became a major outage.

I like WRR selection for back ends, then use a queue or fast fail when your max active conns is exceeded. Humans prefer latency to errors, so let the lb queue (to a limit) on human centric VIPs. Automated clients deal better with errors so have your lb throw a 500 directly, or RST, or direct to a pool that serves static content.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#322

Post by doktrin »

themgt wrote: post #313
skrebbel wrote: post #299

Earlier quoted context omitted.

That's like saying Craigslist did it correctly and AirBnB didn't because AirBnB is only tailoring to a specific segment of the world's supply and demand market. Rails is very widely used. How can you consider that an edge case?

It's like saying EC2 should tailor its virtualization to Fedora 16, or Mac OS X should tailor its windowing system to Photoshop CS4, or Apache should tailor mod_proxy to Joomla. There may be specific attributes of popular applications that need to be adapted to, but those adaptations need to be built in a generic way and exposed through a standard API. Since even many Rails apps now do not follow a single threaded re…

> Mac OS X should tailor its windowing system to Photoshop CS4

I think statements like these obscure away the very tight coupling Heroku has historically had with Rails. While certainly Heroku now perhaps envisions itself as a do-it-all PaaS, there's no denying Rails at one point (and, numerically, perhaps still) was their bread and butter.

While I don't have numbers to support or refute the assertion that "most Rails apps are primarily single threaded", my suspicion is that this is in fact still the case.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#323

Post by gizzlon »

spoiler wrote: post #155
nthj wrote: post #71

I'm inclined to wait until Heroku weighs in to render judgement. Specifically, because their argument depends on this premise: > But elsewhere in their current docs, they make the same old statement loud and clear: > The heroku.com stack only supports single threaded requests. Even if your applicaExplaintion were to fork and support handling multiple requests at once, the routing mesh will never serve more than a sin…

Slightly off topic, but what are everyone's. experience and thoughts about Puma[1]? I am using it on a small production environment with Heroku and I like it, but when we officially launch the app, should we switch to Unicorn? [1] http://puma.io/

I don't have a stake in the Ruby webserver wars, but the unicorn site has a very good discussion about how it works internally, why it's build as it is, pros & cons etc..

This seems to be missing from most of these project sites, which are often just marketing (look! It's better!!), and therefore not very trustworthy.

From the outside it looks like the biggest differentiator in each generation of ruby servers (and, I guess, db managment systems :) is not that the new is better or worse, but simply that has different trade-offs.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#324

Post by donavanm »

nostrademons wrote: post #291
badgar wrote: post #150

Earlier quoted context omitted.

> There's a relatively easy fix for Heroku. They should do random routing with a backup second request sent if the first request times fails to respond after a relatively short period of time (say, 95th percentile latency), killing any outstanding requests when the first response comes back in. The amount of bookkeeping required for this is a lot less than full-on intelligent routing, but it can reduce tail latency d…

Yeah, I know. I figure that for incoming HTTP traffic it's relatively easy to balance the GET requests, and if they're doing anything remotely sane with HTTP those ought to be idempotent (if they're not, Googlebot will come along and delete their site ;-)). For mutating requests, there's a solution as well, but it involves checksumming the request and passing the checksum along so that the database layer knows to dis…

Or use a parameter that's effectively a nonce.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#325

Post by bambax »

teich wrote: post #263

This is Oren Teich, I run Heroku. I've read through the OP, and all of the comments here. Our job at Heroku is to make you successful and we want every single customer to feel that Heroku is transparent and responsive. Getting to the bottom of this situation and giving you a clear understanding of what we’re going to do to make it right is our top priority. I am committing to the community to provide more information…

What's the point of posting a link to the front page of your blog, where the most recent article is 15 days old (4 hours after the comment above)?

What we want to know:

- is the OP right or wrong? That is, did you switch from smart to naive routing, for all platforms, and without telling your existing or future customers?

- if you did switch from smart to naive routing, what was the rationale behind it? (The OP is light on this point; there must be a good reason to do this, but he doesn't really say what it is or might be)

- if the OP is wrong, where do his problems might come from?

- etc.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#326

Post by dangrahn »

I was in contact with Heroku support a couple of weeks ago since we experienced some timeout on our production app. Got a detailed explanation how the routing on heroku works by a Heroku engineer, and thought I could share:

"I am a bit confused by what you mean by an "available" dyno. Requests get queued at the application level, rather than at the router level. Basically, as soon as a request comes in, it gets fired off randomly to any one of your web dynos.

Say your request that takes 2 seconds to be handled by the dyno was dispatched to a dyno that was running a long running request. Eventually, after 29 seconds, it completed serving the response, and started working on the new, faster 2 second request. Now, at this point it had already been waiting in the queue for 29 seconds, so after 1 second, it'll get dropped, and after another 1 second, the dyno will be done processing it, but the router is no longer waiting for the response as it has already returned an H12.

That's how a fast request can be dropped. Now, the one long 29 second request could also be a series of not-that-long-but-still-long requests. Say you had 8 requests dispatched to that dyno at the same time, and they all took 4 seconds to process. The last one would have been waiting for 28 second, and so would be dropped before completion and result in an H12."

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#327

Post by dspillett »

runarb wrote: post #74
toast76 wrote: post #31

Wow. This is explains a lot. We've always been of the opinion that queues were happening on the router, not on the dyno. We consistently see performance problems that, whilst we could tie down to a particular user request (file uploads for example, now moved to S3 direct), we could never figure out why this would result in queuing requests given Heroku's advertised "intelligent routing". We mistakenly thought the occ…

Is it so that a dyno can only handle a single user request at a time? Why dos it not use some kind of scheduling system to handle other task while one task is waiting on i/o?

That makes more efficient use of the nodes, because otherwise idle cpu time and memory allocation within a node has a chance of getting used, but it doesn't stop the queueing problem (it mearly gives you more virtual nodes to hand tasks to unintelligently).

Also as the nodes are virtual machines anyway and may be contending with each other for IO, and for most apps these days you spend more time waiting for IO than you do spinning the CPU (unless you have a lot of static content so don't need to hit the db for many requests - but such requests are better handled by a caching layer above that which handles the fancier stuff), so the benefit of running multiple processes per node is going to be a lot less noticable than if you are talking about the nodes being physical machines with dedicated storage channels.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#328

Post by praptak »

bambax wrote: post #325
teich wrote: post #263

This is Oren Teich, I run Heroku. I've read through the OP, and all of the comments here. Our job at Heroku is to make you successful and we want every single customer to feel that Heroku is transparent and responsive. Getting to the bottom of this situation and giving you a clear understanding of what we’re going to do to make it right is our top priority. I am committing to the community to provide more information…

What's the point of posting a link to the front page of your blog, where the most recent article is 15 days old (4 hours after the comment above)? What we want to know: - is the OP right or wrong? That is, did you switch from smart to naive routing, for all platforms, and without telling your existing or future customers? - if you did switch from smart to naive routing, what was the rationale behind it? (The OP is li…

> if you did switch from smart to naive routing, what was the rationale behind it?

Most probable cause: smart routing is hard to scale. Multiple routers, with each one doing random distribution independently of others will still produce a globally random distribution. No need for inter-router synchronization.

If multiple routers try smart routing, they must do quite a bit of state sharing to avoid situations where N routers try to schedule their tasks on a single dyno. And even if you split dynos between routers then you need to move requests between routers in order to balance them.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#329

Post by willvarfar »

praptak wrote: post #319
willvarfar wrote: post #310

Earlier quoted context omitted.

hint just use a rabbitmq queue or something. Don't have a 'smart' LB that has to know everyone's state; instead, have dynos that get more work as quick as they can.

MQ might be a solution but certainly not in the "just use" class. Unless you want to introduce a bottleneck and a single point of failure, this queue has to be distributed. Managing a distributed queue is hard, for reasons similar to ones making the original problem hard - DQs require global state in a distributed environment. There are tradeoffs involved - the synchronization cost might become a bottleneck in itself…

yes, I know all the ins and outs.

But they will end up building a pull rather than push system in the end.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#330

Post by dspillett »

anon640 wrote: post #52

"For a Rails app, each dyno is capable of serving one request at a time." Is this a deliberate design choice on Heroku's part, or is this just how Ruby and Rails work? It sounds bizarre that you would need multiple virtual OS instances just to serve multiple requests at the same time. What are the advantages of this over standard server fork()/threaded accept designs?

I assume it is intended to keep the nods as simple as possible so easy to schedule.

A node dies? Just kill it and you lose at most the one active request.

Which is the least busy node to send th enext request to? That can be a lot harder to judge reliably than simply "any nodes doing nothing? I'll queue this request then".

Post reply on HN