Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

rapgenius.com

Post reply on HN
Print view View original

71–80 of 437 posts

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#71

Post by nthj »

I'm inclined to wait until Heroku weighs in to render judgement. Specifically, because their argument depends on this premise:

> But elsewhere in their current docs, they make the same old statement loud and clear: > The heroku.com stack only supports single threaded requests. Even if your applicaExplaintion were to fork and support handling multiple requests at once, the routing mesh will never serve more than a single request to a dyno at a time.

They pull this from Heroku's documentation on the Bamboo stack [1], but then extrapolate and say it also applies to Heroku's Cedar stack.

However, I don't believe this to be true. Recently, I wrote a brief tutorial on implementing Google Apps' openID into your Rails app.

The underlying problem with doing so on a free (single-dyno) Heroku app is that while your app makes an authentication request to Google, Google turns around and makes a "oh hey" request to your app. With a single-concurrency system, Google your app times out waiting for Google to get back to you and Google won't get back to you until your app gets back to you so hey deadlock.

However, there is a work-around on the Cedar stack: configure the unicorn server to supply 4 or so worker processes for your web server, and the Heroku routing mesh appropriately routes multiple concurrent requests to Unicorn/my app. This immediately fixed my deadlock problem. I have code and more details in a blog post I wrote recently. [2]

This seems to be confirmed by Heroku's documentation on dynos [3]: > Multi-threaded or event-driven environments like Java, Unicorn, and Node.js can handle many concurrent requests. Load testing these applications is the only realistic way to determine request throughput.

I might be missing something really obvious here, but to summarize: their premise is that Heroku only supports single-threaded requests, which is true on the legacy Bamboo stack but I don't believe to be true on Cedar, which they consider their "canonical" stack and where I have been hosting Rails apps for quite a while.

[1] https://devcenter.heroku.com/articles/http-routing-bamboo

[2] http://www.thirdprestige.com/posts/your-website-and-email-ac...

[3] https://devcenter.heroku.com/articles/dynos#dynos-and-reques...

[edit: formatting]

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#72

Post by zrail »

kawsper wrote: post #58
zrail wrote: post #47

For those of you looking to migrate to other, barer hosting solutions like AWS or another VPS provider, I've put together a Capistrano add-on that let's you use Heroku-style buildpacks to deploy with Nginx doing front-end proxy. I use it for half a dozen apps on my VPSs and it works swimmingly well. https://github.com/peterkeen/capistrano-buildpack

Doesn't that require root or at least sudo permissions for your deploy user?

sudo for the user doing the deploy but not for the user running the app. The default is to run as the deploy user but you can change it.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#74

Post by runarb »

toast76 wrote: post #31

Wow. This is explains a lot. We've always been of the opinion that queues were happening on the router, not on the dyno. We consistently see performance problems that, whilst we could tie down to a particular user request (file uploads for example, now moved to S3 direct), we could never figure out why this would result in queuing requests given Heroku's advertised "intelligent routing". We mistakenly thought the occ…

Is it so that a dyno can only handle a single user request at a time?

Why dos it not use some kind of scheduling system to handle other task while one task is waiting on i/o?

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#75

Post by jasonwatkinspdx »

The problem is the request arrival rate vs the distribution of service times in your app.

New Relic may be giving you an average number you feel happy about, but the 99th percentile numbers are extremely important. If you have a small fraction of requests that take much longer to process, you'll end up with queuing, even with a predictive least loaded balancing policy.

This is a very common performance problem in rails apps, because developers often use active record's associations without any sort of limit on row count, not considering that in the future individual users might have 10000 posts/friends/whatever associated object.

Fix this and you'll see your end user latency come back in line.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#76

Post by kmfrk »

habosa wrote: post #35

Somewhat unrelated: Does anyone else think that RapGenius makes a great blogging platform? I'd love a plugin that enabled similar annotations on any blog, even if they're just by the original author and not crowdsourced.

It's been tried before (Apture and a crowd-sourced proof-reading plug-in I can't remember the name of). It needs critical mass to work, but it might very well on a very community-focused platform.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#77

Post by regularfry »

htsh wrote: post #26

Why not hire a devops guy & rack your own hardware? Or get some massive computing units at amazon (just as good but more expensive)? This reminds me of the excellent 5 stages of hosting story shared on here from a while back: http://blog.pinboard.in/2012/01/the_five_stages_of_hosting/

Because the whole point is that you shouldn't have to.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#78

Post by gleb »

gleb wrote: post #30
bifrost wrote: post #2

I am only going to suggest a small edit -> s/Postgres can’t/Heroku's Postgres can't/ PG can scale up pretty well on a single box, but scaling PG on AWS can be problematic due to the disk io issue, so I suspect they just don't do it. I'd love to be corrected :)

Postgres has a limitation on number of open connections. This is because 1 connection = 1 process. MySQL uses threads, which scales better but has other downsides. The thread-based approach is also possible with Postgres using 3rd party connection pooling apps, e.g. pgbouncer.

BTW, here's a counter-intuitive solution if using pgbouncer is not possible. Simply drop and reestablish connections on every request.

In theory this is horrible, since PG connections are so expensive. In practice the cost of establishing a connection is negligible for a Rails app.

I do suspect this will make performance "fall-off-the cliff" as you get close to capacity.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#79

Post by jarcoal »

nthj wrote: post #71

I'm inclined to wait until Heroku weighs in to render judgement. Specifically, because their argument depends on this premise: > But elsewhere in their current docs, they make the same old statement loud and clear: > The heroku.com stack only supports single threaded requests. Even if your applicaExplaintion were to fork and support handling multiple requests at once, the routing mesh will never serve more than a sin…

This is how I run my apps as well, and they seem to handle more than one request concurrently per dyno, but I'm not smart enough to dispute this post, so I'm just sitting back and watching.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#80

Post by harshreality »

aristus wrote: post #24
mattj wrote: post #12

So the issue here is two-fold: - It's very hard to do 'intelligent routing' at scale. - Random routing plays poorly with request times with a really bad tail (median is 50ms, 99th is 3 seconds) The solution here is to figure out why your 99th is 3 seconds. Once you solve that, randomized routing won't hurt you anymore. You hit this exact same problem in a non-preemptive multi-tasking system (like gevent or golang).

I do perf work at Facebook, and over time I've become more and more convinced that the most crucial metric is the width of the latency histogram. Narrowing your latency band --even if it makes the average case worse -- makes so many systems problems better (top of the list: load balancing) it's not even funny.

I seem to recall Google mentioning on some blog several years ago that high variance in response latency degrades user experience much more than slightly higher average request times. I can't find the link though; if anyone has it, I'd be grateful.
Post reply on HN