Single request per server? What year is this? 2003?
Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails
141–150 of 437 posts
Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails
#142Ugh. These guys are so cocky.
I think that goes with the whole rap deal. They should have challenged them to a dance-off.
Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails
#143Good lord!!!!!
Percentage of the requests served within a certain time (ms)
50% 844
66% 2977
75% 5032
80% 7575
90% 16052
95% 20069
98% 29282
99% 30029
100% 30029 (longest request)
Why in the world would a company spend $20,000 a month for service this awful?
Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails
#144This should be more prominent. I want to love Heroku, and am sure that I could. But really, throwing in the towel at intelligent routing and replacing it with "random routing" is horrific, if true. It's arguable that the routing mesh and scaling dynamics of Heroku are a large part, if not -the- defining reason for someone to choose Heroku over AWS directly. Is it a "hard" problem? I'm absolutely sure it is. That's on…
> That's one reason customers are throwing money at you to solve it, Heroku.
People are throwing money at Heroku because it's really easy to use, not because it's the best long-term technology choice. Seriously - what percentage of Heroku paying users do you think actually read up on the finest technical details like routing algorithms before they put in their credit card? Heroku knows. They know you can't even build a highly-available service on top of it, since it's singly-homed, and they're still making tons of money.
Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails
#145Earlier quoted context omitted.
I seem to recall Google mentioning on some blog several years ago that high variance in response latency degrades user experience much more than slightly higher average request times. I can't find the link though; if anyone has it, I'd be grateful.
Jeff Dean wrote a paper on it for CACM: http://cacm.acm.org/magazines/2013/2/160173-the-tail-at-scal... There's a relatively easy fix for Heroku. They should do random routing with a backup second request sent if the first request times fails to respond after a relatively short period of time (say, 95th percentile latency), killing any outstanding requests when the first response comes back in. The amount of bookkeep…
A relatively easy fix, for read-only or idempotent requests. Also, if long-tail latency requests wind up being run twice, this technique might accelerate tip-over saturation. Still, this 'hedged request' idea is good to keep in mind, thanks for the pointer.
The 'tied request' idea from the Dean paper is neat, too, and Heroku could possibly implement that, and give dyno request-handlers the ability to check, "did I win the race to handle this, or can this request be dropped?"
Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails
#146Earlier quoted context omitted.
I seem to recall Google mentioning on some blog several years ago that high variance in response latency degrades user experience much more than slightly higher average request times. I can't find the link though; if anyone has it, I'd be grateful.
Jeff Dean wrote a paper on it for CACM: http://cacm.acm.org/magazines/2013/2/160173-the-tail-at-scal... There's a relatively easy fix for Heroku. They should do random routing with a backup second request sent if the first request times fails to respond after a relatively short period of time (say, 95th percentile latency), killing any outstanding requests when the first response comes back in. The amount of bookkeep…
[deleted]
Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails
#147randomized routing is not necessarily bad if they look at 2 choices and pick the min. See http://en.wikipedia.org/wiki/2-choice_hashing and http://www.eecs.harvard.edu/~michaelm/postscripts/handbook20...
This is a knee-jerk reply. I know, because my knee jerked as well. Think about the problem a little more: if you have the data necessary to pick the min-of-two, then you have the data you need to do intelligent routing.
Not necessarily. Heroku claims that a global request queue is hard to scale, and therefore they switched to random load balancing. The comment above shows that a global request queue is not necessary. Lets say that minimum-of-n scales up to 10 dynos. If your application requires 40 dynos, you can have one front load balancer which dispatches the requests to 4 back load balancers, each of which has 10 dynos assigned to them on which they perform min-of-10 routing. This gives you routing that's almost as good as min-of-40 but it scales up nonetheless.
Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails
#148Wow. This is explains a lot. We've always been of the opinion that queues were happening on the router, not on the dyno. We consistently see performance problems that, whilst we could tie down to a particular user request (file uploads for example, now moved to S3 direct), we could never figure out why this would result in queuing requests given Heroku's advertised "intelligent routing". We mistakenly thought the occ…
Best thing we can do is follow through on the article's call-to-action for emailing support@heroku.com:
"After reading the following RapGenius article (http://rapgenius.com/James-somers-herokus-ugly-secret-lyrics), we are reevaluating the decision to use Heroku. I understand that using a webserver like unicorn_rails will alleviate the symptoms of the dyno queuing problem, but as a cash-strapped startup, cost-efficiency is of high importance.
I look forward to hearing you address the concerns raised by the article, and hope that the issue can be resolved in a cost-effective manner for your customers."
Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails
#149Given that ElasticBeanstalk has support for rails now, does Heroku still have any advantage over AWS for a new startup?
I found Elastic Beanstalk had extremely serious latency and performance problems for PHP5, that didn't occur when setting up manual EC2/Load balancer. I intended to investigate it, but never had time.
Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails
#150Earlier quoted context omitted.
I seem to recall Google mentioning on some blog several years ago that high variance in response latency degrades user experience much more than slightly higher average request times. I can't find the link though; if anyone has it, I'd be grateful.
Jeff Dean wrote a paper on it for CACM: http://cacm.acm.org/magazines/2013/2/160173-the-tail-at-scal... There's a relatively easy fix for Heroku. They should do random routing with a backup second request sent if the first request times fails to respond after a relatively short period of time (say, 95th percentile latency), killing any outstanding requests when the first response comes back in. The amount of bookkeep…
> There's a relatively easy fix for Heroku. They should do random routing with a backup second request sent if the first request times fails to respond after a relatively short period of time (say, 95th percentile latency), killing any outstanding requests when the first response comes back in. The amount of bookkeeping required for this is a lot less than full-on intelligent routing, but it can reduce tail latency dramatically since it's very unlikely that the second request will hit the same overloaded server.
Your solution doesn't work if requests aren't idempotent.