Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

rapgenius.com

Post reply on HN
Print view View original

221–230 of 437 posts

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#221

Post by DannyBee »

mattj wrote: post #12

So the issue here is two-fold: - It's very hard to do 'intelligent routing' at scale. - Random routing plays poorly with request times with a really bad tail (median is 50ms, 99th is 3 seconds) The solution here is to figure out why your 99th is 3 seconds. Once you solve that, randomized routing won't hurt you anymore. You hit this exact same problem in a non-preemptive multi-tasking system (like gevent or golang).

Yes, it is very hard to do it at scale, but so what? I mean, isn't the whole premise of their company to do intelligent things at scale so you don't have to?

It's not an insurmountable problem by any measure, and it's definitely worth it.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#222

Post by kapilkale »

gleb wrote: post #36
zende wrote: post #27

Earlier quoted context omitted.

Rap Genius is limited more by time than by money if anything. It would make more sense to throw money at the problem instead of people.

It doesn't appear that running on Heroku is free for them in terms of time.

It was before.

http://success.heroku.com/rapgenius

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#223

Post by [deleted] »

clouddevops wrote: post #216

Perhaps easy deployments are not worth the performance and blackbox trade-off. An alternative approach is a cloud infrastructure provider with baremetal and virtual servers on L2 broadcast domain, and one that provides a good API and orchestration framework so that you can easily automate your deployments. Here are some things we at NephoScale suggest you consider when choosing an infrastructure provider: http://www.…

[deleted]

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#224

Post by bbatsell »

CoffeeDregs wrote: post #217
lkrubner wrote: post #143

Good lord!!!!! Percentage of the requests served within a certain time (ms) 50% 844 66% 2977 75% 5032 80% 7575 90% 16052 95% 20069 98% 29282 99% 30029 100% 30029 (longest request) Those numbers are amazingly awful. If I ever run ab and see 4 digits I assume I need to optimize my software or server. But 5 digits? Why in the world would a company spend $20,000 a month for service this awful?

Worse than that: * 89/100 requests failed (according to https://gist.github.com/a-warner/c8cc02565dc214d5f77d ). * Heroku times out requests after 30 seconds, so the 30000ms numbers may be timeouts (I've forgotten if *ab* includes those in the summary). * That said, the *ab* stats could be biased by using overly large concurrency settings (not probably if you're running 50 dynos...), but still... But still WTF. 89/10…

> But still WTF. 89/100 requests failed? That's not happy-making.

Unless something has changed recently, ab doesn't handle dynamic pages very well. It takes the first pageload as a baseline, and any subsequent request with any portion of the page that is randomized, or is a CSRF token, or reflects most recent changes, etc., is marked as "failed" because it doesn't match the baseline's length.

The page in question does have a block in the footer reflecting "hot songs", which I'm guessing changed a bit during the run.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#225

Post by robryan »

DeepDuh wrote: post #213
FireBeyond wrote: post #5

This should be more prominent. I want to love Heroku, and am sure that I could. But really, throwing in the towel at intelligent routing and replacing it with "random routing" is horrific, if true. It's arguable that the routing mesh and scaling dynamics of Heroku are a large part, if not -the- defining reason for someone to choose Heroku over AWS directly. Is it a "hard" problem? I'm absolutely sure it is. That's on…

Why do you 'want to love' Heroku? Because their marketing speak is so great?

Because the premise is a good one, abstracting away some of the lower level details so people can just worry about building their apps. It has always been a tough abstraction to pull off though. It is possible the answer is simply that rap genius has outgrown heroku, in the same way as twitter outgrowing rails it doesn't make it a bad tool for everyone.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#226

Post by bhauer »

mattbillenstein wrote: post #141

Single request per server? What year is this? 2003?

I was thinking the same thing. And 200ms for what exactly? To me the elephant in the room is that 99 server instances are spun up to handle 15M uniques per month.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#227

Post by WillieBKevin »

michaelrkn wrote: post #8

We ran into this exact same problem at Impact Dialing. When we hit scale, we optimized the crap out of our app; our New Relic stats looked insanely fast, but Twilio logs told us that we were taking over 15 seconds to respond to many of their callbacks. After spending a few weeks working with Heroku support (and paying for a dedicated support engineer), we moved to raw AWS and our performance problems disappeared. I w…

We moved our Twilio app off Heroku for the same reasons. Extensive optimizations and we would still get timeouts on Twilio callbacks.

The routing dynamics should be explained better in Heroku's documentation. From an engineering perspective, they're a very important piece of information to understand.

We're with https://bluebox.net now and are very happy.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#228

Post by CoffeeDregs »

bbatsell wrote: post #224
CoffeeDregs wrote: post #217

Earlier quoted context omitted.

Worse than that: * 89/100 requests failed (according to https://gist.github.com/a-warner/c8cc02565dc214d5f77d ). * Heroku times out requests after 30 seconds, so the 30000ms numbers may be timeouts (I've forgotten if *ab* includes those in the summary). * That said, the *ab* stats could be biased by using overly large concurrency settings (not probably if you're running 50 dynos...), but still... But still WTF. 89/10…

> But still WTF. 89/100 requests failed? That's not happy-making. Unless something has changed recently, ab doesn't handle dynamic pages very well. It takes the first pageload as a baseline, and any subsequent request with any portion of the page that is randomized, or is a CSRF token, or reflects most recent changes, etc., is marked as "failed" because it doesn't match the baseline's length. The page in question doe…

AB is not a particularly smart tool (though it's very handy), but I've never seen it reject as not-accurate any non-baseline sized requests... In fact, I can't imagine the rationale that would lead a benchmarking tool to do so. [Obviously, a testing tool might do so...]

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#230

Post by hpguy »

Can anyone explain why this random routing is supposedly good for Node.JS and Java? I mean the net effect is busy dynos might serve more requests while idle ones remain idle and that is certainly not good for Node.JS or anything. What am I missing?
Post reply on HN