Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

rapgenius.com

Post reply on HN
Print view View original

211–220 of 437 posts

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#211

Post by nevinera »

mattj wrote: post #12

So the issue here is two-fold: - It's very hard to do 'intelligent routing' at scale. - Random routing plays poorly with request times with a really bad tail (median is 50ms, 99th is 3 seconds) The solution here is to figure out why your 99th is 3 seconds. Once you solve that, randomized routing won't hurt you anymore. You hit this exact same problem in a non-preemptive multi-tasking system (like gevent or golang).

That isn't the only problem with random routing - the problems aren't as pronounced with uniform response speeds, but you still get a significant difference in net effective queue time, especially if you're operating close to your throughput limit.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#212

Post by adanto6840 »

douglasfshearer wrote: post #118
c3 wrote: post #37

Earlier quoted context omitted.

we just switched 1/3rd of our infrastructure off our existing host (engineyard, which uses AWS) onto raw AWS and saved about $2500/month. You can do it too!

Interested in how you achieved this. Did you change server setup significantly from the default EY stack?

I'm curious about this too, especially the last part (similar / identical stack) ?

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#213

Post by DeepDuh »

FireBeyond wrote: post #5

This should be more prominent. I want to love Heroku, and am sure that I could. But really, throwing in the towel at intelligent routing and replacing it with "random routing" is horrific, if true. It's arguable that the routing mesh and scaling dynamics of Heroku are a large part, if not -the- defining reason for someone to choose Heroku over AWS directly. Is it a "hard" problem? I'm absolutely sure it is. That's on…

Why do you 'want to love' Heroku? Because their marketing speak is so great?

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#214

Post by JuDue »

JuDue wrote: post #210
JuDue wrote: post #205

I'd love to see some better tutorials on how to use AWS Beanstalk to scale Rails apps. There is this one, but it doesn't give me a sense of the scalability or management http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create... Any recommendations?

For example, a small instance is $69/yr for 1.7GB of memory, with additional hourly costs that are quite low This is very economical compared to Heroku, and most startups can survive on that initially if they cache properly. But if there is any level of success, how hard is it to scale compared to the extra cost of Heroku? I'm not convinced it's THAT hard, but would love to see more blog posts about Beanstalk. The AW…

And example of what is confusing...

$69 to reserve an instance for a year.

But that is for "light utilization"?!

What does it mean to reserve and instance, but to commit to light usage?

And if you are expecting heavy usage, the price goes up to $195.

But how can you buy an instance for a year but also commit to your usage level? If it's my instance, why is my utilization anyones business?

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#215

Post by nevinera »

kennystone wrote: post #95
nthj wrote: post #71

I'm inclined to wait until Heroku weighs in to render judgement. Specifically, because their argument depends on this premise: > But elsewhere in their current docs, they make the same old statement loud and clear: > The heroku.com stack only supports single threaded requests. Even if your applicaExplaintion were to fork and support handling multiple requests at once, the routing mesh will never serve more than a sin…

If you have 2 unicorn servers and you happen to get 3 slow requests routed to it, you are still screwed, right? Seems to me like it will still queue on that dyno.

That gets pretty unlikely, especially if you have many dynos and a low frequency of slow requests. The main reason unicorn can drastically reduce queue times here is that it does not use random routing internally.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#216

Post by clouddevops »

Perhaps easy deployments are not worth the performance and blackbox trade-off. An alternative approach is a cloud infrastructure provider with baremetal and virtual servers on L2 broadcast domain, and one that provides a good API and orchestration framework so that you can easily automate your deployments. Here are some things we at NephoScale suggest you consider when choosing an infrastructure provider: http://www.slideshare.net/nephoscale/choosing-the-right-infr...

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#217

Post by CoffeeDregs »

lkrubner wrote: post #143

Good lord!!!!! Percentage of the requests served within a certain time (ms) 50% 844 66% 2977 75% 5032 80% 7575 90% 16052 95% 20069 98% 29282 99% 30029 100% 30029 (longest request) Those numbers are amazingly awful. If I ever run ab and see 4 digits I assume I need to optimize my software or server. But 5 digits? Why in the world would a company spend $20,000 a month for service this awful?

Worse than that:

  * 89/100 requests failed (according to
    https://gist.github.com/a-warner/c8cc02565dc214d5f77d ).  
  * Heroku times out requests after 30 seconds, so the 30000ms
    numbers may be timeouts (I've forgotten if *ab* includes 
    those in the summary).
  * That said, the *ab* stats could be biased by using overly 
    large concurrency settings (not probably if you're running 50 dynos...),
    but still...
But still WTF. 89/100 requests failed? That's not happy-making.

Uncertainty is DiaI (death-in-an-infrastructure). I just created a couple of projects on Heroku and love the service, but this needs to be addressed ASAP (even if addressing it is just a blog post).

Also, if you have fewest-connections available, I've never understood using round-robin or random algorithms for load-balancers...

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#218

Post by reddit_clone »

Mc_Big_G wrote: post #119
pdog wrote: post #69

What's the advantage of randomized routing over intelligent routing? Why would this change be made?

It's easier/cheaper for them to maintain and you have to pay for more dynos. A lot more.

Win/Win. For them.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#219

Post by DigitalJack »

DeepDuh wrote: post #213
FireBeyond wrote: post #5

This should be more prominent. I want to love Heroku, and am sure that I could. But really, throwing in the towel at intelligent routing and replacing it with "random routing" is horrific, if true. It's arguable that the routing mesh and scaling dynamics of Heroku are a large part, if not -the- defining reason for someone to choose Heroku over AWS directly. Is it a "hard" problem? I'm absolutely sure it is. That's on…

Why do you 'want to love' Heroku? Because their marketing speak is so great?

I'm sure it's because the idea of heroku is so great.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#220

Post by sgrock »

jaggederest wrote: post #123
toast76 wrote: post #31

Wow. This is explains a lot. We've always been of the opinion that queues were happening on the router, not on the dyno. We consistently see performance problems that, whilst we could tie down to a particular user request (file uploads for example, now moved to S3 direct), we could never figure out why this would result in queuing requests given Heroku's advertised "intelligent routing". We mistakenly thought the occ…

It's interesting, because initially the way that queue time detection worked within New Relic was via timestamps. Currently, though, I believe it's just fed as a number of milliseconds: https://github.com/newrelic/rpm/blame/master/lib/new_relic/a... This solves the issue of the application seeing out-of-whack queue times if there's clock skew between the front-end routing framework and the actual dyno box, but misses…

I work on New Relic's ruby agent, and you're right (hey Justin). In fact we support both methods (i.e. passing a timestamp or a duration). We rely on the front end server (e.g. nginx, apache) to set a timestamp in an HTTP header and forward that to the ruby application. In the case of heroku there is a special header that they pass which describes the queuing duration. Because we're in the ruby application we don't have control over whether this timestamp is accurate but I'm very interested in ideas on how we could do a better job in this situation.

We do provide javascript based browser instrumentation ("Real User Monitoring") which measures request time from the browser's perspective. This might give you a more accurate idea of what real users are experiencing in this case.

Post reply on HN