Live data from Hacker News

Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

rapgenius.com

201–210 of 437 posts

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#201
post #117

Earlier quoted context omitted.

> Easiest fix: moving to EC2 next week. I've wanted to ever since these issues became evident but it's hard to make a good argument from handwaving about 'problems'. Of course, then you need to solve all these problems yourself. That sounds pretty easy, you'll have it done next week no problem! That was sarcastic, but this isn't: good luck, let us know how it goes.

> Of course, then you need to solve all these problems yourself. That sounds pretty easy, you'll have it done next week no problem! I agree with this, actually. I know it's not simple to do your own servers when you're growing. Yet I'd rather improve my existing ops skills a bit than have to setup everything as async APIs (on EC2 anyway). That's the only way I can see that I can solve this.

"Yet I'd rather improve my existing ops skills a bit than have to setup everything as async APIs (on EC2 anyway). That's the only way I can see that I can solve this."

You're going to discover that a lot of "ops skills" boils down to "do things asynchronously whenever possible". And while nearly any smart engineer can think of the "right" way of doing something, finding the time to do it all is a huge opportunity cost.

That's what the parent is trying to say. It's not that you can't do it; it's that it's a really bad idea to do it, at first.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#202

Those charts of "simulated" load balancing strategies don't look at all reasonable at first glance. You certainly don't see such spiky patterns with normal web loads. I think you'd have to have some crazy amount of std. dev in completion time cranked way, way up in your simulation before you saw a bunch of servers stacked at 30 with others at 1. It's not that there is no benefit to better balancing, it's just that I'…

> If requests are getting distributed randomly shouldn't all your pages show a similar average time in queue?

A common misconception, called "the law of small numbers".

Probability theory tells us this is only true over a large amount of requests, i.e. in the long term (the law of large numebrs). In the short term, results can vary wildly and thus form these kind of queues.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#203
I'd think that most of the requests being served by rapgenius.com would be highly cacheable (99% are likely just people viewing content that rarely changes).

Seems weird that the site would have such a massive load of non-cacheable traffic. Heroku used to offer free and automatic varnish caching, but the cedar stack removed it. Some architectures make it easy to use cloudfront to cache most of the data being served. My guess that refactoring the app to lean on cloudfront would be easier and more cost-effective (and faster) than manually managing custom scaling infrastructure on EC2.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#204
post #12

So the issue here is two-fold: - It's very hard to do 'intelligent routing' at scale. - Random routing plays poorly with request times with a really bad tail (median is 50ms, 99th is 3 seconds) The solution here is to figure out why your 99th is 3 seconds. Once you solve that, randomized routing won't hurt you anymore. You hit this exact same problem in a non-preemptive multi-tasking system (like gevent or golang).

> The solution here is to figure out why your 99th is 3 seconds. Once you solve that, randomized routing won't hurt you anymore. You hit this exact same problem in a non-preemptive multi-tasking system (like gevent or golang).

The Golang runtime uses non-blocking I/O to get around this problem.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#205
I'd love to see some better tutorials on how to use AWS Beanstalk to scale Rails apps.

There is this one, but it doesn't give me a sense of the scalability or management http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create...

Any recommendations?

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#206
I'd love to see some better tutorials on how to use AWS Beanstalk to scale Rails apps.

There is this one, but it doesn't give me a sense of the scalability or management http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create...

Any recommendations for good tutorials?

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#207
Here's a very simple gevent hello world app.

This is run from inside AWS on an m1.large:

https://gist.github.com/dcramer/4950101

For the 50 dyno test, this was the second run, making the assumption that the dynos had to warm up before they could effectively service requests.

You'll see that with 49 more dynos, we only managed to get around 400 more requests/second on an app that isnt even close to real world.

(By no means is this test scientific, but I think it's telling)

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#208
post #24
post #12

So the issue here is two-fold: - It's very hard to do 'intelligent routing' at scale. - Random routing plays poorly with request times with a really bad tail (median is 50ms, 99th is 3 seconds) The solution here is to figure out why your 99th is 3 seconds. Once you solve that, randomized routing won't hurt you anymore. You hit this exact same problem in a non-preemptive multi-tasking system (like gevent or golang).

I do perf work at Facebook, and over time I've become more and more convinced that the most crucial metric is the width of the latency histogram. Narrowing your latency band --even if it makes the average case worse -- makes so many systems problems better (top of the list: load balancing) it's not even funny.

> Narrowing your latency band --even if it makes the average case worse-- makes so many systems problems better (top of the list: load balancing) it's not even funny.

Yeah, it's a lot more practical than implementing QoS, isn't it?

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#209

This kind of validates an idea I've been flirting with: a Heroku-like service which routes requests via AMQP or similar message broker and actually exposes the routing dynamics to the client apps. From a naive, inexperienced view the idea of having web nodes "pull" requests from a central queue rather than the queue taking uneducated guesses seems to be a no-brainer. I can see this making long-running requests (keep-…

That is exactly how an app I develop at $WORK works.

Requests come in on a front-end, it gets passed off to a router that has multiple different workers connected. The workers send a request to the router letting the router know that they are ready to start responding to requests. The router hands the worker a request, marks the request as being worked on, and moves on to the next request. It is a basic Least Recently Used queue at that point, and if all workers are busy but worker number 3 which received work last finished first, he gets handed new work instantly.

The worker then sends the request back to the router, which sends it back to the appropriate front-end that originally responded to the user.

We are using ZeroMQ for our communication.

For our use case we can handle around 200 requests a second from a TCP/IP connected client to our router, to a worker and back to a client. That is with 3 backend workers, which are hitting the disk/database.

It has worked very well for us.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#210
post #205

I'd love to see some better tutorials on how to use AWS Beanstalk to scale Rails apps. There is this one, but it doesn't give me a sense of the scalability or management http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create... Any recommendations?

For example, a small instance is $69/yr for 1.7GB of memory, with additional hourly costs that are quite low

This is very economical compared to Heroku, and most startups can survive on that initially if they cache properly.

But if there is any level of success, how hard is it to scale compared to the extra cost of Heroku?

I'm not convinced it's THAT hard, but would love to see more blog posts about Beanstalk. The AWS doco feels quite mechanical.

Post reply on HN