Question from a non-Ruby-expert: does Thin, which uses Event Machine, help with this at all, or do requests still block on other IO like database calls, etc?
Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails
41–50 of 437 posts
Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails
#42How do they know what algorithm Heroku uses for randomization to stimulate the results?
The differences in simulations are astonishing, I would not think Heroku's engineers were fine with this approach.
'Let's push this random balancing out.. 1000% increase in resources? Oh well, just update documentation!'
Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails
#43So the issue here is two-fold: - It's very hard to do 'intelligent routing' at scale. - Random routing plays poorly with request times with a really bad tail (median is 50ms, 99th is 3 seconds) The solution here is to figure out why your 99th is 3 seconds. Once you solve that, randomized routing won't hurt you anymore. You hit this exact same problem in a non-preemptive multi-tasking system (like gevent or golang).
Wouldn't the bad tails of random routing be an unpredictably long length of time since long running requests times are unpredictable?
Even if you work on narrowing the fat tails, shouldn't you still need to be upfront and clear about how adding a new dyno only gives you an increased chance of better request handling times as you scale?
Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails
#44I am only going to suggest a small edit -> s/Postgres can’t/Heroku's Postgres can't/ PG can scale up pretty well on a single box, but scaling PG on AWS can be problematic due to the disk io issue, so I suspect they just don't do it. I'd love to be corrected :)
Data Dep't here. Postgres scales great on Heroku, AWS, and in general. We've got users doing many thousands of query per second, and terabytes of data. Not a problem.
The issue with the number of connections is that each connection creates a process on the server. We cap the connections at 500, because at that point you start to see problems with O(n^2) data structures in the Postgres internals that start to make all kinds of mischief. This has been improved over the last few releases, but in general, it's still a good idea to try and keep the number of concurrent connections down.
*EDIT: thanks. not a thread. :)
Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails
#45Maybe this is a dumb question, but wouldn't straightforward Round Robin routing by Heroku restore their "one dyno = one more concurrent request" promise without incurring the scaling liabilities of tracking load across an arbitrarily large number of dynos?
Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails
#46Earlier quoted context omitted.
Rap Genius is limited more by time than by money if anything. It would make more sense to throw money at the problem instead of people.
It doesn't appear that running on Heroku is free for them in terms of time.
There's also the outage hell. It's been ok for a month or two, but getting killed whenever AWS has a blip in US-East (there's no cross-region redundancy, and minimal resilience with an AZ or Region-wide service has serious problems) isn't great.
It probably doesn't hurt RG as much as lower overall performance during normal operations does, though.
Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails
#47For those of you looking to migrate to other, barer hosting solutions like AWS or another VPS provider, I've put together a Capistrano add-on that let's you use Heroku-style buildpacks to deploy with Nginx doing front-end proxy. I use it for half a dozen apps on my VPSs and it works swimmingly well.
Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails
#48Personally - I prefer Linode to Heroku, sure there is more of my time consumed with sys admin, but I like having full control over my platform & setup, rather than having it virtually dictated to me. I'm always open to change but this strategy has served me very well for almost 3 years now.
I have a Capistrano config that I can slab in my Rails projects. I can then do a:
cap deploy:setup
cap deploy:cold
cap deploy
And now my app is running on my server, I then add routing and I am good to go.
It is less fancy than Heroku if you want to play with some new technology, you need to install it, and get it configured, and get it to run properly.
Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails
#49Someone from Heroku really needs to weigh in on this.
This is not a new revelation. I got them to admit to it 2 years ago.
http://tiwatson.com/blog/2011-2-17-heroku-no-longer-using-a-...
and specifically:
https://groups.google.com/forum/?fromgroups#!msg/heroku/8eOo...
Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails
#50Aside from the Heroku issue, this is an amazing use of RapGenius for something besides rap lyrics. I didn't have to google anything in the article because of the annotations.