Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

rapgenius.com

Post reply on HN
Print view View original

251–260 of 437 posts

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#251

Post by encoderer »

Scramblejams wrote: post #233
regularfry wrote: post #77

Earlier quoted context omitted.

Because the whole point is that you shouldn't have to.

I don't know, $20k/mo strikes me as an awful lot of money to avoid engaging in work that a scrappy internet startup really ought to be competent at. If you don't know how to run the pipes and they get clogged and your plumber's not picking up the phone, you're screwed. That amount buys a whole lotta dedicated servers and the talent to run them. (Sidenote: Every time I price AWS or one of its competitors for a reasona…

Three thoughts:

1. Employees are expensive. A good ops guy who believes in your cause and wants to work at an early stage startup can be had for $100k. (Maybe NYC is much cheaper than the bay area, but I'll use bay area numbers for now because it's what I know). That's base. Now add benefits, payroll taxes, 401k match, and the cost of his options. So what... $133k?. That's one guy who can then never go on vacation or get hit by the proverbial bus. Now buy/lease your web cluster, database cluster, worker(s), load balancers, dev and staging environments, etc. Spend engineering time building out Cap and Chef/Puppet scripts and myriad other sysops tools. (You'd need some of that on AWS for sure, but less on Heroku which is certainly much much more expensive than AWS)

2. When you price-out these AWS systems are you using the retail rates or are you factoring in the generous discount larger customers are getting from Amazon? You realize large savings first by going for reserved instances and spot pricing and stack on top of that a hefty discount you negotiate with your Amazon account rep.

3. I've worked at 2 successful, talent Bay Area startups in the last few years: One that was built entirely on AWS, and now, currently, one that owns all of their own hardware. Here's what I think: It's a wash. There isn't a huge difference in cost. You should go with whatever your natural talents lead you towards. You have a founding team with solid devops experience? Great, start on the cloud and then transition early to your own hardware. If not, focus on where your value-add is and outsource the ops.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#252

Post by coldtea »

jholman wrote: post #106
46Bit wrote: post #85

Earlier quoted context omitted.

Several Rails apps I develop have been suffering from similar issues. Perhaps 2-3% of requests take 0.4-2s in just processing. If the allocation is a little intelligent, it'll not perform too badly and is less work than much harder optimization. Yet if it's random, it'll queue up horribly . I'm pissed. Spent way too much time unable to explain it to coworkers, thinking I just didn't understand Heroku's platform and t…

> Easiest fix: moving to EC2 next week. I've wanted to ever since these issues became evident but it's hard to make a good argument from handwaving about 'problems'. Of course, then you need to solve all these problems yourself. That sounds pretty easy, you'll have it done next week no problem! That was sarcastic, but this isn't: good luck, let us know how it goes.

>Of course, then you need to solve all these problems yourself. That sounds pretty easy, you'll have it done next week no problem!

Depending on the complexity of their setup, they COULD have it done next week no problem.

After all tens of thousands of other sites have. It's not like everybody except Google and Facebook is using Heroku.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#253

Post by coldtea »

timr wrote: post #201
46Bit wrote: post #117

Earlier quoted context omitted.

> Of course, then you need to solve all these problems yourself. That sounds pretty easy, you'll have it done next week no problem! I agree with this, actually. I know it's not simple to do your own servers when you're growing. Yet I'd rather improve my existing ops skills a bit than have to setup everything as async APIs (on EC2 anyway). That's the only way I can see that I can solve this.

"Yet I'd rather improve my existing ops skills a bit than have to setup everything as async APIs (on EC2 anyway). That's the only way I can see that I can solve this." You're going to discover that a lot of "ops skills" boils down to "do things asynchronously whenever possible". And while nearly any smart engineer can think of the "right" way of doing something, finding the time to do it all is a huge opportunity cos…

>That's what the parent is trying to say. It's not that you can't do it; it's that it's a really bad idea to do it, at first.

This "bad idea" is how 90% of the web works...

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#254

Post by joshwa »

Fundamentally we're talking about a load balancer. Even the most basic load-balancers can use a least-connections algorithm. Even a round-robin algorithm would be better since that would give each dyno (number-of-dynos * msec per request) to finish a long-running request. Random routing is a viable option where the number of concurrent requests a node can handle is large or unknown, but when the limit is known and in the single-digits, random routing is a recipe for disaster.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#255

Post by adrr »

kennystone wrote: post #95
nthj wrote: post #71

I'm inclined to wait until Heroku weighs in to render judgement. Specifically, because their argument depends on this premise: > But elsewhere in their current docs, they make the same old statement loud and clear: > The heroku.com stack only supports single threaded requests. Even if your applicaExplaintion were to fork and support handling multiple requests at once, the routing mesh will never serve more than a sin…

If you have 2 unicorn servers and you happen to get 3 slow requests routed to it, you are still screwed, right? Seems to me like it will still queue on that dyno.

Maybe this is a stupid question, but with unicorn it forks the request and can process multiple requests at the same time. Previously it seems that only one request could be handled by the dyno so requests had to queue on the dynamic routing layer but with multiple request support with unicorn or whatever, wouldn't it be more efficient to dump all the requests to dynos? Followup question, also how would intelligent routing work if it just previously checked to see if which dyno had no requests? That seems like an easy thing to do, now you would have to check CPU/IO whatever and route based on load. Not specifically targeted at you but to everyone reading the thread.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#256

Post by marcamillion »

benihana wrote: post #94

I really like Rap Genius, but I wish they would tone down the blackness of the background. Reading #CCC text on #000 background makes my eyes bug out.

Perspective is a hell of a thing....The way this comment reads, I thought this was going to get racist real quick - but was relieved when I finished reading and did agree with you :)

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#257

Post by 46Bit »

codewright wrote: post #241
jholman wrote: post #106

Earlier quoted context omitted.

> Easiest fix: moving to EC2 next week. I've wanted to ever since these issues became evident but it's hard to make a good argument from handwaving about 'problems'. Of course, then you need to solve all these problems yourself. That sounds pretty easy, you'll have it done next week no problem! That was sarcastic, but this isn't: good luck, let us know how it goes.

I've done dev-ops in the ads industry before, it's really not that hard if you're a competent programmer. You just have to take a more studious approach than most non-dev-ops programmers and read up on things before deploying them. But if you want to let the various PAAS providers put the fear into you, that's your cowardice. Let the others learn as they may. Edit: To clarify, Heroku is making the problem harder on t…

Thanks for this codewright. I've consistently found that learning things others are afraid of is a good business decision - and I reckon this is a big one.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#259

Post by joeya »

toast76 wrote: post #170
Cushman wrote: post #84

Earlier quoted context omitted.

It's not exactly so, if you use a server that spawns child processes: http://michaelvanrooijen.com/articles/2011/06/01-more-concur... you can potentially handle 3-4 requests per dyno at a time. That doesn't fix the root problem, though.

Investigating this approach now. It won't fix the problem, but will certainly reduce the occurrence of blocked dynos. Thx! EDIT: will need to look into our memory perf though, looks like we'll need to do some work to get more than a couple of workers.

I can confirm this. We experimented with Unicorn as a way to get some of the benefits of availability-based routing despite Heroku's random routing. Our medium-sized app (occupying ~230 MB on boot) would quickly exceed Heroku's 512 MB memory limit when forking just 2 unicorn workers, so we had to revert to thin and a greater number of dynos.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#260

Post by pselbert »

spoiler wrote: post #155
nthj wrote: post #71

I'm inclined to wait until Heroku weighs in to render judgement. Specifically, because their argument depends on this premise: > But elsewhere in their current docs, they make the same old statement loud and clear: > The heroku.com stack only supports single threaded requests. Even if your applicaExplaintion were to fork and support handling multiple requests at once, the routing mesh will never serve more than a sin…

Slightly off topic, but what are everyone's. experience and thoughts about Puma[1]? I am using it on a small production environment with Heroku and I like it, but when we officially launch the app, should we switch to Unicorn? [1] http://puma.io/

We are running a smaller app on JRuby using Puma with thread safe on. It has a significantly smaller footprint, as we are only booting one server.

Overall really solid, though more useful if you can use something other than MRI.

Post reply on HN