Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

rapgenius.com

Post reply on HN
Print view View original

311–320 of 437 posts

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#311

Post by newhouseb »

fizx wrote: post #307
newhouseb wrote: post #277

Earlier quoted context omitted.

Queuing theory is cool, but I'm not 100% sure it actually applies here in a meaningful sense (although, disclaimer: I'm no more experienced here than you). A lot of queuing theory assumes that you must route a request to a handler immediately as you receive it, and that reassigning a request is a very expensive process. This intuitively explains why queueing theory is very big in router design - imagine that you send…

Yeah, you want leastconn + app/router affinity. Affinity is the statement that all of your requests for an app go through one router (to avoid distributed connection state tracking). In the past, I've accomplished this by having the next layer up consistent hash the api keys onto the router list. If you don't control the top layer (ELB), you need to add a dumb layer just for the hashing. HAProxy works great for this…

Maybe this was after you did your work, but ELB currently supports affinity, see: http://aws.amazon.com/about-aws/whats-new/2010/04/08/support...

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#312

Post by brusch »

goronbjorn wrote: post #50

Aside from the Heroku issue, this is an amazing use of RapGenius for something besides rap lyrics. I didn't have to google anything in the article because of the annotations.

Funny - I thought this was a really interesting article - but I couldn't stand all these annotations. And when I was selecting text (what i do mindlessly when I am reading an article) all the hell broke loose and it tried to load something.

Very annoying when I want to concentrate on the technical details. So we'll see once again - everyone's different.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#313

Post by themgt »

skrebbel wrote: post #299
themgt wrote: post #294

Earlier quoted context omitted.

Because PaaS is a generic technology for running and scaling applications with a multitude of different language/framework/stacks, and many/most of those apps do not share the specific properties of single-threaded Rails (including many Ruby/Rails apps!) And Rails 4 is going to bake-in "live streaming", making single-threaded app servers even more of an edge case.

That's like saying Craigslist did it correctly and AirBnB didn't because AirBnB is only tailoring to a specific segment of the world's supply and demand market. Rails is very widely used. How can you consider that an edge case?

It's like saying EC2 should tailor its virtualization to Fedora 16, or Mac OS X should tailor its windowing system to Photoshop CS4, or Apache should tailor mod_proxy to Joomla. There may be specific attributes of popular applications that need to be adapted to, but those adaptations need to be built in a generic way and exposed through a standard API.

Since even many Rails apps now do not follow a single threaded request-response model, that model of running a web application needs to be considered as one case of many, and building a platform that supports many/all use-cases as well as possible is more complicated than building a platform that fits one use case like a leather glove.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#314

Post by mixedbit »

genwin wrote: post #266
jhspaybar wrote: post #62

Earlier quoted context omitted.

I can chime in here that I have had similar experiences at another large scale place :). Some requests would take a second or more to complete with the vast majority finishing in under 100MS. A solution was put in place that added about 5 MS to the average request, but also crushed the long tail(it just doesn't even exist anymore) and everything is hugely more stable and responsive.

How was 5 ms added? Multiple sleep states per request? I imagine the long tail disappears in a similar way that a traffic jam is prevented by lowering the speed limit.

I think you misunderstood: they optimized the long running requests and the optimization incurred 5ms performance loss for short requests. It is not that the additional 5ms solved the problem.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#315

Post by fizx »

newhouseb wrote: post #311
fizx wrote: post #307

Earlier quoted context omitted.

Yeah, you want leastconn + app/router affinity. Affinity is the statement that all of your requests for an app go through one router (to avoid distributed connection state tracking). In the past, I've accomplished this by having the next layer up consistent hash the api keys onto the router list. If you don't control the top layer (ELB), you need to add a dumb layer just for the hashing. HAProxy works great for this…

Maybe this was after you did your work, but ELB currently supports affinity, see: http://aws.amazon.com/about-aws/whats-new/2010/04/08/support...

You want affinity by Host: header, not by cookie/session.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#316

Post by ajsharp »

Dylan16807 wrote: post #245
ajsharp wrote: post #235

My hunch is that Heroku isn't doing this to bleed customers dry. I know more than a few really, really great people who work there, and I don't think they'd stand for that type of corporate bullshittery. If this were the case, I think we'd have heard about it by now. My best guess is that they hit a scaling problem with doing smart load balancing. Smart load balancing, conceptually, requires persistent TCP connection…

The thing that baffles me is that you could do high-level random balancing onto smaller clusters that do smart balancing. This would solve most of the problem of overloaded servers. An entire cluster would have to clog up with slow requests before there was any performance impact. So why don't they do this?

My off the cuff answer to your question is, because it's probably not quite that simple ;)

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#317

Post by bherms »

c3 wrote: post #37
htsh wrote: post #26

Why not hire a devops guy & rack your own hardware? Or get some massive computing units at amazon (just as good but more expensive)? This reminds me of the excellent 5 stages of hosting story shared on here from a while back: http://blog.pinboard.in/2012/01/the_five_stages_of_hosting/

we just switched 1/3rd of our infrastructure off our existing host (engineyard, which uses AWS) onto raw AWS and saved about $2500/month. You can do it too!

I've been around a few companies migrating from EY/Heroku -> AWS and the cost effectiveness is always astonishing. In addition you gain full control over your architecture, which is a major plus.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#318

Post by humbledrone »

vicks711 wrote: post #287
nsrivast wrote: post #262

OP is a friend of mine, and when I first heard of his problem I wondered if there might be an analytical solution to quantify the difference between intelligent vs naive routing. I took this problem as an opportunity to teach myself a bit of Queueing Theory[1], which is a fascinating topic! I'm still very much a beginner, so bear with me and I'd love to get any feedback or suggestions for further study. For this exam…

I cant stop myself from saying this. You wrote all this instead of doing what?

Normally I would just downvote you and move on, but in this case your comment is frustrating enough that I have to say something. I found the comment you responded to (by nsrivast) quite fascinating. A well-written but brief analysis of the problem, with sources attached for further reading -- what's not to like? In-depth and thoughtful comments like that are what keep me coming back to this site, and are what make the community great.

I for one am very thankful that nsrivast took the time to write something so technical and detailed. However, I found your response to be in extremely poor taste. It added nothing to the conversation, and IMHO was rude and unnecessary.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#319

Post by praptak »

willvarfar wrote: post #310
teich wrote: post #263

This is Oren Teich, I run Heroku. I've read through the OP, and all of the comments here. Our job at Heroku is to make you successful and we want every single customer to feel that Heroku is transparent and responsive. Getting to the bottom of this situation and giving you a clear understanding of what we’re going to do to make it right is our top priority. I am committing to the community to provide more information…

hint just use a rabbitmq queue or something. Don't have a 'smart' LB that has to know everyone's state; instead, have dynos that get more work as quick as they can.

MQ might be a solution but certainly not in the "just use" class. Unless you want to introduce a bottleneck and a single point of failure, this queue has to be distributed.

Managing a distributed queue is hard, for reasons similar to ones making the original problem hard - DQs require global state in a distributed environment. There are tradeoffs involved - the synchronization cost might become a bottleneck in itself.

Pushing the problem on the distributed brokers is making a big bet on the queuing solution. Nope, definitely not in the "just use" category.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#320

Post by dschiptsov »

Why on Earth any sane engineer would think that adding layers of "virtualized" crap in front of your application will be of any benefit?)

The only advantage of virtualization is on the developing stage and it is an ability to add quickly more slow and crappy resources you not own.)

Production is an entirely different realm, and the less layers of crap is in between of your TCP request and DB storage - the better. As for load balancing - it is Cisco level problem.)

Last question: why each web site must be represented as a hierarchy of some objects, instead of thinking in terms of what it is - a list of static files and some cached content generation on demand?)

Post reply on HN