Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

rapgenius.com

Post reply on HN
Print view View original

231–240 of 437 posts

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#231

Post by homosaur »

barmstrong wrote: post #125

We were very surprised to discover Heroku no longer has a global request queue, and spent a good bit of time debugging performance issues to find this was the culprit. Heroku is a great company, and I imagine there was some technical reason they did it (not an evil plot to make more money). But not having a global request queue (or "intelligent routing") definitely makes their platform less useful. Moving to Unicorn…

While I generally agree with your thoughts, I also wonder what's the reason for continuing to misrepresent their service until you dig 20 layers deep in docs.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#232

Post by bbatsell »

CoffeeDregs wrote: post #228
bbatsell wrote: post #224

Earlier quoted context omitted.

> But still WTF. 89/100 requests failed? That's not happy-making. Unless something has changed recently, ab doesn't handle dynamic pages very well. It takes the first pageload as a baseline, and any subsequent request with any portion of the page that is randomized, or is a CSRF token, or reflects most recent changes, etc., is marked as "failed" because it doesn't match the baseline's length. The page in question doe…

AB is not a particularly smart tool (though it's very handy), but I've never seen it reject as not-accurate any non-baseline sized requests... In fact, I can't imagine the rationale that would lead a benchmarking tool to do so. [Obviously, a testing tool might do so...]

I've seen it frequently, and yes, this is how it behaves, e.g.:

[1]: http://stackoverflow.com/a/579466/94830

[2]: https://issues.apache.org/bugzilla/show_bug.cgi?id=42040

I imagine the rationale was something along the lines of many servers/apps are written to incorrectly return 200 with a descriptive error page rather than 500 or whatever the appropriate status code would be. And at the time ab was first written, pages were a lot more static than they are now, so a different page would be more likely to indicate an incorrect response.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#233

Post by Scramblejams »

regularfry wrote: post #77
htsh wrote: post #26

Why not hire a devops guy & rack your own hardware? Or get some massive computing units at amazon (just as good but more expensive)? This reminds me of the excellent 5 stages of hosting story shared on here from a while back: http://blog.pinboard.in/2012/01/the_five_stages_of_hosting/

Because the whole point is that you shouldn't have to.

I don't know, $20k/mo strikes me as an awful lot of money to avoid engaging in work that a scrappy internet startup really ought to be competent at. If you don't know how to run the pipes and they get clogged and your plumber's not picking up the phone, you're screwed.

That amount buys a whole lotta dedicated servers and the talent to run them. (Sidenote: Every time I price AWS or one of its competitors for a reasonably busy site, my eyes seem to pop out at the cost when compared to dedicated hardware and the corresponding sysadmin salary.)

The larger issue is: Invest in your own sysadmin skills, it'll pay off in spades, especially when your back's up against the wall and you figure out that the vendor-which-solves-all-your-problems won't.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#234

Post by DeepDuh »

DigitalJack wrote: post #219
DeepDuh wrote: post #213

Earlier quoted context omitted.

Why do you 'want to love' Heroku? Because their marketing speak is so great?

I'm sure it's because the idea of heroku is so great.

As others here have pointed out in different discussions: Ideas aren't worth much by themselves. It's the subset that's implementable of which it's the subset that has gotten implemented that's worth anything. If we just love ideas we should better become philosophers.

Sorry to be a bit harsh, but I find it a bit shocking how even in this field where we can basically play god and do whatever we want and what we think is best on increasingly powerful bit-cruncher-and-storer-machines, so many here seem to behave like a herd of sheep and just do what 'everyone else' does. Just sit down for a moment and think! What are my requirements right now? What could be a requirement in the near future? What technologies are there which can help me? Am I sure about these feature? Better read up on it first! How difficult is it to get it to behave in ways that are or can be important for me?

Now list that stuff down. If it's puzzling sleep over it, forget it for a few days. Then suddenly, for example under a hot shower you get an idea - that requirement I had isn't really one, I can solve it differently! Come back, take the now fitting piece of the puzzle and do your job in 20% of the time that would have been needed if you would just have blindly followed some path. That's how it usually works for me. Be picky, be exact, but be lazy.

Now about that routing dispatcher problem: Couldn't we solve that in one to two weeks on a generic plattform, but specifically for a certain use case? Let's say you want to have a worker queue of rails request handlers that work in parallel. Just write that damn router! Maybe I'd be lazy, learn Erlang for a week and think about it afterwards.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#235

Post by ajsharp »

My hunch is that Heroku isn't doing this to bleed customers dry. I know more than a few really, really great people who work there, and I don't think they'd stand for that type of corporate bullshittery. If this were the case, I think we'd have heard about it by now.

My best guess is that they hit a scaling problem with doing smart load balancing. Smart load balancing, conceptually, requires persistent TCP connections to backend servers. There's some upper limit per LB instance or machine at which maintaining those connections causes serious performance degredations. Maybe that overhead became too great at a certain point, and the solution was to move to a simpler random round-robin load balancing scheme.

I'd love to hear a Heroku employee weigh in on this.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#236

Post by dangrossman »

JuDue wrote: post #214
JuDue wrote: post #210

Earlier quoted context omitted.

For example, a small instance is $69/yr for 1.7GB of memory, with additional hourly costs that are quite low This is very economical compared to Heroku, and most startups can survive on that initially if they cache properly. But if there is any level of success, how hard is it to scale compared to the extra cost of Heroku? I'm not convinced it's THAT hard, but would love to see more blog posts about Beanstalk. The AW…

And example of what is confusing... $69 to reserve an instance for a year. But that is for "light utilization"?! What does it mean to reserve and instance, but to commit to light usage? And if you are expecting heavy usage, the price goes up to $195. But how can you buy an instance for a year but also commit to your usage level? If it's my instance, why is my utilization anyones business?

You're not committing to a usage level, you're committing to a pricing level. If you reserve a small instance, you get a small instance, no matter what utilization level you choose. It's the same exact resources no matter what you pick.

The utilization levels are pricing tiers:

Light utilization = lowest upfront cost, highest hourly rate.

Medium utilization = medium upfront cost, medium hourly rate.

Heavy utilization = highest upfront cost, lowest hourly rate.

The names are meant to signify the trade-off you're making. If you run your instance only an hour a day, you will pay the least by choosing "light utilization": the hourly cost is high but you're only going to multiply that by a small number, so the savings in the up-front cost will dominate the total cost. If you run your instance 24 hours a day, then the hourly rate will dominate your total costs, so you'll save money by choosing "heavy utilization" with a higher up-front cost but lower hourly cost.

Segmenting the costs makes the pricing table more difficult to read, but it optimizes for everything else: you pay the lowest possible price for guaranteed resources, and Amazon has better knowledge of how much spare capacity it actually needs to handle the reservations.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#237

Post by scottshea »

nthj wrote: post #71

I'm inclined to wait until Heroku weighs in to render judgement. Specifically, because their argument depends on this premise: > But elsewhere in their current docs, they make the same old statement loud and clear: > The heroku.com stack only supports single threaded requests. Even if your applicaExplaintion were to fork and support handling multiple requests at once, the routing mesh will never serve more than a sin…

To amplify, I do a ton of queue adjustment work with Unicorn at the backlog level. It is so frequent that we set up Unicorn startup to read from an ENV variable on Heroku that we set as needed.

With two Unicorn workers we found that 25 was the best backlog threshold to accept (it refuses additional requests). When we were able to go to 5 Unicorn workers on Heroku we had to start to adjust that.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#238

Post by nacho2sweet »

Why is everyone like against rapgenius.com for "forcing the issue with a public spat". They are the customer not getting a service they are paying for. I would be fucking pissed too. Heroku isn't being the darling the service they advertised. They tried to work on it with Heroku. This is useful information to most of you. Are most of you against Yelp?

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#239

Post by loopdoend »

jholman wrote: post #106
46Bit wrote: post #85

Earlier quoted context omitted.

Several Rails apps I develop have been suffering from similar issues. Perhaps 2-3% of requests take 0.4-2s in just processing. If the allocation is a little intelligent, it'll not perform too badly and is less work than much harder optimization. Yet if it's random, it'll queue up horribly . I'm pissed. Spent way too much time unable to explain it to coworkers, thinking I just didn't understand Heroku's platform and t…

> Easiest fix: moving to EC2 next week. I've wanted to ever since these issues became evident but it's hard to make a good argument from handwaving about 'problems'. Of course, then you need to solve all these problems yourself. That sounds pretty easy, you'll have it done next week no problem! That was sarcastic, but this isn't: good luck, let us know how it goes.

It's a lot easier than you think when you aren't limited by artificial restrictions on the number of concurrent requests you can serve. Taking out what could be considered here to be a hostile intermediary will free up tech resources to fix problems that actually exist.

I can only imagine how these guys must have been beating their heads against the wall. Heroku charges a premium price and should be providing a premium service.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#240

Post by jaggederest »

sgrock wrote: post #220
jaggederest wrote: post #123

Earlier quoted context omitted.

It's interesting, because initially the way that queue time detection worked within New Relic was via timestamps. Currently, though, I believe it's just fed as a number of milliseconds: https://github.com/newrelic/rpm/blame/master/lib/new_relic/a... This solves the issue of the application seeing out-of-whack queue times if there's clock skew between the front-end routing framework and the actual dyno box, but misses…

I work on New Relic's ruby agent, and you're right (hey Justin). In fact we support both methods (i.e. passing a timestamp or a duration). We rely on the front end server (e.g. nginx, apache) to set a timestamp in an HTTP header and forward that to the ruby application. In the case of heroku there is a special header that they pass which describes the queuing duration. Because we're in the ruby application we don't h…

My thoughts before I left the project were to add increased granularity of queue times via having headers added at each passing server and show a rainbow chart for the 'depth' of queue at each layer, not sure if that ever got added.

There's facility for that in the Agent, to allow multiple copies of the header and use whichever came first (for the beginning) and whichever came last (for the end ), it'd be relatively easy to hook metrics into each of those.

Post reply on HN