Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

rapgenius.com

Post reply on HN
Print view View original

301–310 of 437 posts

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#301

Post by stock_toaster »

CoffeeDregs wrote: post #228
bbatsell wrote: post #224

Earlier quoted context omitted.

> But still WTF. 89/100 requests failed? That's not happy-making. Unless something has changed recently, ab doesn't handle dynamic pages very well. It takes the first pageload as a baseline, and any subsequent request with any portion of the page that is randomized, or is a CSRF token, or reflects most recent changes, etc., is marked as "failed" because it doesn't match the baseline's length. The page in question doe…

AB is not a particularly smart tool (though it's very handy), but I've never seen it reject as not-accurate any non-baseline sized requests... In fact, I can't imagine the rationale that would lead a benchmarking tool to do so. [Obviously, a testing tool might do so...]

use wrk[1] or weighttp[2] instead.

[1]: https://github.com/wg/wrk

[2]: redmine.lighttpd.net/projects/weighttp/wiki

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#302

Post by jaytaylor »

bad_user wrote: post #283

I noticed problems with Heroku's router too. However, contrary to the author, I'm serving 25,000 real requests per second with only 8 dynos. The app is written in Scala and runs on top of the JVM. And I was dissatisfied that 8 dynos seem like too much for an app that can serve over 10K requests per sec on my localhost.

This sounds interesting, but kind of suspicious. What app is serving that kind of volume continuously other than fb or goog?

You running zynga on Heroku or something?

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#303

Post by Vitaly »

c3 wrote: post #37
htsh wrote: post #26

Why not hire a devops guy & rack your own hardware? Or get some massive computing units at amazon (just as good but more expensive)? This reminds me of the excellent 5 stages of hosting story shared on here from a while back: http://blog.pinboard.in/2012/01/the_five_stages_of_hosting/

we just switched 1/3rd of our infrastructure off our existing host (engineyard, which uses AWS) onto raw AWS and saved about $2500/month. You can do it too!

And how much more engineering time do you waste on it now?

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#304

Post by csense »

character0 wrote: post #271
teich wrote: post #263

This is Oren Teich, I run Heroku. I've read through the OP, and all of the comments here. Our job at Heroku is to make you successful and we want every single customer to feel that Heroku is transparent and responsive. Getting to the bottom of this situation and giving you a clear understanding of what we’re going to do to make it right is our top priority. I am committing to the community to provide more information…

While I think it is appropriate for Heroku to respond to this thread (and other important social media outlets covering this), linking to a blog without any messaging concerning your efforts might not be the greatest move... This may not be a sink or swim moment for Heroku, but tight management of your PR is key to mitigating damage. Best of luck, Heroku is a helpful product and I want to see you guys bounce back fro…

Telling people where to look for a reply when they have one is a great idea, IMHO.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#305

Post by aneth4 »

badgar wrote: post #144
FireBeyond wrote: post #5

This should be more prominent. I want to love Heroku, and am sure that I could. But really, throwing in the towel at intelligent routing and replacing it with "random routing" is horrific, if true. It's arguable that the routing mesh and scaling dynamics of Heroku are a large part, if not -the- defining reason for someone to choose Heroku over AWS directly. Is it a "hard" problem? I'm absolutely sure it is. That's on…

> That's one reason customers are throwing money at you to solve it, Heroku. People are throwing money at Heroku because it's really easy to use, not because it's the best long-term technology choice. Seriously - what percentage of Heroku paying users do you think actually read up on the finest technical details like routing algorithms before they put in their credit card? Heroku knows. They know you can't even build…

So heroku doesn't want these $20,000/mo accounts, just technically understaffed startups paying $1000/mo?

I think heroku does want to be a long-term technology choice.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#306

Post by doktrin »

teich wrote: post #263

This is Oren Teich, I run Heroku. I've read through the OP, and all of the comments here. Our job at Heroku is to make you successful and we want every single customer to feel that Heroku is transparent and responsive. Getting to the bottom of this situation and giving you a clear understanding of what we’re going to do to make it right is our top priority. I am committing to the community to provide more information…

Thanks for the response, but I have to admit that the lack of a clear-cut answer here is a little worrisome.

Anyone who wants to like Heroku would hope that the OP is flat out, 100%, wrong. The fact that Heroku's official answer requires a bit of managing implies otherwise.

On a related tangent, I would also encourage future public statements to be a little less opaque than some Heroku has put out previously.

For instance, the cause of the outage last year was attributed to "...the streaming API which connects the dyno manifold to the routing mesh" [1]. While that statement is technically decipherable, it's far from clear.

[1] https://status.heroku.com/incidents/372

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#307

Post by fizx »

newhouseb wrote: post #277
nsrivast wrote: post #262

OP is a friend of mine, and when I first heard of his problem I wondered if there might be an analytical solution to quantify the difference between intelligent vs naive routing. I took this problem as an opportunity to teach myself a bit of Queueing Theory[1], which is a fascinating topic! I'm still very much a beginner, so bear with me and I'd love to get any feedback or suggestions for further study. For this exam…

Queuing theory is cool, but I'm not 100% sure it actually applies here in a meaningful sense (although, disclaimer: I'm no more experienced here than you). A lot of queuing theory assumes that you must route a request to a handler immediately as you receive it, and that reassigning a request is a very expensive process. This intuitively explains why queueing theory is very big in router design - imagine that you send…

Yeah, you want leastconn + app/router affinity. Affinity is the statement that all of your requests for an app go through one router (to avoid distributed connection state tracking).

In the past, I've accomplished this by having the next layer up consistent hash the api keys onto the router list. If you don't control the top layer (ELB), you need to add a dumb layer just for the hashing.

HAProxy works great for this extra layer. In practice, all you end up doing is adding a "balance hdr(host)" directive (see http://haproxy.1wt.eu/download/1.5/doc/configuration.txt) to get the hashing right, and you're spending <1ms inside HAProxy.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#308

Post by bad_user »

jaytaylor wrote: post #302
bad_user wrote: post #283

I noticed problems with Heroku's router too. However, contrary to the author, I'm serving 25,000 real requests per second with only 8 dynos. The app is written in Scala and runs on top of the JVM. And I was dissatisfied that 8 dynos seem like too much for an app that can serve over 10K requests per sec on my localhost.

This sounds interesting, but kind of suspicious. What app is serving that kind of volume continuously other than fb or goog? You running zynga on Heroku or something?

It's an integration with an OpenRTB bidding marketplace that's sending our way that traffic.

And 25K is not the whole story. In a lot of ways it's similar to high frequency trading. Not only do you need to decide in real time if you want to respond with a bid or not, but the total response time should be under 200ms, preferably under 100ms, otherwise they start bitching about latency and they could drop you off from the exchange.

And the funny thing is 25K is actually nothing, compared to the bigger marketplace that we are pursuing and that will probably send us 800K per second at peak.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#310

Post by willvarfar »

teich wrote: post #263

This is Oren Teich, I run Heroku. I've read through the OP, and all of the comments here. Our job at Heroku is to make you successful and we want every single customer to feel that Heroku is transparent and responsive. Getting to the bottom of this situation and giving you a clear understanding of what we’re going to do to make it right is our top priority. I am committing to the community to provide more information…

hint just use a rabbitmq queue or something. Don't have a 'smart' LB that has to know everyone's state; instead, have dynos that get more work as quick as they can.
Post reply on HN