Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

rapgenius.com

Post reply on HN
Print view View original

361–370 of 437 posts

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#361

Post by jcromartie »

nsrivast wrote: post #262

OP is a friend of mine, and when I first heard of his problem I wondered if there might be an analytical solution to quantify the difference between intelligent vs naive routing. I took this problem as an opportunity to teach myself a bit of Queueing Theory[1], which is a fascinating topic! I'm still very much a beginner, so bear with me and I'd love to get any feedback or suggestions for further study. For this exam…

Just a little thing: why can't we just say "servers" and "requests" instead of "registers" and "customers", because stores don't get 9000 customers per minute and don't have 500 registers. Everybody here would understand servers and requests.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#362

Post by kmfrk »

sqs wrote: post #135
kmfrk wrote: post #120

Earlier quoted context omitted.

SSL pain can be a major pain to set up. Is the process of setting it up remotely easy compared to Heroku?

Setting up SSL on Elastic Beanstalk was very easy for us. The documentation explained the entire process. It is easier if you get a wildcard SSL cert, so then you can use the same SSL cert for your various deployments under the same domain.

That's very intriguing. Do they support postinstall/post-deploy scripts/hooks like dotCloud to run some framework set-up?

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#363

Post by zen_boy »

jonnycat wrote: post #351

This might be the case "out of the box", but it's very simple to go multithreaded on the Cedar stack and avoid this issue (provided that your app is threadsafe, of course). You can do this pretty easily with a Procfile and thin: bundle exec thin -p $PORT -e $RACK_ENV --threaded start And then config.threadsafe! in the app Regarding Rails app threadsafety, there are some gotchas around class-level configuration and ce…

Are the some resources on the general topic what it means to build multi-threaded Rails app instead of a traditional one?

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#365

Post by [deleted] »

themgt wrote: post #274
pdenya wrote: post #272

Earlier quoted context omitted.

We're discussing Rails on Heroku specifically which, non-unicorn, should be a "next available checkout counter" situation. Ideally it should be possible to make this an optional behavior that you can choose to turn on for Rails apps.

I agree there should be a better way - it's just important to understand than Rails doesn't get any special treatment on a PaaS done correctly, so it's important to come up with a generic solution. I think part of the solution would be customizable option(i.e.. how many requests can each dyno handle simultaneously), probably combined with intelligently monitoring/balancing proxy load so new requests always go to the…

[deleted]

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#366

Post by shabble »

donavanm wrote: post #321
CoffeeDregs wrote: post #217

Earlier quoted context omitted.

Worse than that: * 89/100 requests failed (according to https://gist.github.com/a-warner/c8cc02565dc214d5f77d ). * Heroku times out requests after 30 seconds, so the 30000ms numbers may be timeouts (I've forgotten if *ab* includes those in the summary). * That said, the *ab* stats could be biased by using overly large concurrency settings (not probably if you're running 50 dynos...), but still... But still WTF. 89/10…

> I've never understood using round-robin or random algorithms for load-balancers... LeastConns/FastestConn selection is very dangerous when a backend host fails. Imagine a host has a partial failure, allowing health checks to pass. This host now fast fails and returns a 500 faster than other hosts in the pool generate 200s. This poison host will have less active connections and your LB will route more requests to it…

You'd have some sort of error threshold/rate limit[1] at which point the server is marked dead and falls out of the backends list, surely?

Or even, an alarm threshold if responses are averaging /too fast/, based on your expected load & response times.

I've not done any deployment/ops beyond hte trivial/theoretical though, so I don't know how this would work in reality.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#367

Post by bradleyjg »

themgt wrote: post #294
skrebbel wrote: post #286

Earlier quoted context omitted.

> it's just important to understand than Rails doesn't get any special treatment on a PaaS done correctly Why is it only "done correctly" if it does not account for specific properties of the technology used by a particular customer?

Because PaaS is a generic technology for running and scaling applications with a multitude of different language/framework/stacks, and many/most of those apps do not share the specific properties of single-threaded Rails (including many Ruby/Rails apps!) And Rails 4 is going to bake-in "live streaming", making single-threaded app servers even more of an edge case.

That sounds a lot more like IaaS to me. PaaS should be providing the entire platform, hence the name.

The entire promise of the space is that the customer only has to worry about his own code and perhaps tweaking a few knobs.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#368

Post by shabble »

bambax wrote: post #325
teich wrote: post #263

This is Oren Teich, I run Heroku. I've read through the OP, and all of the comments here. Our job at Heroku is to make you successful and we want every single customer to feel that Heroku is transparent and responsive. Getting to the bottom of this situation and giving you a clear understanding of what we’re going to do to make it right is our top priority. I am committing to the community to provide more information…

What's the point of posting a link to the front page of your blog, where the most recent article is 15 days old (4 hours after the comment above)? What we want to know: - is the OP right or wrong? That is, did you switch from smart to naive routing, for all platforms, and without telling your existing or future customers? - if you did switch from smart to naive routing, what was the rationale behind it? (The OP is li…

>> I am committing to the community to provide more information as soon as possible, including a blog post on http://blog.heroku.com

> What's the point of posting a link to the front page of your blog, where the most recent article is 15 days old (4 hours after the comment above)?

I think OP is saying 'I am going to investigate the situation; when I am finished here [the blog] is where I will post my response', not that there is something there already.

That said, it's all a little too PR-Bot for my taste (although there's probably only so many ways to say the same info without accidentally accepting liability or something).

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#369

Post by kmfrk »

kmfrk wrote: post #76
habosa wrote: post #35

Somewhat unrelated: Does anyone else think that RapGenius makes a great blogging platform? I'd love a plugin that enabled similar annotations on any blog, even if they're just by the original author and not crowdsourced.

It's been tried before (Apture and a crowd-sourced proof-reading plug-in I can't remember the name of). It needs critical mass to work, but it might very well on a very community-focused platform.

gooseGrade! I remembered the name. Here's a link: http://www.crunchbase.com/company/goosegrade.

Re: Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails

#370

Post by vineet »

It seems that the delay is in the variance in the length of the different jobs. Having slow jobs is generally not a good idea, and I can imagine that they are happening for uncommon tasks.

When you are running a 100+ servers it seems like a simple answer would be to think about these uncommon tasks differently. Options would be for prioritizing them differently, showing different UI indicators, and also wanting them happening on a separate set of machines.

Doing these would mean that an intelligent routing mechanism would not have as much use. Am I wrong here?

I do believe that Heroku should document such problems of theirs more clearly, so that we know what challenges that we are facing as we develop applications, but in this particular case, it seems that they do have the right plumbing, and that they just need to be used differently.

Post reply on HN