Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Answering your questions about Heroku routing and web performance

blog.heroku.com

Post reply on HN
Print view View original

111–120 of 168 posts

Re: Answering your questions about Heroku routing and web performance

#111

Post by adamwiggins »

latchkey wrote: post #93

I am working for a fairly large heroku app running Node on ~50-100 web dyno's with another 20-50 backends. Here are the problems as I see it: We get H12's all the time. Randomly. The only suggestion we get from Heroku is to make the requests process as fast as possible. Thus, we've spent considerable amount of time going through everything we can possibly do to make all requests respond as fast as possible. I've give…

We're aware of the random H12s problems. Some apps are affected pretty badly, others not at all, and we're not sure why yet. Sorry that you've had such a bad experience with this. We're continuing to investigate. If we're not able to find a solution in a timely fashion, I'll completely understand if you no longer want to use our product for this app.

Knowing how many dynos you need is definitely a problem. We have implemented autoscaling in the past... but it always sucked. It's hard to find a one-size-fits-all-solution. Rather than ship something sub-par we chose not to ship anything at all.

I understand a lot of people do well with autoscaling libraries and 3rd party add-ons. Would be curious to hear your experience with any of these.

I completely agree that it shouldn't take complaining in public to get a company to listen to its customers. That's was our biggest mistake in all of this, IMO — not listening.

For dyno load, have you tried log-runtime-metrics? https://devcenter.heroku.com/articles/log-runtime-metrics It provides per-dyno load averages.

I gladly accept your compliment that our git deploy remains the best on the market. :)

I'm sorry we haven't been able to serve you better. Let me know if you'd be willing to talk via skype sometime — even if you end up leaving the platform (or already have), I'd like to understand in more depth where we went wrong so we can do better in the future.

Re: Answering your questions about Heroku routing and web performance

#112

Post by senderista »

ozgune wrote: post #56
badgar wrote: post #26

> 1k req/min Also known as trickle of traffic. Hooray for using bigger numbers and a nonstandard unit to hide inadequacy! Does Heroku use req/min throughout their service? I can't understand why they would, unless they also can't build the infrastructure to measure on a per-second basis. > After extensive research and experimentation, we have yet to find either a theoretical model or a practical implementation that b…

(I was in the bigger engineering team at Amazon that looked into this between between '04-'08.) After having notable issues with Cisco's hardware load balancers, there was an internal project at Amazon aimed at developing scalable routing solutions. After years of development effort, it turned out that the "better" solutions didn't work well in production, at least not for our workloads. So we went back to million $…

Nope, DRR is still dead :)

Re: Answering your questions about Heroku routing and web performance

#113

Post by adamwiggins »

cjackson27 wrote: post #107

I can't speak to the issues that people are running into when they reach large scale, but I run a small app with two dynos and we've been having issues with H12 request timeout errors for weeks now. This has been bringing down our production app for periods of about fifteen minutes almost daily. I've been completely disappointed with Heroku's support so far. First they obviously skimmed my support request and provide…

The difficulty of diagnosing H12 errors is really challenging. One thing I can recommend is using the http-request-id labs feature: https://devcenter.heroku.com/articles/http-request-id With this enabled and some code in your app, you can correlate your app's request logs directly against the router logs and trace what happened with any particular H12.

I'd be happy to help you do this if you're game. Contact me via adam at heroku dot com.

Could you also email me some links to your support tickets so I can check out what happened there?

Re: Answering your questions about Heroku routing and web performance

#114

Post by latchkey »

adamwiggins wrote: post #111
latchkey wrote: post #93

I am working for a fairly large heroku app running Node on ~50-100 web dyno's with another 20-50 backends. Here are the problems as I see it: We get H12's all the time. Randomly. The only suggestion we get from Heroku is to make the requests process as fast as possible. Thus, we've spent considerable amount of time going through everything we can possibly do to make all requests respond as fast as possible. I've give…

We're aware of the random H12s problems. Some apps are affected pretty badly, others not at all, and we're not sure why yet. Sorry that you've had such a bad experience with this. We're continuing to investigate. If we're not able to find a solution in a timely fashion, I'll completely understand if you no longer want to use our product for this app. Knowing how many dynos you need is definitely a problem. We have im…

Your response only re-enforces my hard won opinion that Heroku should never be used for a production environment for any business that is trying to be successful and popular. Admitting that you have no idea why critical areas of your infrastructure is causing issues, while at the same time charging people an arm and a leg for services (we pay ~$4k/mo) feels like theft to me. I've built solutions for a large porn company that runs on significantly less infrastructure than what we are running on Heroku and handles 100x more traffic. Something is wrong here with the dyno/router model and maybe it is that you guys are just oversubscribed and not admitting to it in public.

Yes, autoscaling is hard. I have apps on Google AppEngine and see their issues as well. That said, at least they are trying. Maybe even take one of those 3rd party libraries and try to harden and adapt that and make it a real solution? I think the real problem though lies in the fact that there isn't any good metrics for what dynos are doing so there is no metric for when something is too busy or not. Yes, log-runtime-metrics puts out some numbers, but those numbers are meaningless when all I have is a slider to change the amount of money we are paying you.

I should qualify that git deploy compliment because there are issues with that as well. For example, why do you have to rebuild the npm modules from scratch each time? Why not have a directory full of pre-built modules for your dyno's that are just copied into my slug? This relatively simple change would increase the speed of deployments greatly. Never mind that deployments aren't reliable and fail randomly. At least it is easy to just try again.

Re: Answering your questions about Heroku routing and web performance

#115

Post by jules »

tptacek wrote: post #72
jules wrote: post #66

Earlier quoted context omitted.

Random routing vs fully centralized request routing is a false dichtonomy. Suppose you have 100 nodes, and you have a router that routes randomly to one of those 100 nodes. This works very poorly. Now suppose you have 100 nodes, and you have a router that routes intelligently too one of those 100 nodes, e.g. to the one with the smallest request queue. From a theoretical perspective this works really well but it may b…

I understand and don't disagree with everything you are saying, but the focus of my attention is on what you're talking about in your 3rd graf. When you talk about your example problems (1) and (2) with routing to concurrent systems on large number of dynos, what you're really discussing is an engineering flaw in the typical Rails stack. There's a tradeoff between: * a well-engineered request handler (a solved proble…

I'm not sure I agree. Yes, it's a Rails problem that it is using large amounts of memory (on the other hand (2) isn't Rails specific at all, it applies equally to e.g. Node). But it's a Heroku problem that it gives Dynos just 512MB of memory. It's a Heroku problem that it doesn't have a good load balancer. Heroku is in the business of providing painless app hosting, and part of that is painless request routing. These problems may not be completely trivial to solve, but they're not rocket science either. Servers these days can hold hundreds of gigabytes of memory, the 512MB limitation is completely artificial on Heroku's part. Intelligent routing in groups is also very much achievable. Sure, it requires engineering effort, but that's the business Heroku is in.

Of course Heroku is under no obligation to do anything, but its customers have to justify its cost and low performance relative to a dedicated server. And most applications run just fine on a single or at most a couple dedicated servers, which means you don't have routing problems at all, whereas to get reasonable throughput on Heroku you have to get many Dynos, plus a database server. A database server with 64GB ram costs $6400 per month. You can get a dedicated server with that much ram for $100 per month. Heroku is supposed to be worth that premium because it is convenient to deploy on and scale. Because of these routing problems which may require a lot of engineering effort in your application it's not even clear that Heroku is more convenient (e.g. making it use less memory so that you can run many concurrent request handlers on a single Dyno).

Re: Answering your questions about Heroku routing and web performance

#116

Post by [deleted] »

senderista wrote: post #112
ozgune wrote: post #56

Earlier quoted context omitted.

(I was in the bigger engineering team at Amazon that looked into this between between '04-'08.) After having notable issues with Cisco's hardware load balancers, there was an internal project at Amazon aimed at developing scalable routing solutions. After years of development effort, it turned out that the "better" solutions didn't work well in production, at least not for our workloads. So we went back to million $…

Nope, DRR is still dead :)

[deleted]

Re: Answering your questions about Heroku routing and web performance

#117

Post by tptacek »

jules wrote: post #115
tptacek wrote: post #72

Earlier quoted context omitted.

I understand and don't disagree with everything you are saying, but the focus of my attention is on what you're talking about in your 3rd graf. When you talk about your example problems (1) and (2) with routing to concurrent systems on large number of dynos, what you're really discussing is an engineering flaw in the typical Rails stack. There's a tradeoff between: * a well-engineered request handler (a solved proble…

I'm not sure I agree. Yes, it's a Rails problem that it is using large amounts of memory (on the other hand (2) isn't Rails specific at all, it applies equally to e.g. Node). But it's a Heroku problem that it gives Dynos just 512MB of memory. It's a Heroku problem that it doesn't have a good load balancer. Heroku is in the business of providing painless app hosting, and part of that is painless request routing. These…

If there is another provider that seamlessly operates at Heroku's scale (ie, that can handle arbitrarily busy Rails apps) at a reasonable price that has better request dispatching, I think it's very easy to show that you're right.

I'm not sure there are such providers, and if there aren't, I think it's safe to point the finger towards Rails.

As a system for efficiently handling database-backed web requests, Rails is archaic. Not just because of its memory use requirements! It is simultaneously difficult to thread and difficult to run as asynchronous deferrable state machines.

These are problems that Schmidt and the ACE team wrote textbooks about more than 10 years ago.

(Again, Rails has a lot of compensating virtues; I like Rails.)

Re: Answering your questions about Heroku routing and web performance

#118

Post by omfg »

latchkey wrote: post #93

I am working for a fairly large heroku app running Node on ~50-100 web dyno's with another 20-50 backends. Here are the problems as I see it: We get H12's all the time. Randomly. The only suggestion we get from Heroku is to make the requests process as fast as possible. Thus, we've spent considerable amount of time going through everything we can possibly do to make all requests respond as fast as possible. I've give…

Have you checked out Elastic Beanstalk yet? http://aws.amazon.com/elasticbeanstalk/

Any thoughts on that. It offers a Heroku'esque deploy.

Re: Answering your questions about Heroku routing and web performance

#119

Post by latchkey »

omfg wrote: post #118
latchkey wrote: post #93

I am working for a fairly large heroku app running Node on ~50-100 web dyno's with another 20-50 backends. Here are the problems as I see it: We get H12's all the time. Randomly. The only suggestion we get from Heroku is to make the requests process as fast as possible. Thus, we've spent considerable amount of time going through everything we can possibly do to make all requests respond as fast as possible. I've give…

Have you checked out Elastic Beanstalk yet? http://aws.amazon.com/elasticbeanstalk/ Any thoughts on that. It offers a Heroku'esque deploy.

EBS just recently added support for Node. It's definitely on the list as the next PaaS to try out. AppFog is another one that we've done a JVM deployment to and like a lot except for the fact that it just feels very alpha quality. Their website is painfully slow and under documented and the 'af' command isn't nearly as cool as just doing git deploy remote master.

I should also add that one thing that Heroku did get 100% correct is the heroku logs -t command (aka: tailing the logs). Nobody else does that one quite as well.

Re: Answering your questions about Heroku routing and web performance

#120

Post by dblock »

latchkey wrote: post #93

I am working for a fairly large heroku app running Node on ~50-100 web dyno's with another 20-50 backends. Here are the problems as I see it: We get H12's all the time. Randomly. The only suggestion we get from Heroku is to make the requests process as fast as possible. Thus, we've spent considerable amount of time going through everything we can possibly do to make all requests respond as fast as possible. I've give…

We have been experimenting with AWS OpsWorks as an option to move forward. It is excellent. Does git.
Post reply on HN