Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Answering your questions about Heroku routing and web performance

blog.heroku.com

Post reply on HN
Print view View original

101–110 of 168 posts

Re: Answering your questions about Heroku routing and web performance

#101

Post by adamwiggins »

hglaser wrote: post #11
adamwiggins wrote: post #6

Earlier quoted context omitted.

Agreed, I wish we could have done it much sooner. It took a shocking amount of time to sort through all the entangled issues, emotion, and speculation to try to get to the heart of the matter, which ultimately was about web performance and visibility. Also, we wanted to respond to our customers first and foremost, and general community discussion second. So we spent close to a month on skype/hangout/phone with hundre…

Thanks for replying. As a paying Heroku customer (who's not affected by the routing issue), while seeing a blog post earlier would have been nice, it's great to hear that you spent so much time with affected customers.

Glad to hear you're not affected. But we always like talking to customers, feel free to drop me a line at adam at heroku dot com if you'd ever like to spend a few minutes chatting on skype or jabber.

Re: Answering your questions about Heroku routing and web performance

#102

Post by thruflo »

adamwiggins wrote: post #89
nantes wrote: post #75

Earlier quoted context omitted.

I'd be really interested to see some public information resulting from debugging Python apps. We're holding pretty steady, but see a fairly constant stream of timeouts due, apparently, to variance in response times. To be sure, we're working on that. But, in the meantime, our experiments with gevent and Heroku have been less than inspiring.

I've connected Nathaniel (poster above) with one of our Python folks. Looking forward to seeing what they discover. Would you be willing to pair up with one of our developers on your app's performance? If so email me (adam at heroku dot com).

I'm an existing customer using python with gunicorn. I'd be very keen to see any learnings about an optimal setup.

Fwiw, I've found the addon / db connection limits to be the primary blocker when load testing so far.

Re: Answering your questions about Heroku routing and web performance

#103

Post by adamwiggins »

tjb wrote: post #43
adamwiggins wrote: post #13

Earlier quoted context omitted.

Yes, because that is the solution. Empirically. We've run many experiments over the past month to try other approaches to routing, including recreating the exact layout of the Bamboo routing layer (which would never scale to where we are today, but just as a point of reference). None have produced results that are anywhere near as good as using a concurrent backend. (I'd love to publish some of these results so that…

We also adopted Unicorn pretty early on but still suffered issues as dynos simply ran out of memory. In fact with some apps we have seen improvements (but still far from acceptable) in stability by un-adopting this method. The issues raised by scottshea below, as a consequence concern me, what/will be the charge for these as well? To be honest it's the fumbling around in the dark that has annoyed me. I am with you 10…

Thanks for your support. Indeed, communication and transparency into how the product works as far as how it affects your app are two things we'd like to get better at.

Regarding your app: indeed, Unicorn is a huge improvement, but far from the end of the story. "Performance" is like "security" or "uptime" — it's not a one-time feature, something you check off a list and move on. It's something that requires constant work, and every time you fix one problem or bottleneck that just leads you to the next one.

Over time, though, your vigilance pays off with a service that its users deem to be fast or secure or have good uptime. Yet there's no such thing as a finish line on these.

Bringing it back to details. Kazuki from Treasure Data made this Unicorn worker killer that might help you: https://github.com/kzk/unicorn-worker-killer If you're still not happy with your app's performance, give me a shout at adam at heroku dot com and we'll see if we can help.

Re: Answering your questions about Heroku routing and web performance

#104

Post by adamwiggins »

persei8 wrote: post #38

Is it true that you don't fully buffer body of POST request on router? This limitation works against Unicorn design and makes it hardly a "fix": http://rubyforge.org/pipermail/mongrel-unicorn/2013-April/00...

Plenty of room for improvement here, sure. Some technical discussion in this thread: https://blog.heroku.com/archives/2013/4/3/routing_and_web_pe...

Re: Answering your questions about Heroku routing and web performance

#105

Post by adamwiggins »

derengel wrote: post #63

Heroku's message: "If you have slow clients you are screwed" Unicorn is designed to only serve fast clients on low-latency. And no, they don't do any buffering.

I think that overstates it a bit, but yes, there are problems with Unicorn and slow clients. We're investigating: https://blog.heroku.com/archives/2013/4/3/routing_and_web_pe...

If this is an immediate problem for you, it might be worth your while to make your app threadsafe, which gives you more concurrent webserver options.

Re: Answering your questions about Heroku routing and web performance

#106

Post by adamwiggins »

thatthatis wrote: post #69

What data do you have to show that the random selection algo has superior performance to a round-robin algo?

We investigated round-robin. With N routing (or load balancer) nodes, and any degree of request variance, round robin effectively becomes random very quickly.

Re: Answering your questions about Heroku routing and web performance

#107

Post by cjackson27 »

I can't speak to the issues that people are running into when they reach large scale, but I run a small app with two dynos and we've been having issues with H12 request timeout errors for weeks now. This has been bringing down our production app for periods of about fifteen minutes almost daily.

I've been completely disappointed with Heroku's support so far. First they obviously skimmed my support request and provided a canned response that was completely off base. Their next response didn't come for four days and only after I called their sales team to see what I could do to get better support. Their only option is a $1k / mo support contract. If you're running a mission critical app, I'd think twice before choosing Heroku.

Re: Answering your questions about Heroku routing and web performance

#109

Post by adamwiggins »

eaurouge wrote: post #77
adamwiggins wrote: post #30

Earlier quoted context omitted.

Well said. It's interesting — very few customers are actually bent out of shape about this. (A few are, for sure.) It's more non-customers who are watching from the sidelines that seem to be upset. I do want to try to explain ourselves to the community in general, and that's what this post was for. But my first loyalty is to serving our current customers well.

"It's interesting — very few customers are actually bent out of shape about this" Seems to me you don't get it. Sure there are some very vocal non-customers but you also have a lot of potential customers and users (spinning up free instances) evaluating your product and hoping for a better product. I agree that your true value is the abstraction you provide. Some of these potential customers want to ensure Heroku is…

Fair enough. I think the best thing we can do for those potential future customers is be really clear about what the product does and give them good tools and guidance to evaluate whether it fits their needs.

I'd argue that we dropped the ball on that before (on web performance, at least), and are rectifying it now.

Re: Answering your questions about Heroku routing and web performance

#110

Post by adamwiggins »

perucoder wrote: post #83
adamwiggins wrote: post #30

Earlier quoted context omitted.

Well said. It's interesting — very few customers are actually bent out of shape about this. (A few are, for sure.) It's more non-customers who are watching from the sidelines that seem to be upset. I do want to try to explain ourselves to the community in general, and that's what this post was for. But my first loyalty is to serving our current customers well.

If its only a few customers that are bent out of shape, how come you haven't quickly offered them refunds?

We did.
Post reply on HN