Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Answering your questions about Heroku routing and web performance

blog.heroku.com

Post reply on HN
Print view View original

51–60 of 168 posts

Re: Answering your questions about Heroku routing and web performance

#51

Post by [deleted] »

res0nat0r wrote: post #50
dangrossman wrote: post #47

Earlier quoted context omitted.

Is promising one service but delivering another not fraud? Promises from the Heroku website pre-Rap Genius posts: "Incoming web traffic is automatically routed to web dynos, with intelligent distribution of load instantly" "Intelligent routing: The routing mesh tracks the availability of each dyno and balances load accordingly. Requests are routed to a dyno only once it becomes available. If a dyno is tied up due to…

Your definition of fraud differs from most: "Wrongful or criminal deception intended to result in financial or personal gain." They weren't intentionally and purposefully misleading people. Not having docs up to date on your website, or you not knowing how the underlying backend works is not fraud. As I've mentioned before if every AWS customer could sue Amazon for not understanding how all of the underlying tech wor…

[deleted]

Re: Answering your questions about Heroku routing and web performance

#52

Post by [deleted] »

res0nat0r wrote: post #50
dangrossman wrote: post #47

Earlier quoted context omitted.

Is promising one service but delivering another not fraud? Promises from the Heroku website pre-Rap Genius posts: "Incoming web traffic is automatically routed to web dynos, with intelligent distribution of load instantly" "Intelligent routing: The routing mesh tracks the availability of each dyno and balances load accordingly. Requests are routed to a dyno only once it becomes available. If a dyno is tied up due to…

Your definition of fraud differs from most: "Wrongful or criminal deception intended to result in financial or personal gain." They weren't intentionally and purposefully misleading people. Not having docs up to date on your website, or you not knowing how the underlying backend works is not fraud. As I've mentioned before if every AWS customer could sue Amazon for not understanding how all of the underlying tech wor…

[deleted]

Re: Answering your questions about Heroku routing and web performance

#53

Post by streptomycin »

res0nat0r wrote: post #34
streptomycin wrote: post #8

He forgot to explain why they won't be refunding customers who were defrauded.

Because they weren't victims of fraud.

Even if that's true, you're just distracting from the main point. If I buy X from a company and they don't provide it, I expect a refund.

Like if I order a laptop from Dell and, oops, it gets lost in the mail, it's not okay to just say "oh, we changed our shipping company, so that should happen less in the future."

Re: Answering your questions about Heroku routing and web performance

#54

Post by adamwiggins »

wmf wrote: post #45
tptacek wrote: post #7

Earlier quoted context omitted.

Is anybody operating at Heroku's scale offering centralized request routing queues? At what price?

Amazon ELB? It does cost significantly more than Heroku AFAIK.

My understanding is that ELBs are HAProxy, and they may be set to use the leastconn algorithm (a global request queue that is friendly to concurrent backends). However, once you get any amount of traffic they start to scale out the nodes in the ELB, which produces essentially the same results as the degradation of the Bamboo router that we've documented.

The difference, of course, is that ELBs are single-tenant. So a big app might only end up with half a dozen nodes, instead of the much larger number in Heroku's router fleet.

Offering some kind of single-tenant router is one possibility we've considered. Partitioning the router fleet, homing... all are ideas we've experimented with and continue to explore. If one of these produces conclusive evidence that it provides a better product for our customers and in keeping with the Heroku approach to product design, obviously we'll do it.

Re: Answering your questions about Heroku routing and web performance

#55

Post by rhizome »

ebbv wrote: post #4

It seems to me that Heroku is still failing to understand (or at least cop to) the fact that the switch from intelligent to randomized routing was a loss of a major reason people chose Heroku in the first place. A lot of Heroku's apparent value came from the intelligent routing feature. Everybody knew that it was harder to implement than random routing, that's why they were willing to pay Heroku for it. Nobody's argu…

I'm sure at least one person's job depends on this working out in the direction they've been trying to go, cf. Blaine Cook (who I think got a raw deal).

Re: Answering your questions about Heroku routing and web performance

#56

Post by ozgune »

badgar wrote: post #26

> 1k req/min Also known as trickle of traffic. Hooray for using bigger numbers and a nonstandard unit to hide inadequacy! Does Heroku use req/min throughout their service? I can't understand why they would, unless they also can't build the infrastructure to measure on a per-second basis. > After extensive research and experimentation, we have yet to find either a theoretical model or a practical implementation that b…

(I was in the bigger engineering team at Amazon that looked into this between between '04-'08.)

After having notable issues with Cisco's hardware load balancers, there was an internal project at Amazon aimed at developing scalable routing solutions.

After years of development effort, it turned out that the "better" solutions didn't work well in production, at least not for our workloads. So we went back to million $ hardware load balancers and random routing.

I don't know if things changed after I left, but I can tell you it wasn't an easy problem. So I completely buy the robustness and simplicity argument these guys are making.

Re: Answering your questions about Heroku routing and web performance

#57

Post by bgentry »

wmf wrote: post #45
tptacek wrote: post #7

Earlier quoted context omitted.

Is anybody operating at Heroku's scale offering centralized request routing queues? At what price?

Amazon ELB? It does cost significantly more than Heroku AFAIK.

ELBs have a least-conn per node routing behavior. If your ELB is present in more than 1 AZ, then you have more than one node. If you have any non-trivial amount of scale, then you probably have well more than 1 node.

Re: Answering your questions about Heroku routing and web performance

#58

Post by natmaster »

adamwiggins wrote: post #30
ebbv wrote: post #24

Earlier quoted context omitted.

I doubt anyone chose Heroku solely because of the routing algorithm, but the sales point of intelligent routing was certainly an appealing one. Developers don't want to pay for abstraction just for abstraction's sake, they want to pay for abstraction of difficult things. Intelligent routing is one of those difficult things. Random routing is easy, which is of course why it's also more reliable, but also why you're se…

Well said. It's interesting — very few customers are actually bent out of shape about this. (A few are, for sure.) It's more non-customers who are watching from the sidelines that seem to be upset. I do want to try to explain ourselves to the community in general, and that's what this post was for. But my first loyalty is to serving our current customers well.

What about potential customers? I've been evaluating your platform and just be completely frustrated with 3 days wasted trying to solve these performance issues.

I'm using gunicorn with python, and if I use the sync worker the request queue easily hits 10 seconds and nothing works; if I switch to gevent or eventlet, new-relic tells me that redis is taking the same time stuck getting a value. This is using the same code in my current provider that works just fine with eventlet and scales well.

To add insult to injury, adding dynos actually degrades performance.

Re: Answering your questions about Heroku routing and web performance

#59

Post by adamwiggins »

jfabre wrote: post #48
adamwiggins wrote: post #32

Earlier quoted context omitted.

Harsh. I'm going to ignore the more inflammatory parts of this (feel free to restate if you want me to engage in discussion), but one bit did grab my attention: "A major selling point of Heroku is that scaling wouldn't be a risk" This is interesting, especially the word "risk." Can you expand on this?

Very small customer here. I don't know much about the abstraction you provide us and I don't want to know as long as things go well. To my point of view, the routing is "random" thus kind of unpredictable. If scaling becomes more of an issue with my business, the last thing I want is to have random scaling issues that I can not do anything about because the load balancer is queuing requests randomly to my dynos. I wa…

Absolutely right, I totally agree. Random scaling issues that you can't either see or control is exactly the opposite of what we want to be providing.

Can you email me (adam at heroku dot com) some links to your past support tickets? I'd like to investigate.

Thanks for running your app with us. Naturally, I expect you to move elsewhere if we can't provide you the service you need. That's the beauty of contract-free SaaS and writing your app with open standards: we have to do a good job serving you, or you'll leave. I wouldn't want it any other way.

Re: Answering your questions about Heroku routing and web performance

#60

Post by aphyr »

ebbv wrote: post #4

It seems to me that Heroku is still failing to understand (or at least cop to) the fact that the switch from intelligent to randomized routing was a loss of a major reason people chose Heroku in the first place. A lot of Heroku's apparent value came from the intelligent routing feature. Everybody knew that it was harder to implement than random routing, that's why they were willing to pay Heroku for it. Nobody's argu…

"the switch from intelligent to randomized routing"

As I understand it, Heroku (on the Bamboo stack) didn't up and decide "Hey, we're gonna switch from intelligent to random routing." The routers are still (individually) intelligent. It's just that there are more of them now, and they were never designed to distribute their internal queue state across the cluster. The system as a whole behaves more and more like a random router as the number of intelligent bamboo routing nodes increases.

Post reply on HN