Answering your questions about Heroku routing and web performance • Hacker News Board

Re: Answering your questions about Heroku routing and web performance

#41

Post by cynicalkane » Wed, Apr 03, 2013, 6:21 PM UTC

bcgraham wrote: post #33

cynicalkane wrote: post #22

Earlier quoted context omitted.

You sound like a politician talking to someone of the opposite party, in that you say "I hear you", but then completely fail to address anyone's concerns. Selling a "magic black box" that guarantees certain properties, changes them, then lies about having changed them presents a liability for users who want to do serious work. A major selling point of Heroku is that scaling dynos wouldn't be a risk. This guarantee is…

This is really unfair. This comment: >A lot of Heroku's apparent value came from the intelligent routing feature. Everybody knew that it was harder to implement than random routing, that's why they were willing to pay Heroku for it. is being addressed by Adam in this comment: >Heroku's value proposition is that we abstract away infrastructure tasks and let you focus on your app. Keeping you from needing to deal with…

Intelligent routing and random routing have different Big O properties. For someone familiar with routing, or someone who's looked into the algorithmic properties, "intelligent routing" gives one high-level picture of what the performance will be like (good with sufficient capacity), whereas random routing gives a different one (deep queues at load factors where you wouldn't expect deep queues).

This is why it was good marketing for Heroku to advertise intelligent routing, instead of just saying 'oh it's a black box, trust us'. You need to know, at the very least, the asymptotic performance beavhior of the black box.

And that's why the change had consequences. In particular, RapGenius designed their software to fit intelligent routing. For their design the dynos needed to guarantee good near-worst-case performance increased with the square of the load, and my back-of-the-envelope math suggests the average case increases by O(n^1.5).

The original RapGenius post documents them here: http://rapgenius.com/James-somers-herokus-ugly-secret-lyrics

The alleged fix, "switch to a concurrent back-end", is hardly trivial and doesn't solve the underlying problem of maldistribution and underutilization. Maybe intelligent routing doesn't scale but 1) there are efficient non-deterministic algorithms that have the desired properties and 2) it appears the old "doesn't scale" algorithm actually worked better at scale, at least for RapGenius.

Re: Answering your questions about Heroku routing and web performance

#42

Post by res0nat0r » Wed, Apr 03, 2013, 6:22 PM UTC

hglaser wrote: post #9

steveklabnik wrote: post #5

Earlier quoted context omitted.

When admitting fault in response to a customer that spends 5 figures/month with you, wise to have a lawyer go over it first.

I would argue that the brand risk -- e.g. the risk of what ended up happening -- outweighs the legal risk. That's not to say I wouldn't send it to a lawyer. But I'd do it with some hustle. Something like: "This is going out in 24 hours. Please comment ASAP!"

As Adam mentioned in so many words: most of the people pissed off about all of this are most likely right here on HN, and not even customers of Heroku. He mentioned most customers were not that mad about the issue overall.

Re: Answering your questions about Heroku routing and web performance

#43

Post by tjb » Wed, Apr 03, 2013, 6:24 PM UTC

adamwiggins wrote: post #13

scottshea wrote: post #2

So their solution to the random routing is for the customer to switch to Unicorn/Puma on JRuby. Wow.

Yes, because that is the solution. Empirically. We've run many experiments over the past month to try other approaches to routing, including recreating the exact layout of the Bamboo routing layer (which would never scale to where we are today, but just as a point of reference). None have produced results that are anywhere near as good as using a concurrent backend. (I'd love to publish some of these results so that…

We also adopted Unicorn pretty early on but still suffered issues as dynos simply ran out of memory. In fact with some apps we have seen improvements (but still far from acceptable) in stability by un-adopting this method. The issues raised by scottshea below, as a consequence concern me, what/will be the charge for these as well?

To be honest it's the fumbling around in the dark that has annoyed me. I am with you 100% on your manifesto and your points about the type of service you provide. However the time we have spent on this (starting before you came clean about the issues) and the time spent on other increasingly suspicious advice to "up dynos" or spend time "optimising your app" sours this slightly. I accept the "magic black box" comes with its compromises and required understanding at our end but it also means needing to be far more communicative and honest about it at yours. Something which you are putting right I can see.

I for one think the premise of Heroku is a great one and you have succeeded for us in many of the things you have set out to achieve. This whole situation has been a real shame, I'm sure this must have a been a pretty shody time for you guys and I hope you come out the better for it. The quicker the better to be honest so you can focus on the new features we'd like to see.

Re: Answering your questions about Heroku routing and web performance

#44

Post by bradleyjg » Wed, Apr 03, 2013, 6:27 PM UTC

Random routing to concurrent servers works fairly well if the kind of long running requests you need to worry about spend a lot of time blocking for some external service (e.g. a database call). Then you can get a lot of benefit on the from cooperative or preemptive multitasking on the server, and so the performance characteristics, from the point of view of a new request, of each server is roughly the same, and so going to a random server is pretty good.

However, if you have long running requests because they make intensive use of server resources (CPU, RAM) then concurrent servers buys you very little. In that case, sending a new request to a server that is chugging along on a difficult problem is significantly different than sending it to one that isn't. That's where knowing the resource state of each server, and routing accordingly is of huge benefit.

While load balancing is a very difficult problem, with some counterintuitive aspects, it is an area of active research, and there are some very clever algorithms out there.

For example, this article (http://research.microsoft.com/pubs/153348/idleq.pdf) from UIUC and Microsoft introduces the Join-Idle-Queue, which is suitable for distributed load balancers, has considerably lower communication overhead than Shortest Queue (AFAICT the original 'intelligent' routing algorithm), and compares its charateristics to both SQ and random routing.

Re: Answering your questions about Heroku routing and web performance

#45

Post by wmf » Wed, Apr 03, 2013, 6:28 PM UTC

tptacek wrote: post #7

ebbv wrote: post #4

It seems to me that Heroku is still failing to understand (or at least cop to) the fact that the switch from intelligent to randomized routing was a loss of a major reason people chose Heroku in the first place. A lot of Heroku's apparent value came from the intelligent routing feature. Everybody knew that it was harder to implement than random routing, that's why they were willing to pay Heroku for it. Nobody's argu…

Is anybody operating at Heroku's scale offering centralized request routing queues? At what price?

Amazon ELB? It does cost significantly more than Heroku AFAIK.

Re: Answering your questions about Heroku routing and web performance

#46

Post by adamwiggins » Wed, Apr 03, 2013, 6:28 PM UTC

meritt wrote: post #36

badgar wrote: post #26

> 1k req/min Also known as trickle of traffic. Hooray for using bigger numbers and a nonstandard unit to hide inadequacy! Does Heroku use req/min throughout their service? I can't understand why they would, unless they also can't build the infrastructure to measure on a per-second basis. > After extensive research and experimentation, we have yet to find either a theoretical model or a practical implementation that b…

I too was shocked at that. Also that 6 dynos is apparently the average size to handle that load. It takes $179/mo (6 dynos) to handle 17 requests/second? That's insane.

Didn't intend to imply that. Number of dynos needed varies extremely widely, with the app's response time and language/framework being used as the main variables.

There are apps on Heroku that serve 30k–50k reqs/min on 10–20 dynos, typically written in something like Scala/Akka or Node.js and serving incredibly short (~30ms) response times with very little variation. But these are unusual.

The more common case of a website, written in non-threadsafe Rails, with median response times of ~200ms but 95th percentile at 3+ seconds, would probably use those same 10 dynos to do only a few thousand requests per minute. Whether or not you use a CDN and page caching also makes a big difference (see Urban Dictionary for an example that does it well).

But it really depends. We were trying to quantify when you should be worried. If you're running a blog that serves 600 rpm / 10 reqs/sec off of two dynos, you don't need to sweat it.

Re: Answering your questions about Heroku routing and web performance

#47

Post by dangrossman » Wed, Apr 03, 2013, 6:29 PM UTC

res0nat0r wrote: post #34

streptomycin wrote: post #8

He forgot to explain why they won't be refunding customers who were defrauded.

Because they weren't victims of fraud.

Is promising one service but delivering another not fraud?

Promises from the Heroku website pre-Rap Genius posts:

    "Incoming web traffic is automatically routed to web dynos, with intelligent distribution of load instantly"

    "Intelligent routing: The routing mesh tracks the availability of each dyno and balances load accordingly. Requests are routed to a dyno only once it becomes available. If a dyno is tied up due to a long-running request, the request is routed to another dyno instead of piling up on the unavailable dyno’s backlog."

    "Intelligent routing: The routing mesh tracks the location of all dynos running web processes (web dynos) and routes HTTP traffic to them accordingly."

    "the routing mesh will never serve more than a single request to a dyno at a time"

Actual service provided: requests are routed randomly to dynos regardless of how many requests they are currently handling or their current load.

Re: Answering your questions about Heroku routing and web performance

#48

Post by jfabre » Wed, Apr 03, 2013, 6:34 PM UTC

adamwiggins wrote: post #32

cynicalkane wrote: post #22

Earlier quoted context omitted.

You sound like a politician talking to someone of the opposite party, in that you say "I hear you", but then completely fail to address anyone's concerns. Selling a "magic black box" that guarantees certain properties, changes them, then lies about having changed them presents a liability for users who want to do serious work. A major selling point of Heroku is that scaling dynos wouldn't be a risk. This guarantee is…

Harsh. I'm going to ignore the more inflammatory parts of this (feel free to restate if you want me to engage in discussion), but one bit did grab my attention: "A major selling point of Heroku is that scaling wouldn't be a risk" This is interesting, especially the word "risk." Can you expand on this?

Very small customer here. I don't know much about the abstraction you provide us and I don't want to know as long as things go well.

To my point of view, the routing is "random" thus kind of unpredictable. If scaling becomes more of an issue with my business, the last thing I want is to have random scaling issues that I can not do anything about because the load balancer is queuing requests randomly to my dynos.

I want my business to be predictable and if I'm not able to have it I'm going to pack my stuff and move somewhere else.

For now, I'm happy with you except for your damn customer service. They take way too long to answer our questions!

Cheers! :)

Re: Answering your questions about Heroku routing and web performance

#49

Post by streptomycin » Wed, Apr 03, 2013, 6:34 PM UTC

dangrossman wrote: post #47

res0nat0r wrote: post #34

Earlier quoted context omitted.

Because they weren't victims of fraud.

Is promising one service but delivering another not fraud? Promises from the Heroku website pre-Rap Genius posts: "Incoming web traffic is automatically routed to web dynos, with intelligent distribution of load instantly" "Intelligent routing: The routing mesh tracks the availability of each dyno and balances load accordingly. Requests are routed to a dyno only once it becomes available. If a dyno is tied up due to…

Also the misleading performance metrics which hid the fraud.

Re: Answering your questions about Heroku routing and web performance

#50

Post by res0nat0r » Wed, Apr 03, 2013, 6:35 PM UTC

dangrossman wrote: post #47

res0nat0r wrote: post #34

Earlier quoted context omitted.

Because they weren't victims of fraud.

Is promising one service but delivering another not fraud? Promises from the Heroku website pre-Rap Genius posts: "Incoming web traffic is automatically routed to web dynos, with intelligent distribution of load instantly" "Intelligent routing: The routing mesh tracks the availability of each dyno and balances load accordingly. Requests are routed to a dyno only once it becomes available. If a dyno is tied up due to…

Your definition of fraud differs from most: "Wrongful or criminal deception intended to result in financial or personal gain."

They weren't intentionally and purposefully misleading people. Not having docs up to date on your website, or you not knowing how the underlying backend works is not fraud.

As I've mentioned before if every AWS customer could sue Amazon for not understanding how all of the underlying tech worked, or could sue when some of the docs were out of date, there would be more lawyers working there than engineers.