Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Queues don't fix overload (2014)

ferd.ca

Post reply on HN
Print view View original

81–90 of 156 posts

Re: Queues don't fix overload (2014)

#81

Post by politician »

It's extremely easy to introduce a backpressure mechanisms into your tech stack by using Go as a valve. If you can arrange data flow through a Go process, even if it's just a small tool inside your PHP/Rust/Javascript dream stack, then you can get backpressure done in about 5 lines of code:

   func valve[T any](ch chan
The other half of the valve is a goroutine that reads from `ch` and pushes values further down your pipeline. The channel can be buffered or unbuffered, it doesn't matter. Flow through this coupling will only proceed at the rate supported by the goroutine, and you can use this fact to partition your stack into rated flow domains.

Sometimes it's better to deploy a simple tool into existing stacks to help patch overload conditions until a "native" fix can be developed.

Re: Queues don't fix overload (2014)

#82

Post by macintux »

bhaak wrote: post #42

What's the snarky bit at the end of article about? > And then of course, there's the use case where you use the queue as a messaging mechanism between front-end threads/processes [...] because your language doesn't support inter-process communications. Isn't it the other way around? A message broker is used as a simple queue but then you immediately have the option to scale up to n producers and m consumers if demand…

I imagine he's being snarky about the fact that Erlang was designed around simple queues and thus you get that abstraction essentially for free, while in other languages you have to add more infrastructure once you recognize the need.

Re: Queues don't fix overload (2014)

#83

Post by cdowns »

I recently ran into an overload issue and it turned out to be a not-obvious "hard limit" that was mentioned. Everything would be smooth for a bit and then my throughput would be halved after I walked away, backing up the queues indefinitely and paging me again.

I had moved a single-broker RabbitMQ from GCP to AWS and the instance type I chose had bandwidth "up to" 10Gbps. Being less familiar with AWS, I did not realize they will actively throttle based on credits because "up to" means "burstable to" regardless of available capacity. My messages are pretty large and I was running out of credits after about an hour.

Bandwidth was the last thing I considered since I hadn't had the issue on GCP. Switching to a larger instance with guaranteed bandwidth was a band-aid. Clustering to spread the load between multiple instances will be my longer term fix. Lesson learned, hopefully this helps someone someday.

Re: Queues don't fix overload (2014)

#84

Post by pjdesno »

PaulHoule wrote: post #40
mhandley wrote: post #18

What queues do is smooth out the mismatch between supply and demand. If the mismatch lasts long enough, the queue will overflow, and then you need to load shed (and you need to plan for what the least bad way of load shedding is). But queues do increase overall throughput, up to a point. If the demand was bursty on short timescales and you only allow a small queue to build before load-shedding, you may be wasting cap…

See https://en.wikipedia.org/wiki/Bufferbloat I lived in Germany in 1999 and then the internet connection from Germany to the US would get overloaded during the day. At maybe 9am the latency would be then it would start dropping packets. I don't know if it was the intention but it was about as good as a ban on VoIP at preventing people from making international VoIP calls. Today there is more consciousness about the…

See RFC 970, "On Packet Switches With Infinite Storage" by John Nagle: https://datatracker.ietf.org/doc/html/rfc970

Back then (I was studying networking as an undergrad at the time, and interned with the Arpanet team) people really did think of network congestion as a buffer allocation problem, so the obvious solution was more buffering - i.e. adding queues. Nagel was one of the first people to point out the problem with this.

Re: Queues don't fix overload (2014)

#85

Post by taneq »

Ultimately if you over load, you overload. A system has a capacity, and a load. If the load is over the capacity, then not everything is gonna happen. This seems like a pretty basic law of, like, doing stuff.

[reads more]

Oh, OK that's a lot of words to say "benchmark to find bottlenecks" and "focus on bottlenecks when optimising" but I understand, sometimes it takes a lot of words to get this simple idea across. There's a few times over the years since it was published that this article would have been perfect to send to a special someone who keeps insisting we implement Merkle trees or whatever in order to speed up a product that reads in all its files using

    while (fscanf(cur_file, "%c", ch)) {
        // process character here
    }

Re: Queues don't fix overload (2014)

#86

Post by john-tells-all »

jmull wrote: post #19

Queues aren't really the problem here. It's that the people making changes don't have a decent understanding of the system they are trying to fix. If you don't actually know what the problem is, your fix is not likely to work. I have seen groups put huge amounts of work into a "fix" for a system when they are only really guessing at what the problem is. (Besides queues, people also seem to like adding "caches" -- oft…

Strong agree.

Caches can be horrible:

- adds complexity of code and architecture

- adds run-time complexity, with new failure modes

- adds semi-random delays

- see also: Buffer Bloat

There's a reason it's one of the Two Hard Problems in Computer Science! https://martinfowler.com/bliki/TwoHardThings.html

Re: Queues don't fix overload (2014)

#87

Post by taneq »

jmull wrote: post #19

Queues aren't really the problem here. It's that the people making changes don't have a decent understanding of the system they are trying to fix. If you don't actually know what the problem is, your fix is not likely to work. I have seen groups put huge amounts of work into a "fix" for a system when they are only really guessing at what the problem is. (Besides queues, people also seem to like adding "caches" -- oft…

> I have seen groups put huge amounts of work into a "fix" for a system when they are only really guessing at what the problem is.

This is sadly far more the rule than the exception. So many times I've heard "we think the problem is [wild-ass guess] and so we're going to do [lengthy re-write] to fix it." And when I say "hey uh maybe we could do [simple test] to prove whether [wild-ass guess] is actually happening?" it's "oh no we can't waste time we have to start the rewrite" or even just "huh, yeah I guess we could test it that way, anyway... we start the rewrite tomorrow".

Edit: one specific case I remember talking to a mechanical engineer about how a particular mass-spring system would react to a momentary impulse and he said "well there's no way to know what'll happen so let's just try it." Thinking back I can rationalise this answer as "we don't fully understand the dynamics of the piece of third-party equipment generating the impulse so let's just use this as a starting point" but at the time it made me so mad. Like, if only there were a discipline entirely dedicated to using mathematical modeling techniques to predict the behaviour of physical systems in order to back-calculate the necessary physical parameters to achieve a given behaviour.

Re: Queues don't fix overload (2014)

#88

Post by john-tells-all »

This article is a version of Theory of Constraints[0] aka Value Stream Mapping:

- every system has a bottleneck

- fixing things _after_ the bottleneck will have no effect

- fixing things _before_ the bottleneck will make the bottleneck worse

https://en.wikipedia.org/wiki/Theory_of_constraints

Re: Queues don't fix overload (2014)

#89

Post by 10000truths »

naasking wrote: post #51
marcosdumay wrote: post #35

Earlier quoted context omitted.

No, load shedding and back pressure present you trade-offs to deal with an overloaded system. Queues don't. Queues just present you problems. If you take an overloaded system and add a queue, every single feature either gets worse or doesn't get any better. And people like to deny this, and pretend that queues will help. They absolutely help with a lot of things but they do nothing but harm in front of an overloaded…

Your mistake is characterising all overloads as the same. Adding a queue to a system that is consistently overloaded won't solve the overload, but many overloads are temporary/bursty, like the Slashdot/HN effect. Queues absolutely do solve these kinds of overloads simply by increasing latency, assuming increased latency is an acceptable choice in your context, of course.

Maximum acceptable response times for websites is a few seconds. The HN/Slashdot effect lasts for minutes to hours. That time scale is way too long for a queue to be effective.

Re: Queues don't fix overload (2014)

#90

Post by kccqzy »

pjdesno wrote: post #84
PaulHoule wrote: post #40

Earlier quoted context omitted.

See https://en.wikipedia.org/wiki/Bufferbloat I lived in Germany in 1999 and then the internet connection from Germany to the US would get overloaded during the day. At maybe 9am the latency would be then it would start dropping packets. I don't know if it was the intention but it was about as good as a ban on VoIP at preventing people from making international VoIP calls. Today there is more consciousness about the…

See RFC 970, "On Packet Switches With Infinite Storage" by John Nagle: https://datatracker.ietf.org/doc/html/rfc970 Back then (I was studying networking as an undergrad at the time, and interned with the Arpanet team) people really did think of network congestion as a buffer allocation problem, so the obvious solution was more buffering - i.e. adding queues. Nagel was one of the first people to point out the problem…

That's because it is obvious and intuitive even though it's wrong. In the first year of my undergrad in an assignment I added unlimited buffering to every single place that needed buffering. I remember a sense of superiority from all the extra code I wrote to do it. Of course buffering felt correct! That was long before I heard of the word backpressure.
Post reply on HN