Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Job queues are deceptively tricky

typesanitizer.com

Post reply on HN
Print view View original

11–20 of 59 posts

Re: Job queues are deceptively tricky

#11

Post by nachexnachex »

nostrademons wrote: post #9
esafak wrote: post #8

Earlier quoted context omitted.

Care to share more about the issues?

Not OP but also worked on Google Search once upon a time. I'm not sure if I'm remembering the same issues as OP, but basically the two biggest issues are: 1.) What they do to your 95th percentile latency. Users are often very sensitive to tail latency: a service that responds in 150ms 19 times and then takes 2s on the 20th is still perceived as annoyingly slow. With job queues, the reason for that slowness could be a…

Properly scaling queue consumers is a problem I've spent a lot of time on in the last few years. Working on a messaging platform with highly variable traffic, including close to zero during the night, means that capacity provisioning according to the max will be very costly, and lead to a lot of frustration when you are saturated anyway.

Indeed you need backpressure but the traditional methods (CPU usage or similar metrics) are difficult because many consumers aren't high on those metrics --imagine a messaging plaform, pure IO. Also you'd have to tailor to the consumer itself and that's difficult, which is what you mention on the next-to-last paragraph.

In the end I helped solving it by scaling based on queue size and input/output rates, agnostic to the consumer itself, but with the hypothesis that you can scale consumers linearly (or at least monotonically, some sublinearity is allowed). The queue scaler watches for incoming and outgoing traffic on the queue, plus items on the queue itself, and it can scale from 0 to 11 in seconds for gusts, then shutting everything down.

It's a satisfying problem to work on, but its proper solution demanded quite the investigation. Now every queue we've got in the system is managed by this autoscaler -- except when we can't ensure linearity.

Re: Job queues are deceptively tricky

#12

Post by tzs »

nostrademons wrote: post #9
esafak wrote: post #8

Earlier quoted context omitted.

Care to share more about the issues?

Not OP but also worked on Google Search once upon a time. I'm not sure if I'm remembering the same issues as OP, but basically the two biggest issues are: 1.) What they do to your 95th percentile latency. Users are often very sensitive to tail latency: a service that responds in 150ms 19 times and then takes 2s on the 20th is still perceived as annoyingly slow. With job queues, the reason for that slowness could be a…

> Users are often very sensitive to tail latency: a service that responds in 150ms 19 times and then takes 2s on the 20th is still perceived as annoyingly slow.

This reminds me of some research I read about in the 1980s or 1990s on perceptions of the speed of command line commands. If command time varied over a range from nearly instantaneous to say 100 ms fairly uniformly through that range, people would perceive the system as overall being faster when the researchers added a variable delay to all the commands that made them all take 100 ms.

Humans apparently really like consistency.

> When SRE attempted restart Service X, requests queued in the job queue were all retried en masse, which led to an overload of the partially-restarted service and a subsequent failure

...and this reminds me of something else, from around 1983. I was working at a small Unix workstation maker. The guy in the office across the hall found one morning that the battery for the clock on his workstation had died, and the system time had come up after boot as the Unix epoch.

He shut down, put in a new battery, booted, and then set the clock to the current time, 13 or 14 years after the epoch.

Almost immediately his hard disk light came steadily along, and he could hear the disk furiously seeking, and the system became completely unresponsive.

It turned out AT&T cron in the early '80s wasn't smart about time changes. It had tried to all at once every cron job that should have run in the 13 or 14 years that the time just jumped.

Re: Job queues are deceptively tricky

#13

Post by [deleted] »

tzs wrote: post #12
nostrademons wrote: post #9

Earlier quoted context omitted.

Not OP but also worked on Google Search once upon a time. I'm not sure if I'm remembering the same issues as OP, but basically the two biggest issues are: 1.) What they do to your 95th percentile latency. Users are often very sensitive to tail latency: a service that responds in 150ms 19 times and then takes 2s on the 20th is still perceived as annoyingly slow. With job queues, the reason for that slowness could be a…

> Users are often very sensitive to tail latency: a service that responds in 150ms 19 times and then takes 2s on the 20th is still perceived as annoyingly slow. This reminds me of some research I read about in the 1980s or 1990s on perceptions of the speed of command line commands. If command time varied over a range from nearly instantaneous to say 100 ms fairly uniformly through that range, people would perceive th…

[deleted]

Re: Job queues are deceptively tricky

#14

Post by keynha »

nosefrog wrote: post #6

Major lesson from when I worked on Google Search indexing is that queues have a lot of hidden complexity and can make your outages much longer than they need to be. We had a big project to get rid of a bunch of queues by just scaling up our synchronous backends and making them faster.

[flagged]

Re: Job queues are deceptively tricky

#15

Post by lkjdsklf »

nosefrog wrote: post #6

Major lesson from when I worked on Google Search indexing is that queues have a lot of hidden complexity and can make your outages much longer than they need to be. We had a big project to get rid of a bunch of queues by just scaling up our synchronous backends and making them faster.

The solution to the complexity of any queuing system is to add another queue

Re: Job queues are deceptively tricky

#16

Post by teleforce »

The most popular resource manager for job submission and queueing system is Slurm. It's being used in majority of TOP500 supercomputers, and overwhelming majority of the world HPCs [1].

SchedMD the leading developer of Slurm has recently being acquired by Nvidia, while Slurm remaining free and open source, but somehow it's Wikipedia entry is not yet updated accordingly.

[1] Slurm Workload Manager:

https://en.wikipedia.org/wiki/Slurm_Workload_Manager

[2] Nvidia Acquires SchedMD (7 comments):

https://news.ycombinator.com/item?id=46277190

Re: Job queues are deceptively tricky

#17

Post by cyberpunk »

nostrademons wrote: post #9
esafak wrote: post #8

Earlier quoted context omitted.

Care to share more about the issues?

Not OP but also worked on Google Search once upon a time. I'm not sure if I'm remembering the same issues as OP, but basically the two biggest issues are: 1.) What they do to your 95th percentile latency. Users are often very sensitive to tail latency: a service that responds in 150ms 19 times and then takes 2s on the 20th is still perceived as annoyingly slow. With job queues, the reason for that slowness could be a…

Pre-Emptive scaling ala erlang can help with scenario one somewhat, if the jobs aren’t locked on some resource. For example, on my erlang system 20 would all run just each slightly slower as they get a smaller amount of scheduler reductions each.

It’s a hard/interesting problem, and harder still once you’re running across a lot of machines — but if they all get slower under the load it turns out that it’s easier to scale / work out a good balance of idle capacity to guarantee x time sla under x requests.

Fast ramp up for additional capacity is important too, but less so if you know to start the process once median execution time drops to some % of your worst target

Re: Job queues are deceptively tricky

#18

Post by rienbdj »

zdc1 wrote: post #10

Tangential, but when dealing with queues, the first thing you want to do is have a basic grounding of queuing theory, and know whether you're optimising for throughput or worker utilisation (i.e. what are your SLAs and efficiency targets?). IME each goal involves fairly different metrics and scaling rules, so you'll want to know what you're prioritising.

Anyone know a good into to queuing theory?
Post reply on HN