Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Background job queues and priorities may be the wrong path

alexis.bernard.io

Post reply on HN
Print view View original

21–30 of 52 posts

Re: Background job queues and priorities may be the wrong path

#21

Post by hamandcheese »

I largely agree, however I do believe that it is necessary to reserve capacity for lower latency jobs if the variance in job durations is large.

For example, suppose you have a burst of 1 hour latency jobs, each of which processes in 10 minutes. It will not take many of these to consume all available workers.

If that burst is followed by a single high priority, 10s latency job. Whelp, that jobs latency objective will not be met, since the soonest that a worker will free up to take this work is 10 minutes.

So I think the ideal worker pool design does include some amount of reserved capacity for low-latency work.

A general purpose workers can of course grab low latency work if it's idle! But the reverse is not true - an idle low-latency worker should not be picking up any long-running job.

Re: Background job queues and priorities may be the wrong path

#22

Post by arakageeta »

karmakaze wrote: post #15
AlotOfReading wrote: post #13

The article is a bit unclear because it's lacking the proper vocabulary. Priorities and deadlines (what the article calls "SLOs") are both valid ways to approach scheduling problems with different tradeoffs. The fixed priority systems the article talks about trade off optimal "capacity" utilization for understandable failure dynamics in the overcapacity case. When you're over capacity, the messages that don't go thro…

Yeah, I don't see how deadlines/SLOs does any better when there's insufficient capacity to meet those of queued jobs at their concurrency. > I don’t think it’s possible to meet a latency target through prioritization when there is a fundamental lack of capacity. This seemingly implies that it would be achievable with target deadlines/SLOs. At capacity I don't see a better solution than give each priority (or target l…

Under deadline scheduling, every pending job _eventually_ has highest priority as time elapses. (Assuming new jobs can’t arrive with deadlines in the past.) Every job is eventually serviced.

The “pain” experienced in an overload situations is spread among all late jobs. Contrast this with fixed-priority scheduling, where lowest priority jobs will be starved completely, until the overload is resolved.

Re: Background job queues and priorities may be the wrong path

#23

Post by andrelaszlo »

hbrundage wrote: post #5

Its interesting to see how the Rails world still thinks in terms of the number of processes listening to a queue, instead of thinking in the cloud-native, elastic, serverless terms. There's always an autoscaling delay, but Rails itself (and the community) don't seem to fit into the serverless paradigm well such that these questions around how to design your queues come up. I think a lot of Lambda developers or Cloud…

Yeah when your Rails monolith's image is several GiBs, uses roughly the same amount of memory, and takes almost a minute to cold start, autoscaling has a lot of inertia and gets pretty expensive.

Re: Background job queues and priorities may be the wrong path

#24

Post by andrelaszlo »

Another important aspect that seems to be overlooked in a lot of these discussions (specifically for Rails, but probably relevant in other contexts as well) is that if your jobs are very heterogeneous in terms of their degree of parallelizability then it will be very difficult to do effective provisioning and capacity planning.

Let's say you have two types of jobs: one that is highly parallelizable (Eg request to third party API), and one that doesn't parallelize well (video transcoding using 100% of available CPU). Then you'd want a lot of threads assigned to each CPU for the first type, but very few for the second type. If they're on the same queue, you can have a few threads and be blocked by job type one most of the time, or have a lot of threads and get tons of context switching when the second type of job dominates. Either way your throughput will be very bad.

My current idea is to partition job types first by their degree of parallelizability (as in Amdahl's law), then by priority if necessary.

Has anyone tried this?

Re: Background job queues and priorities may be the wrong path

#25

Post by reactordev »

taeric wrote: post #14
jupp0r wrote: post #11

Earlier quoted context omitted.

Ideally, you don't have blocking APIs at all. There isn't any reason nowadays. Every blocking API in your system is there for historic reasons (ie it's >20 years old) or because somebody made a terrible design decision in the last 20 years. I am obviously referring to operations that do actual IO or wait for work on other threads only.

I don't know that I agree. Blocking logic is far easier to understand, as it lets you ignore partial operations. It also lets you specify, in a more direct manner, where data is buffered.

One would argue the best approach is non-blocking written in a blocking fashion for clarity. async/await instead of Promises and Thens.

Most languages have this now. For the ones that don’t, they have pointers, so writing a thread pool event loop is trivial.

Re: Background job queues and priorities may be the wrong path

#26

Post by jayunit »

rewmie wrote: post #17
AlotOfReading wrote: post #13

The article is a bit unclear because it's lacking the proper vocabulary. Priorities and deadlines (what the article calls "SLOs") are both valid ways to approach scheduling problems with different tradeoffs. The fixed priority systems the article talks about trade off optimal "capacity" utilization for understandable failure dynamics in the overcapacity case. When you're over capacity, the messages that don't go thro…

Thanks for sharing some of your insight onto the problem domain. Do you happen to have any reference for laymen like me to onboard onto the subject? It's an awfully interesting topic but when I stumbled upon it I tend to fall back to JIT learning and troubleshooting, which is far from the best position to be.

The article from @jph (who commented it at the top level here) discussed a month ago https://news.ycombinator.com/item?id=37532439 might be helpful, too!

Re: Background job queues and priorities may be the wrong path

#29

Post by ickyforce »

hbrundage wrote: post #6
timeagain wrote: post #3

In a similar vein, something about queuing that has annoyed me as a developer for multiple large FANG corporations is poor thinking about queue metrics. The TLDR is that metics provided by the queue itself are rarely helpful for knowing if your service is healthy, and when it is not healthy they are not very useful for determining why. Most queue processing services that I have seen have an alarm on (a) oldest messag…

Makes sense! I was just talking with a Temporal solutions engineer this week and this metric is their recommended one for autoscaling on. Instead of autoscaling on queue depth, you scale on queue latency! Specifically for them they split up the time from enqueue to start, and then the time from start to done, and you scale on the former, not the total ("ScheduleToStart" in their terms).

Time from enqueue to start isn't a good metric - it completely disregards the queue size. Enqueuing 1M jobs won't change this metric as it only updates once the job reaches the front of the queue, and when the 1Mth job does that the situation is already over.

I had much better results with a metric that shows estimated queue time for jobs that are getting enqueued right now (queue_size * running_avg_job_processing_time / parallelism).

Re: Background job queues and priorities may be the wrong path

#30

Post by taeric »

reactordev wrote: post #25
taeric wrote: post #14

Earlier quoted context omitted.

I don't know that I agree. Blocking logic is far easier to understand, as it lets you ignore partial operations. It also lets you specify, in a more direct manner, where data is buffered.

One would argue the best approach is non-blocking written in a blocking fashion for clarity. async/await instead of Promises and Thens. Most languages have this now. For the ones that don’t, they have pointers, so writing a thread pool event loop is trivial.

Maybe, though that is a bit of having your cake and eating it, too.

I am very open to the argument that you should be able to do both. I'm even open to the idea that this is closer to preemptive multitasking. Where the manual version basically lost, and it is taking time for some of us to realize it. :)

Post reply on HN