Seems like OP has conflated the job queue and the job scheduler here.
Job queues are deceptively tricky
31–40 of 59 posts
Re: Job queues are deceptively tricky
#32Earlier quoted context omitted.
> Pre-Emptive scaling ala erlang can help with scenario one somewhat, if the jobs aren’t locked on some resource. For example, on my erlang system 20 would all run just each slightly slower as they get a smaller amount of scheduler reductions each. Well, memory is one of these resources that you are often locked on. If you have enough memory, running 20x the load just goes 20x slower. But if memory is congested, then…
I’ve never enabled swap on a production system in ~25 years, so i have to say it’s not really my experience. Shared memory can be a problem, sure, which is why I don’t generally use that either. Erlang, of course, generally does not use shared memory outside of some cases with ETS etc which must be used carefully but i’d rather solve those problems myself. Concurrency systems in other languages i’ve written, for exam…
To make my point more abstractly:
Multiple tasks can share the CPU and just get a bit slower. But if you are out of RAM, you are better off running one thing after another.
Whether you hit swap or OOM was a distraction.
Re: Job queues are deceptively tricky
#33Major lesson from when I worked on Google Search indexing is that queues have a lot of hidden complexity and can make your outages much longer than they need to be. We had a big project to get rid of a bunch of queues by just scaling up our synchronous backends and making them faster.
[deleted]
Re: Job queues are deceptively tricky
#34I remember learning about CSV parsing and how it's conceptually simple, yet beyond the simple , and quotes: the corner cases bloat your parser 10-15x.
It’s pretty simple to make one that’s RFC compliant. The rules aren’t much more than what you said.
Are you talking about trying to interpret malformed data?
Re: Job queues are deceptively tricky
#35[flagged]
Re: Job queues are deceptively tricky
#36The UNIX pipe really is an incredible concurrency invention which is not well understood, and attempts to work around its features turn into bugs.
A buffer to accumulate data that blocks when it’s full allows you to handle bursty loads. It solves the back pressure problem of readers and writers operating at different speeds. It doesn’t over consume resources.
It also solves the architecture problem of when to trigger work. Both consumer and producer act on the pipe imperatively, rather than one end being imperative and the other being a declarative graph of callbacks (all those “reactive” libraries).
Even using a term like “back pressure” is a tell to me that someone is confused snd doing something architecturally wrong.
Re: Job queues are deceptively tricky
#37Earlier quoted context omitted.
Anyone know a good into to queuing theory?
There are lots of good books and some great vids on youtube, but I'd start with this statement and work backward, because this is the non-obvious thing most bootcamp trained, promoted to CTO don't know: The single most important lesson from queuing theory for software systems is the non-linear relationship between utilisation and latency. As system utilisation approaches 1.0 (100% capacity), the average waiting time…
To explain this: if the system is 95% utilized, and a new request comes in, there's a 95% chance the system is already busy and the request has to wait. But after the currently in-progress request finishes, there's still a 95% chance the system is busy (with another request that was already queued behind it) and request X has to wait. After that one finishes, same thing. On average, request X has to wait for about 20 other requests at 95% utilisation - or 5 requests at 80% utilisation - or 10000 requests at 99.99% utilisation. And that's just the mean, not percentiles.
Re: Job queues are deceptively tricky
#38Earlier quoted context omitted.
Care to share more about the issues?
Not OP but also worked on Google Search once upon a time. I'm not sure if I'm remembering the same issues as OP, but basically the two biggest issues are: 1.) What they do to your 95th percentile latency. Users are often very sensitive to tail latency: a service that responds in 150ms 19 times and then takes 2s on the 20th is still perceived as annoyingly slow. With job queues, the reason for that slowness could be a…
I also worked on services at Google, both as an SRE and a SWE.
I think one subtlety worth bearing in mind here is that Google, at least during my tenure (2006-2014) didn't actually have a proper message queueing system outside of Gmail, which was used only for email delivery. It offered engineers:
1. RPC with infinite backoff/retry (a fun default).
2. Batch jobs that processed files.
but there was no equivalent to a standard enterprise MQ product like ActiveMQ, Artemis, Oracle AQ and so on.
So when we talk about "job queues" it's worth being very precise about what is meant. A standard enterprise message queue broker doesn't have problems with overload because workers pop work off the queue at whatever speed they can operate. The problem of cascading failures and services coming back only to be immediately overloaded again was a problem caused by the design of Google's infrastructure, in which RPCs would back up in memory in the clients and be retried in a loop until the service came back. So of course this required a lot of custom work to create proper backpressuring, which wasn't normally done and especially not on interactive serving paths, leading to this kind of repetitive failure mode. It was all made harder by the practice of making everything fully async, so thread pool sizes also didn't exert backpressure.
Looking back on my time there, one of the pieces of "normal" enterprise infrastructure that I really think Google could have benefited from was a proper JMS compliant scalable message broker. You can buy these - the Oracle Database has one - but Google neither bought one nor developed its own.
Probably they have long since rectified this oversight.
Re: Job queues are deceptively tricky
#39The schedule interval vs. hard timeout distinction is the useful bit here. Treating them as one setting makes backlog behavior much harder to reason about.
Re: Job queues are deceptively tricky
#40Earlier quoted context omitted.
Properly scaling queue consumers is a problem I've spent a lot of time on in the last few years. Working on a messaging platform with highly variable traffic, including close to zero during the night, means that capacity provisioning according to the max will be very costly, and lead to a lot of frustration when you are saturated anyway. Indeed you need backpressure but the traditional methods (CPU usage or similar m…
> capacity provisioning according to the max will be very costly I’m skeptical. You can support a pretty massive messaging system with one box. What did you try? What went wrong?
[deleted]