21–30 of 59 posts
I remember learning about CSV parsing and how it's conceptually simple, yet beyond the simple , and quotes: the corner cases bloat your parser 10-15x.
Tangential, but when dealing with queues, the first thing you want to do is have a basic grounding of queuing theory, and know whether you're optimising for throughput or worker utilisation (i.e. what are your SLAs and efficiency targets?). IME each goal involves fairly different metrics and scaling rules, so you'll want to know what you're prioritising.
Anyone know a good into to queuing theory?
The single most important lesson from queuing theory for software systems is the non-linear relationship between utilisation and latency.
As system utilisation approaches 1.0 (100% capacity), the average waiting time does not scale linearly, it scales hyperbolically. A system running at 95% utilisation is vastly more fragile and slow than one running at 80%, even though the load difference is minor.
I think the the second part was kinda obvious? The moment I read this: > If you’re anything like me, you would probably have said Parallel Spawn, Prefer New, and Wait are perhaps defensible, whereas Prefer Old feels weird/backward. it was pretty obvious I was not anything like him. My intuitive answers are pretty different. - Parallel Spawn is useful, but it's orthogonal to the rest. Even if you have 4 workers, you'l…
One example was running certain ad auctions when rendering websites, or something like that. You don't want to delay serving the side, if the ads are delayed.
So you have a certain wall clock time budget until the rest of the page is assembled to be sent to the user, and if you can fit your ad-serving in there, that's good.
If you have more work than you can currently handle, then it makes sense to continue with the newest open request after you handled the previous request.
The actual system was a lot more complicated, and combined a short, bounded queue on the inside with a large stack on the outside or something like that.
Similar considerations can apply, when you are one of many competing market makers for some financial assets on an exchange.
Basically, if you have a situation where serving quick is a lot more important than the distinction between late and very late (or even dropping the request).
Earlier quoted context omitted.
Not OP but also worked on Google Search once upon a time. I'm not sure if I'm remembering the same issues as OP, but basically the two biggest issues are: 1.) What they do to your 95th percentile latency. Users are often very sensitive to tail latency: a service that responds in 150ms 19 times and then takes 2s on the 20th is still perceived as annoyingly slow. With job queues, the reason for that slowness could be a…
Pre-Emptive scaling ala erlang can help with scenario one somewhat, if the jobs aren’t locked on some resource. For example, on my erlang system 20 would all run just each slightly slower as they get a smaller amount of scheduler reductions each. It’s a hard/interesting problem, and harder still once you’re running across a lot of machines — but if they all get slower under the load it turns out that it’s easier to s…
Well, memory is one of these resources that you are often locked on.
If you have enough memory, running 20x the load just goes 20x slower. But if memory is congested, then this can go arbitrarily slower than running your jobs one after another. Eg when you are swapping to a spinning disk.
I think the the second part was kinda obvious? The moment I read this: > If you’re anything like me, you would probably have said Parallel Spawn, Prefer New, and Wait are perhaps defensible, whereas Prefer Old feels weird/backward. it was pretty obvious I was not anything like him. My intuitive answers are pretty different. - Parallel Spawn is useful, but it's orthogonal to the rest. Even if you have 4 workers, you'l…
At Google we actually had 'prefer new' (ie a stack instead of a queue) for certain jobs, that were likely no longer useful after some time had passed; and where less and less useful the longer you waited until you started. One example was running certain ad auctions when rendering websites, or something like that. You don't want to delay serving the side, if the ads are delayed. So you have a certain wall clock time…
Earlier quoted context omitted.
At Google we actually had 'prefer new' (ie a stack instead of a queue) for certain jobs, that were likely no longer useful after some time had passed; and where less and less useful the longer you waited until you started. One example was running certain ad auctions when rendering websites, or something like that. You don't want to delay serving the side, if the ads are delayed. So you have a certain wall clock time…
That's just another word for priority queueing.
Earlier quoted context omitted.
Pre-Emptive scaling ala erlang can help with scenario one somewhat, if the jobs aren’t locked on some resource. For example, on my erlang system 20 would all run just each slightly slower as they get a smaller amount of scheduler reductions each. It’s a hard/interesting problem, and harder still once you’re running across a lot of machines — but if they all get slower under the load it turns out that it’s easier to s…
> Pre-Emptive scaling ala erlang can help with scenario one somewhat, if the jobs aren’t locked on some resource. For example, on my erlang system 20 would all run just each slightly slower as they get a smaller amount of scheduler reductions each. Well, memory is one of these resources that you are often locked on. If you have enough memory, running 20x the load just goes 20x slower. But if memory is congested, then…
Shared memory can be a problem, sure, which is why I don’t generally use that either. Erlang, of course, generally does not use shared memory outside of some cases with ETS etc which must be used carefully but i’d rather solve those problems myself.
Concurrency systems in other languages i’ve written, for example Go, there are ways to architect to avoid it also. I’d rather go slightly slower and copy be value than have to solve mutex contention and trying to make everything atomic and so on. YMMV, I don’t work in HPC just large complex busy systems.
I think the the second part was kinda obvious? The moment I read this: > If you’re anything like me, you would probably have said Parallel Spawn, Prefer New, and Wait are perhaps defensible, whereas Prefer Old feels weird/backward. it was pretty obvious I was not anything like him. My intuitive answers are pretty different. - Parallel Spawn is useful, but it's orthogonal to the rest. Even if you have 4 workers, you'l…
The obvious use-case is if the result depends on a state and the state has changed since the job has started. Examples would be updating the landing page when a new article has come in (you don't need the outdated landing page w/o the new article) or if you did a code change while compiling (assuming it's not a commit).
Earlier quoted context omitted.
Not OP but also worked on Google Search once upon a time. I'm not sure if I'm remembering the same issues as OP, but basically the two biggest issues are: 1.) What they do to your 95th percentile latency. Users are often very sensitive to tail latency: a service that responds in 150ms 19 times and then takes 2s on the 20th is still perceived as annoyingly slow. With job queues, the reason for that slowness could be a…
Properly scaling queue consumers is a problem I've spent a lot of time on in the last few years. Working on a messaging platform with highly variable traffic, including close to zero during the night, means that capacity provisioning according to the max will be very costly, and lead to a lot of frustration when you are saturated anyway. Indeed you need backpressure but the traditional methods (CPU usage or similar m…
I’m skeptical. You can support a pretty massive messaging system with one box.
What did you try? What went wrong?