71–80 of 99 posts
Many engineers don’t truly care about the correctness issue, until it’s too late. Similar to security. Or they care but don’t bother checking whether what they’re doing is correct. For example, in my field, where microservices/actors/processes pass messages between each other over a network, I dare say >95% of implementations I see have edge cases where messages might be lost or processed out of order. But there isn’…
At work we use Temporal and ended up using a dedicated workflow and signals to do distributed locking. Working well so far and the implementation is rather simple, relying on Temporal’s facilities to do the distributed parts of the lock.
At work we use Temporal and ended up using a dedicated workflow and signals to do distributed locking. Working well so far and the implementation is rather simple, relying on Temporal’s facilities to do the distributed parts of the lock.
I'm keen to use Temporal, but I've heard it can be flaky. In your experience has it worked well?
This overcomplicates things... * If you have something like what the article calls a fencing token, you don't need any locks. * The token doesn't need to be monotonically increasing, just a passive unique value that both the client and storage have. Let's call it a version token. It could be monotonically increasing, but a generated UUID, which is typically easier, would work too. (Technically, it could even be a has…
Git push's `--force-with-lease` option does essentially this. (Honestly, they should rename `--force-with-lease` to just `--force`, and rename the old `--force` behaviour to `--force-with-extreme-prejudice` or something like that. Basically make the new behaviour the default `--force` behaviour.)
Earlier quoted context omitted.
To be clear, my objection is to the premise, not to the offered solution. To your question, could you clarify what exactly you mean by the rack "going down"? This encompasses a lot of different scenarios, I'm not sure which one you're asking about. The obvious interpretation would break all the connections the program has to the outside world, thus preventing the problem by construction.
The rack could go down from the point of view of the storage service, but the machine/VM itself could be perfectly fine.
Earlier quoted context omitted.
The assumption that your server will always receive RST or FIN from your client is incorrect. There are some cases when these packets are being dropped, and your server will stay with an open connection while the client on the remote machine is already dead. P.S. BTW, it's not me who downvoted you
I made no such assumption this will always happen though? That's why the comment was so much longer than just "isn't TCP RST enough?"... I listed a ton of ways to deal with this that didn't involve letting the program continue happily on its path.
Earlier quoted context omitted.
I made no such assumption this will always happen though? That's why the comment was so much longer than just "isn't TCP RST enough?"... I listed a ton of ways to deal with this that didn't involve letting the program continue happily on its path.
Sorry didn't see your message. What I mean is that if you are not getting RST/FIN or any other indication for your closed communication channel, you only left to the mechanism of timeouts to recognize a partitioned/dead/slow worker client. Basically, you've mentioned them yourself ("timeouts, lack of heartbeats, etc" in your post are all forms of timeouts). So you can piggyback on these timeouts or use a smaller time…
Timeouts were a red herring in my comment. My problem wasn't with the mere existence of timeouts in corner cases, it was the fact that the worker is assumed to keep working merrily on, despite the timeouts. That's what I don't understand the justification for. If the worker is dead, then it's a non-issue, and the lease can be broken. If the system is alive, the host can discover (via RST, heartbeats, or other timeouts) that the storage system is unreachable, and thus prevent the program from continuing execution -- and at that point the storage service can still break the lease (via a timeout), but it would actually come with a timing-based guarantee that the program will no longer continue execution.
Earlier quoted context omitted.
I'm keen to use Temporal, but I've heard it can be flaky. In your experience has it worked well?
Rock solid in my experience and kind of a game changer. I’m surprised it’s not more widespread in large orgs.
Earlier quoted context omitted.
So basically your answer (and the correct answer most of the time) was that you don't really need distributed locks even if you think you do :)
Heh, in my local developer community I have a bit of a reputation for being “the guy” to talk to about distributed systems. I’d done a bunch of work in the early days of the horizontal-scaling movement (vs just buying bigger servers) and did an M.Sc focused on distributed systems performance. Whenever anyone would come and ask for help with a planned distributed system the first question I would always ask is: does t…
Earlier quoted context omitted.
The rack could go down from the point of view of the storage service, but the machine/VM itself could be perfectly fine.
In that scenario the machine would become aware that it can't reach the storage service either, no? In which case the host can terminate the program, or the network can break all the connections between them, or whatever. By default I would think that the lease shouldn't be broken until the network partition gets resolved, but I think the storage system could have a timeout for breaking the lease in that scenario if…