11–20 of 99 posts
I suggest reading the comment I left back then in this blog post comments section, and the reply I wrote in my blog. Btw, things to note in random order: 1. Check my comment under this blog post. The author had missed a fundamental point in how the algorithm works. Then he based the refusal of the algorithm on the remaining weaker points. 2. It is not true that you can't wait an approximately correct amount of time,…
This overcomplicates things... * If you have something like what the article calls a fencing token, you don't need any locks. * The token doesn't need to be monotonically increasing, just a passive unique value that both the client and storage have. Let's call it a version token. It could be monotonically increasing, but a generated UUID, which is typically easier, would work too. (Technically, it could even be a has…
This neglects the first reason listed in the article for why you would use a lock. > Efficiency: Taking a lock saves you from unnecessarily doing the same work twice (e.g. some expensive computation). If the lock fails and two nodes end up doing the same piece of work, the result is a minor increase in cost (you end up paying 5 cents more to AWS than you otherwise would have) or a minor inconvenience (e.g. a user end…
Efficiency is one, as you say.
The other main one that comes to mind is to implement other "business rules" (hate that term, but that's what people use), like for a online shopping app, the stock to fulfill an order might be reserved for a time when the user starts the checkout process.
Earlier quoted context omitted.
This neglects the first reason listed in the article for why you would use a lock. > Efficiency: Taking a lock saves you from unnecessarily doing the same work twice (e.g. some expensive computation). If the lock fails and two nodes end up doing the same piece of work, the result is a minor increase in cost (you end up paying 5 cents more to AWS than you otherwise would have) or a minor inconvenience (e.g. a user end…
As mentioned in the article, a non-100%-correct lock can be used for efficiency purposes. So basically use an imperfect locking mechanism for efficiency and a reliable one for correctness.
To be clear, my point is don't use distributed locking for correctness. There are much better options.
Now, the atomicity I mention implies some kind of internal synchronization mechanism for multiple requests, which could be based on locks, but those would be real, non-distributed ones.
This overcomplicates things... * If you have something like what the article calls a fencing token, you don't need any locks. * The token doesn't need to be monotonically increasing, just a passive unique value that both the client and storage have. Let's call it a version token. It could be monotonically increasing, but a generated UUID, which is typically easier, would work too. (Technically, it could even be a has…
I.e. your storage system has two nodes and there are two read-modify-write processes running. Process 1 acquires the first token "abc" and process two also acquires the token "abc". Now process 1 commits, the token is changed to "cde" and the change streamed to node 2. Due to network delay, the change to node 2 is delayed. Meanwhile process 2 commits to node 2 with token "abc". Node 2 accepts the change because it has not received the message from node 1 and your system is now in an inconsistent state.
Note that this cannot happen in a scenario where we have monotonically increasing fencing tokens because that requirement forces the nodes to agree on a total order of operations before they can supply the fencing token.
I suggest reading the comment I left back then in this blog post comments section, and the reply I wrote in my blog. Btw, things to note in random order: 1. Check my comment under this blog post. The author had missed a fundamental point in how the algorithm works. Then he based the refusal of the algorithm on the remaining weaker points. 2. It is not true that you can't wait an approximately correct amount of time,…
Would you use RedLock in a situation where the timeout is fairly short (1-2 seconds maybe), the work done usually takes ~90% of that timeout, and the work you do while holding a RedLock lock MUST NOT be done concurrently with another lock holder?
I think the correct answer here is always "No" because the risk of the lease sometimes expiring before the client has finished its work is very high. You must alter your work to be idempotent because RedLock cannot guarantee mutual exclusion under all circumstances. Optimistic locking is a good way to implement this type of thing while the work done is idempotent.
Many engineers don’t truly care about the correctness issue, until it’s too late. Similar to security. Or they care but don’t bother checking whether what they’re doing is correct. For example, in my field, where microservices/actors/processes pass messages between each other over a network, I dare say >95% of implementations I see have edge cases where messages might be lost or processed out of order. But there isn’…
Eek. This sort of thing can end up with innocent people in jail, or dead.
[0] https://en.wikipedia.org/wiki/British_Post_Office_scandal
I suggest reading the comment I left back then in this blog post comments section, and the reply I wrote in my blog. Btw, things to note in random order: 1. Check my comment under this blog post. The author had missed a fundamental point in how the algorithm works. Then he based the refusal of the algorithm on the remaining weaker points. 2. It is not true that you can't wait an approximately correct amount of time,…
Could you provide links?
I suggest reading the comment I left back then in this blog post comments section, and the reply I wrote in my blog. Btw, things to note in random order: 1. Check my comment under this blog post. The author had missed a fundamental point in how the algorithm works. Then he based the refusal of the algorithm on the remaining weaker points. 2. It is not true that you can't wait an approximately correct amount of time,…
To be honest I've long been puzzled by your response blog post. Maybe the following question can help achieve common ground: Would you use RedLock in a situation where the timeout is fairly short (1-2 seconds maybe), the work done usually takes ~90% of that timeout, and the work you do while holding a RedLock lock MUST NOT be done concurrently with another lock holder? I think the correct answer here is always "No" b…
We had corrupted data bacause of this.
This overcomplicates things... * If you have something like what the article calls a fencing token, you don't need any locks. * The token doesn't need to be monotonically increasing, just a passive unique value that both the client and storage have. Let's call it a version token. It could be monotonically increasing, but a generated UUID, which is typically easier, would work too. (Technically, it could even be a has…
Won't this lead to inconsistent states if you don't do monotonically increasing tokens? I.e. your storage system has two nodes and there are two read-modify-write processes running. Process 1 acquires the first token "abc" and process two also acquires the token "abc". Now process 1 commits, the token is changed to "cde" and the change streamed to node 2. Due to network delay, the change to node 2 is delayed. Meanwhi…
This overcomplicates things... * If you have something like what the article calls a fencing token, you don't need any locks. * The token doesn't need to be monotonically increasing, just a passive unique value that both the client and storage have. Let's call it a version token. It could be monotonically increasing, but a generated UUID, which is typically easier, would work too. (Technically, it could even be a has…