31–40 of 73 posts
Edit: Why is this downvoted? It is a serious question.
Earlier quoted context omitted.
etcd would be a fine choice, but Consul provides a lock feature[1] out of the box. 1. https://www.consul.io/docs/guides/leader-election.html
As the article explains at length, using Zookeeper or etcd as a "locking service" in this way is not safe, even if the service is perfect and failure-free. There is a fundamental race condition between finding out that you've obtained a lock, and doing some other operation that assumes mutual exclusion.
The author's solution is to push serialization logic into the resource/storage layer (by checking fencing tokens). But what if the resource is itself distributed? Then it needs it's own synchronization mechanism? It's locks all the way down.
Thanks to Martin for analyzing Redlock, I looked forward to an analysis. I don't agree with the two main arguments of the analysis. TLDR: I think the unique random token of Redlock is enough for Check & Set when the lock holder work materializes into a database write (also note that it requires a linearizable database, to check token_A_ID > token_B_ID in the Martin proposed solution), and I think the system model is…
The quality of your C code running on a single node is by all accounts really good. But you sometimes speak about distributed systems in a way that is not correct to people who have really spent a lot of time making these things work at scale. I've done distributed systems, and in fact I usually don't implement systems that are 100% correct under maximally-adverse conditions, for performance and convenience reasons.…
About the ad-hoc logic, in this case I think the problem is actually that you want to accept only what already exists. For example there are important distributed systems papers where the assumption of absolute time bound error among processes is made, using GPS units. This is a much stronger system model (much more synchronous) compared to what I assume, yet this is regarded as acceptable. So you have to ask yourself if, regardless of the fact this idea is from me, if you think that a computer, using the monotonic clock API, can count relative time with a max percentage of error. Yes? Then it's a viable system model.
About you linking to Aphyr, Sentinel was never designed in order to make Redis strongly consistent nor to to make it able to retain writes, I always analyzed the system for what it is: a failover solution that, mixed with the asynchronous replication semantics of Redis, has certain failure modes, that are considered to be fine for the intended use case of Redis. That is, it has just a few best-effort checks in order to try to minimize the data loss, and it has ways to ensure that only one configuration wins, eventually (so that there are no permanent split brain conditions after partitions heal) and nothing more.
To link at this I'm not sure what sense it makes. If you believe Redlock is broken, just state it in a rigorous form, and I'll try to reply.
If you ask me to be rigorous while I'm trying to make it clear with facts what is wrong about Martin analysis, you should do the same and just use arguments, not hand waving about me being lame at DS. Btw in my blog post I'll detail everything and show why I think Redlock is not affected by network unbound delays.
If you want to have a truly parallel system, you cannot share resources between processes. If you need a lock, then that implies sharing of resources... Which implies limited scalability according to Amdahl's law. To achieve unlimited scalability, you need to make sure that no single data store is shared between all processes and that the amount of interprocess communication (on a per-process basis) doesn't increase…
Interesting document a colleague of mine wrote: https://specs.openstack.org/openstack/openstack-specs/specs/...
Why not just use the database to handle the "locking" for you? For example, to ensure that an email with ID=123 gets sent only once, just check if "email 123 sent" is in the database, otherwise commit it to the database, wait for the transaction to be committed, and send the email. Edit: Why is this downvoted? It is a serious question.
Why not just use the database to handle the "locking" for you? For example, to ensure that an email with ID=123 gets sent only once, just check if "email 123 sent" is in the database, otherwise commit it to the database, wait for the transaction to be committed, and send the email. Edit: Why is this downvoted? It is a serious question.
Perhaps because your example fails to describe what happens if there is a crash, after updating the database to say "email sent", but before the email has actually been sent.
Why not just use the database to handle the "locking" for you? For example, to ensure that an email with ID=123 gets sent only once, just check if "email 123 sent" is in the database, otherwise commit it to the database, wait for the transaction to be committed, and send the email. Edit: Why is this downvoted? It is a serious question.
In the scenario you describe, the database commit works, you send the email but... you get an error back from the email send function. Now what?
This is a basic distributed computing problem. Choosing how to solve it can have a drastic effect on your code and your infrastructure. If the secondary function is "charge a credit card" obviously you can't do that twice. But if it's merely "send an email" then maybe it's okay if people get a duplicate.
Google lease/lock/deadlock/race condition and read up on why databases tend to make for bad implementations.
Earlier quoted context omitted.
Perhaps because your example fails to describe what happens if there is a crash, after updating the database to say "email sent", but before the email has actually been sent.
Well, I didn't want to go into that much detail here, because that also presents a problem in the case of locks (if the locking process crashes, the lock is never released).
Earlier quoted context omitted.
Well, I didn't want to go into that much detail here, because that also presents a problem in the case of locks (if the locking process crashes, the lock is never released).
Good observation! The article also covers that exact scenario - plus a solution called fencing which a proper locking system can help facilitate.
Why I'm thinking more in terms of a database-oriented solution is because there is still the problem of crashing. What happens if the system crashes just before the email is sent? The system somehow needs to remember to restart that task (sending the email) when it comes back up. And this is probably best done through a database anyway.