61–70 of 72 posts
[1] https://ignite.apache.org/docs/latest/key-value-api/continuo...
I liked the article a lot. I prefer the term retryable to idempotent. If there's a failure in the first call, to be truly idempotent it should fail on the second. Retryable on the other hand is easier to argue about. Important thing is not the response but the end state of the system.
Idempotent means that repeating the same operation eventually stabilizes, not that Op^2 = Op. It combines retryability of failures with retryability of successes. Alternatively, idempotency applies to successful operations, orthogonal from error cases. Retryability doesn't help in the case of the (bad) operation "+1" which is not idempotent.
I'm a newbie and a little confused. On one hand there are posts like this that claim exactly-once delivery and distributed locks are impossible. But on the other hand, if I look at the docs of a distributed database, say Apache Ignite, they will say that they have exactly-once delivery [1] and distributed locks [2]. So ... which is it? [1] https://ignite.apache.org/docs/latest/key-value-api/continuo... [2] https://ig…
If you want to see a ton of lies, go and start reading through all the Jepsen posts tearing down those dubious claims.
I'm a newbie and a little confused. On one hand there are posts like this that claim exactly-once delivery and distributed locks are impossible. But on the other hand, if I look at the docs of a distributed database, say Apache Ignite, they will say that they have exactly-once delivery [1] and distributed locks [2]. So ... which is it? [1] https://ignite.apache.org/docs/latest/key-value-api/continuo... [2] https://ig…
The second (lock) is actually a lease fwict, and writing code in the locked body that assumes it will be truly mutually exclusive is pretty dangerous (see the linked post from Martin [1] for why).
[1] https://martin.kleppmann.com/2016/02/08/how-to-do-distribute...
Earlier quoted context omitted.
> a lease with an infinite timeout unless manually returned I would argue that "infinite timeout" is another negative shibboleth. every operation in a distributed system has some duration after which you can be 99.9% confident (or 99.9999%, or whatever threshold you want to pick) that it was lost to the void and will never return a result. in a robust distributed system, you want to pick a reasonable timeout value, a…
I agree; it's also one of the few issues I have with Linux (and possibly Unix generally). Zombie processes (dependent on some lock that will never clear) shouldn't be possible. At the very least abort (kill -9) should always be possible. Failure should always be an option; it should be the default assumption. All other order must be wrested from that chaos.
The kernel retains minimal state about them because the system has made a promise to report that the process exited to its parent process, and the parent process hasn't gotten around to asking for that yet.
(Don't confuse zombies with uninterruptible I/O sleep, or buggy kernel workers.)
Earlier quoted context omitted.
Is there a ‘standard’ way to test the positive shibboleths for existence? I am not necessarily thinking just tests running as code. Although that would be nice.
I think it's basically pay Aphyr/Jepsen $mondoConsultingRates. :)
Earlier quoted context omitted.
I feel like the correct approach is accepting that determinacy is nonsensical in a world where time is relative and instead doubling down on nondeterministic (but predictable!) algorithms. This means leveraging concepts like commutativity and associativity to ensure predictability.
> nondeterministic (but predictable!) Huh? How could a nondeterministic algorithm be predictable? Do you mean algorithms with a nondeterministic but overall irrelevant component ("pick a random element from this set")?
Earlier quoted context omitted.
I agree; it's also one of the few issues I have with Linux (and possibly Unix generally). Zombie processes (dependent on some lock that will never clear) shouldn't be possible. At the very least abort (kill -9) should always be possible. Failure should always be an option; it should be the default assumption. All other order must be wrested from that chaos.
Zombie processes are already dead and aborted, there's nothing more a kill -9 would do to them. The kernel retains minimal state about them because the system has made a promise to report that the process exited to its parent process, and the parent process hasn't gotten around to asking for that yet. (Don't confuse zombies with uninterruptible I/O sleep, or buggy kernel workers.)
Stuck mounts have a half solution (lazy unmounts) but even _that_ interface really also needs a timeout value after which operations on the target should be assumed to fail rather than return correctly.
Offhand, I wonder if there's currently or previously been a DoS attack based on defunct uninterruptible sleep. Theoretically a system could be exhausted of PIDs which could lead to nasty issues.
Earlier quoted context omitted.
Zombie processes are already dead and aborted, there's nothing more a kill -9 would do to them. The kernel retains minimal state about them because the system has made a promise to report that the process exited to its parent process, and the parent process hasn't gotten around to asking for that yet. (Don't confuse zombies with uninterruptible I/O sleep, or buggy kernel workers.)
I might have done that last line, but I do mean generally. Even if it isn't kill -9 (which it should be, somehow, since that's the human's intent when they use it); there should be some mechanism for reaching the 'failed' state and the process itself leaving the accounting tables. Stuck mounts have a half solution (lazy unmounts) but even _that_ interface really also needs a timeout value after which operations on th…
Once again, that cannot be done until the parent process consumes the exit status. That's what the zombie is there for. Zombies don't take up much space.
> Stuck mounts have a half solution (lazy unmounts) but even _that_ interface really also needs a timeout value after which operations on the target should be assumed to fail rather than return correctly.
These days most NFS etc mounts are "soft mounts", that is operations will eventually time out.
Lazy unmount doesn't really apply here, it makes the mountpoint disappear from the global namespace, but all existing open files remain untouched, and the mount lives as long as anything is still using it; it just removes the "entry point" to the mount.
On today's Linux, it's up to each filesystem to provide abort/timeout mechanism. For timeouts, this is the right design, as demonstrated by macOS complications with FUSE. I do wish there was a common way to make things abort.
There was a patch in circulation a long time ago, that could seamlessly switch all open FDs of any given mountpoint into a whole different filesystem named badfs. badfs would just return an error on any operation. As far as I know, that patch never got merged, probably because nobody ever got it working 100%.
That kind of a DoS would require a local attacker, and then the victim to access a mountpoint owned by the attacker. Using FUSE, you could get a lot of processes hanging like that, for sure. I guess you could trap a mail delivery agent, if you still had a system where mail was delivered to users' home directories.
However, forcibly aborting any FUSE mount is a single `echo 1 >/sys/fs/fuse/connection/NNNN/abort`, the only challenge is finding the right ID. (See https://github.com/bazil/fuse/blob/fb710f7dfd05053a3bc9516dd...)
> database vendors might try just a little harder to tell the truth... Come on, you know that's not what's going to happen. If they notice at all, they'll just incorporate the magic phrases into their BS so you have to hunt harder for a real signal.
Oh shoot I forgot Shibboleths have to remain secret, I have made a terrible mistake.
Less /s, I'd be more worried if I thought the salescritters were listening, but they're mostly not.