221–230 of 247 posts
Earlier quoted context omitted.
> Do you know what idempotency is? Yes. > This is exactly what he described. So? Idempotency and an exactly-once delivery abstraction are not the same thing.
Agreed. Idempotent _processing_, not delivery.
Earlier quoted context omitted.
> Because you need to understand that your processing code is constrained by the fact that you can't get exactly-once delivery. Why do I need to understand that? Why can I not put an abstraction layer that provides me with the illusion of exactly-once delivery? > There is no library that can make that just go away Well, this is the thing that I dispute. I believe that there is a library I can write to make it go away…
TCP implementations are an abstraction that work 99.99% of the time, but are still vulnerable to two generals when you look close. TCP is implemented in the kernel with a buffer, the kernel responds with ACKs before an application reads the data. There is no guarantee that the application reads from that buffer (e.g. the process could crash), so the client on the other end believes that the application has received t…
What makes you think the 2GP is relevant here? The 2GP has to do with coordination and consensus, not exactly-once delivery.
> TCP is implemented in the kernel with a buffer, the kernel responds with ACKs before an application reads the data.
True. Why do you think that matters?
> There is no guarantee that the application reads from that buffer
So? What does that have to do with exactly-once delivery? Even if the application does read the data, there's no guarantee that it does anything with it afterwards.
> The kernel is handling at-least-once delivery with the network boundary and turning it into at-most-once with the process boundary.
OK, if that's how you're defining your terms, I agree that you cannot have exactly-once delivery. But that makes it a vacuous observation. You can always get at-most-once delivery by disconnecting the network entirely. That provides a 100% guarantee that no message will be delivered more than once (because no message will ever be delivered at all). But that doesn't seem very useful.
Earlier quoted context omitted.
This isn't about indefinite communication loss. Obviously no progress is possible in that case. The two generals' problem has nothing to do with a permanent failure. I think there is a lot of literature out there if you're really interested in understanding and I'm happy to provide more links if you'd like.
What makes you think the 2GP is relevant here? The 2GP has to do with coordination and consensus, not exactly-once delivery.
Particularly:
> The sender cannot know if a message was delivered since transport is unreliable; thus, one or more acknowledgement messages are required. Moreover, the sender cannot distinguish between message delivery errors, acknowledgement delivery errors, or delays (either in processing or because or network unreliability).
> The recipient is forced to send the acknowledgement only after the message is either processed (or persisted for processing) because an acknowledgement before processing would not work: if the recipient exhibits a fault before processing that would cause the loss of the message.
> In the case that that acknowledgement is lost, the sender can’t know (due to lack of insight in the recipient’s state) whether the recipient failed before scheduling the message for processing (in essence losing the message) or if the recipient is just running a bit slow, or if the acknowledgement message was lost. Now, if the sender decides to re-deliver, then the recipient may end up receiving the message twice if the acknowledgement was dropped (for example). On the other hand, if the sender decides to not re-deliver, then the recipient may end up not processing the message at all if the issue was that the message wasn’t scheduled for processing.
Earlier quoted context omitted.
What makes you think the 2GP is relevant here? The 2GP has to do with coordination and consensus, not exactly-once delivery.
The link I posted above explains the connection: https://blog.bulloak.io/post/20200917-the-impossibility-of-e... Particularly: > The sender cannot know if a message was delivered since transport is unreliable; thus, one or more acknowledgement messages are required. Moreover, the sender cannot distinguish between message delivery errors, acknowledgement delivery errors, or delays (either in processing or because or n…
"Let’s assume that a protocol exists which guarantees that a recipient receives a message from the sender once and only once. Such a protocol could then solve the two generals problem! Representing the time of the attack as the message, the first general (the sender) would only need to adhere to the protocol for the second general (recipient) to have received the attack time exactly one time. However, since we know that this is not possible, we also know that exactly once is not possible."
The 2GP is not just the first general knowing that the second general received the message. The 2GP is the problem of achieving common knowledge between the two generals, i.e. it's not just that G1 needs to know that G2 got the message, it is that G2 needs to know that G1 knows that G2 got the message, and G1 needs to know that G2 knows that G1 knows that G2 got the message, and so on.
Exactly-once delivery is possible. The only thing that is not possible is for the sender to know when the message has been received so that no duplicates are sent. But exactly-once delivery is not only possible, it's trivial. All you need to do is discard duplicates at the receiver.
Earlier quoted context omitted.
How so?
More immediate response to disconnection. Rather than "IDK keep sending but I haven't heard ACK in a while"
It's the conceptual equivalent of your phone "helpfully hanging up" when you change the phone from one ear to the other during a call.
Earlier quoted context omitted.
The link I posted above explains the connection: https://blog.bulloak.io/post/20200917-the-impossibility-of-e... Particularly: > The sender cannot know if a message was delivered since transport is unreliable; thus, one or more acknowledgement messages are required. Moreover, the sender cannot distinguish between message delivery errors, acknowledgement delivery errors, or delays (either in processing or because or n…
This proof is flawed: "Let’s assume that a protocol exists which guarantees that a recipient receives a message from the sender once and only once. Such a protocol could then solve the two generals problem! Representing the time of the attack as the message, the first general (the sender) would only need to adhere to the protocol for the second general (recipient) to have received the attack time exactly one time. Ho…
...yes, which would be exactly once _processing_ but not exactly once _delivery_.
Unless you're wanting to redefine "exactly once delivery" to mean "at least once delivery but I'm calling it exactly once because I have a strategy to cope with duplicate messages"
Earlier quoted context omitted.
> The mechanism you're describing already exists. Yes, I know, which makes it all the more bizarre that people are claiming that this is impossible. > Systems which have "exactly once" delivery typically do so... Ah, so exactly-once delivery is possible after all?
If your goal is simply to look smart then absorbing the subtlety of what is said to you before you reply should be top of list.
Earlier quoted context omitted.
Agreed. Idempotent _processing_, not delivery.
Uh, what exactly do you think is "agreed" here? My claim is that idempotent processing can produce exactly-once delivery, and so the original claim that you "cannot have exactly-once delivery" is false.
Idempotent processing is exactly once processing. This is true. Delivery is from the sender's point of view, not the recipient. How the recipient processes the information, idempotently or not, is not the concern of the sender.
"Cannot have exactly-once delivery" is true and is as true as not dividing by zero. Read about the two general's problem. It is, quite literally, impossible to have exactly once delivery. Like, your friend is in the other room, and you shout "someone is at the door" - how do you _know_ your friend heard you? If you shout again, you are attempting another delivery. What do you do if your friend doesn't respond? In exactly once delivery, you guarantee they heard you in the other room. In exactly once processing, you can shout a couple of times until you hear them acknowledge.
You may think that this is not material and at the end of the day, as long as one thing is processed, then who cares? Well, you have to understand how delivery can fail otherwise you will handle the failure incorrectly. Is it safe to try again? If I say "transfer money from me to you", and I don't hear back, is it safe to again say "transfer money from me to you" again? Will I be double charged?
Earlier quoted context omitted.
This proof is flawed: "Let’s assume that a protocol exists which guarantees that a recipient receives a message from the sender once and only once. Such a protocol could then solve the two generals problem! Representing the time of the attack as the message, the first general (the sender) would only need to adhere to the protocol for the second general (recipient) to have received the attack time exactly one time. Ho…
> All you need to do is discard duplicates at the receiver. ...yes, which would be exactly once _processing_ but not exactly once _delivery_. Unless you're wanting to redefine "exactly once delivery" to mean "at least once delivery but I'm calling it exactly once because I have a strategy to cope with duplicate messages"
What exactly is the difference? What counts as "delivery"? How do you do "delivery" (on a computer) without doing at least some "processing"?
Earlier quoted context omitted.
Uh, what exactly do you think is "agreed" here? My claim is that idempotent processing can produce exactly-once delivery, and so the original claim that you "cannot have exactly-once delivery" is false.
Change your words from "delivery" to "processing" and you in alignment with reality. Idempotent processing is exactly once processing. This is true. Delivery is from the sender's point of view, not the recipient. How the recipient processes the information, idempotently or not, is not the concern of the sender. "Cannot have exactly-once delivery" is true and is as true as not dividing by zero. Read about the two gene…
You are not the first to say this, but so far no one has been able to explain what the difference between "delivery" and "processing" is. How do you do "delivery" (on a computer) without also doing (at least some) "processing"?
> Delivery is from the sender's point of view, not the recipient.
I don't see what difference that makes. In fact, I don't see why a "point of view" should enter into it at all. Whether a message has been "delivered" or not (whatever that actually turns out to mean) is (it seems to me) a property of the system, independent of anyone's point of view.