Yes, you can have exactly-once delivery • Hacker News Board

Re: Yes, you can have exactly-once delivery

#111

Post by lisper » Fri, Sep 20, 2024, 11:27 PM UTC

computerfan494 wrote: post #105

lisper wrote: post #76

Earlier quoted context omitted.

Yeah, but that just seems like a bizarre definition on which to base the claim that you cannot have exactly-once delivery. Obviously, if you define delivery to preclude de-duplication, then you can't have exactly-once delivery. But you can have something that delivers messages (for some reasonable definition of "delivers") exactly once. It seems weird to define delivery in such a way that such a system does not provi…

I'm not sure how to convince you to get over your hangup around how other people feel about the word "delivery", but this is exactly why others distinguish between "delivery" and "processing". If you think there are better terms than those two to describe "the system that receives a message and must be responsible for de-duplication" and "the system that can rely on messages being already de-duplicated", then feel fr…

I don't have any hangups about how other people feel about anything. My "hangup" is that I see no evidence that there is a consensus on a technical definition of the word "delivery". In the absence of such a definition, there is no basis for asserting that exactly-once delivery is impossible, particularly when there are existence proofs to the contrary based on reasonable informal definitions of the word "delivery".

Re: Yes, you can have exactly-once delivery

#112

Post by lisper » Fri, Sep 20, 2024, 11:32 PM UTC

mlyle wrote: post #97

lisper wrote: post #96

Earlier quoted context omitted.

Really? That is what causes you to question whether or not I'm acting in good faith? If that's what you call "calling people out by name" I guess we'll just have to agree to disagree.

The whole refusal to accept that a field could legitimately define something differently than how you prefer, and then running off to blog about it and name names... and then coming around for round II of flamewar... with ever more splitting of hairs in definitions... is not awesome. You are especially well-answered here, I think: https://news.ycombinator.com/item?id=41599131 One reason the delivery / processing dist…

> You are especially well-answered here

I don't see anything there that is at odds with anything I have said. All I see there is a restatement of my position.

> One reason the delivery / processing distinction exists because very often the application needs to atomically...

Yes. Do you really think I did not already know that?

> Thus, the application needs to deal with multiple delivery.

That depends on your requirements. What does that have to do with the possibility or impossibility of exactly-once delivery?

> Imagine a "subtract $5 from my bank account" message with no ID on the message itself

I have never denied that you can invent scenarios that will fail. I explicitly said that exactly-once delivery is likely not what you want. What does that have to do with whether or not it is possible?

Re: Yes, you can have exactly-once delivery

#113

Post by daymanstep » Fri, Sep 20, 2024, 11:35 PM UTC

sam_lowry_ wrote: post #94

Groxx wrote: post #20

Earlier quoted context omitted.

Yea... this is kinda inflammatory but I honestly have to agree with it. The post largely summarizes as "you can have exactly-once delivery if you re-define it to be at-least-once processing with idempotency". Those are different things. In fact, that's the entire point behind saying that it's impossible. You can't design a system that is exactly-once at any level, so don't even bother trying. If someone wants you to…

Gosh, only after this comment I understood why so many programmers litter the code with retries, even though they seem superfluous.

I mean, sometimes it is superfluous and you can get rid most of the code by wrapping everything in a big try-retry handler or something like that.

Re: Yes, you can have exactly-once delivery

#114

Post by lisper » Fri, Sep 20, 2024, 11:36 PM UTC

mlyle wrote: post #88

lisper wrote: post #56

Earlier quoted context omitted.

> A 4 year old piece laying out the exact difference... "Exactly-once delivery guarantee is the guarantee that a message can be delivered to a recipient once, and only once." That seems circular to me. Also, the author's proof is flawed. The 2GP requires more than exactly-once delivery, it requires common knowledge . It is not enough for the first general to know that the message will be received, it is required that…

The piece makes a very simple distinction. Delivery is the property of a message showing up at a receiver, irrespective of the receiver making state changes. Processing is making state changes. You can't dedupe messages without some kind of state change. Your guards, writing down that a given message has been here before, have been delivered the message. An endpoint on a lossy medium has to cope with either (0 or 1)…

> The guards have "processed" the message for this purpose, and the fact that they can deliver it to you over a perfectly reliable channel is moot.

No, it isn't, because the situation inside the fort is different than the situation outside. The odds of a courier being intercepted inside the fort are effectively zero. If your computational model includes a non-zero probability of failure "inside the fort" then you are no longer in the realm of distributed systems, but are now talking about fault-tolerant computing, which is a whole nuther kettle o worms.

Re: Yes, you can have exactly-once delivery

#115

Post by mlyle » Sat, Sep 21, 2024, 12:05 AM UTC

lisper wrote: post #114

mlyle wrote: post #88

Earlier quoted context omitted.

The piece makes a very simple distinction. Delivery is the property of a message showing up at a receiver, irrespective of the receiver making state changes. Processing is making state changes. You can't dedupe messages without some kind of state change. Your guards, writing down that a given message has been here before, have been delivered the message. An endpoint on a lossy medium has to cope with either (0 or 1)…

> The guards have "processed" the message for this purpose, and the fact that they can deliver it to you over a perfectly reliable channel is moot. No, it isn't, because the situation inside the fort is different than the situation outside. The odds of a courier being intercepted inside the fort are effectively zero. If your computational model includes a non-zero probability of failure "inside the fort" then you are…

No. An element of a distributed system that wants to debit from bank accounts must acknowledge the receipt of a message after making its effects durable. That system may fail between the change being durable and the acknowledgment making it back to the original sender.

That element must tolerate that debit being delivered multiple times. And we can't even solve that problem by a higher-order system providing "reliable delivery" on that element, unless that thing persists atomically with the application performing the transaction (or the application itself tolerates multiple delivery).

Re: Yes, you can have exactly-once delivery

#116

Post by mlyle » Sat, Sep 21, 2024, 12:09 AM UTC

lisper wrote: post #112

mlyle wrote: post #97

Earlier quoted context omitted.

The whole refusal to accept that a field could legitimately define something differently than how you prefer, and then running off to blog about it and name names... and then coming around for round II of flamewar... with ever more splitting of hairs in definitions... is not awesome. You are especially well-answered here, I think: https://news.ycombinator.com/item?id=41599131 One reason the delivery / processing dist…

> You are especially well-answered here I don't see anything there that is at odds with anything I have said. All I see there is a restatement of my position. > One reason the delivery / processing distinction exists because very often the application needs to atomically... Yes. Do you really think I did not already know that? > Thus, the application needs to deal with multiple delivery. That depends on your requirem…

> Yes. Do you really think I did not already know that?

Well, if the application and this mythical higher-level thing have to do things atomically and be tightly wed, but you're insistent on calling them different entities so that you can win an internet argument that the second one is not getting duplicate "deliveries" ... then that's honestly kind of sad.

The literature has used the term "delivery" like this basically 100% of the time for the past 20 years, and the majority of the time somewhere else. You can argue that your definition makes sense to you, but when everyone else uses the term the other way it's not helpful. Anyone can choose to define words differently from everyone else and then try to lawyer it out, but it's not likely to be useful or accepted.

Re: Yes, you can have exactly-once delivery

#117

Post by lisper » Sat, Sep 21, 2024, 1:07 AM UTC

mlyle wrote: post #116

lisper wrote: post #112

Earlier quoted context omitted.

> You are especially well-answered here I don't see anything there that is at odds with anything I have said. All I see there is a restatement of my position. > One reason the delivery / processing distinction exists because very often the application needs to atomically... Yes. Do you really think I did not already know that? > Thus, the application needs to deal with multiple delivery. That depends on your requirem…

> Yes. Do you really think I did not already know that? Well, if the application and this mythical higher-level thing have to do things atomically and be tightly wed, but you're insistent on calling them different entities so that you can win an internet argument that the second one is not getting duplicate "deliveries" ... then that's honestly kind of sad. The literature has used the term "delivery" like this basica…

> you're insistent on calling them different entities so that you can win an internet argument

No, I'm insistent on calling them different entities because in actual practice they can be, and indeed usually are, different entities. De-duping is usually done in the operating system, and applications usually run in user space.

Re: Yes, you can have exactly-once delivery

#118

Post by lisper » Sat, Sep 21, 2024, 1:08 AM UTC

mlyle wrote: post #115

lisper wrote: post #114

Earlier quoted context omitted.

> The guards have "processed" the message for this purpose, and the fact that they can deliver it to you over a perfectly reliable channel is moot. No, it isn't, because the situation inside the fort is different than the situation outside. The odds of a courier being intercepted inside the fort are effectively zero. If your computational model includes a non-zero probability of failure "inside the fort" then you are…

No. An element of a distributed system that wants to debit from bank accounts must acknowledge the receipt of a message after making its effects durable. That system may fail between the change being durable and the acknowledgment making it back to the original sender. That element must tolerate that debit being delivered multiple times. And we can't even solve that problem by a higher-order system providing "reliabl…

> An element of a distributed system that wants to debit from bank accounts must acknowledge the receipt of a message after making its effects durable.

I do not and never have disputed that. But I fail to see what that has to do with the matter at hand, namely, whether or not exactly-once delivery is possible.

Re: Yes, you can have exactly-once delivery

#119

Post by mlyle » Sat, Sep 21, 2024, 1:20 AM UTC

lisper wrote: post #117

mlyle wrote: post #116

Earlier quoted context omitted.

> Yes. Do you really think I did not already know that? Well, if the application and this mythical higher-level thing have to do things atomically and be tightly wed, but you're insistent on calling them different entities so that you can win an internet argument that the second one is not getting duplicate "deliveries" ... then that's honestly kind of sad. The literature has used the term "delivery" like this basica…

> you're insistent on calling them different entities so that you can win an internet argument No, I'm insistent on calling them different entities because in actual practice they can be, and indeed usually are, different entities. De-duping is usually done in the operating system, and applications usually run in user space.

No, as we're saying repeatedly at this point-- the application itself needs to tolerate multiple delivery, because if the deduplication isn't atomic with the application's actions, incorrect behavior results. Stacking on top of TCP doesn't fix this.

Re: Yes, you can have exactly-once delivery

#120

Post by lisper » Sat, Sep 21, 2024, 7:10 PM UTC

mlyle wrote: post #119

lisper wrote: post #117

Earlier quoted context omitted.

> you're insistent on calling them different entities so that you can win an internet argument No, I'm insistent on calling them different entities because in actual practice they can be, and indeed usually are, different entities. De-duping is usually done in the operating system, and applications usually run in user space.

No, as we're saying repeatedly at this point-- the application itself needs to tolerate multiple delivery, because if the deduplication isn't atomic with the application's actions, incorrect behavior results. Stacking on top of TCP doesn't fix this.

> if the deduplication isn't atomic with the application's actions, incorrect behavior results

So? What do the application requirements have to do with the question of whether or not exactly-once delivery is possible? The application is a red herring. Why do you keep bringing it up?

If you want to argue that exactly-once delivery is generally undesirable, that is not in dispute. What is in dispute is whether or not it is possible, and the application requirements cannot possibly have any bearing on that.