Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Yes, you can have exactly-once delivery

blog.rongarret.info

Post reply on HN
Print view View original

81–90 of 140 posts

Re: Yes, you can have exactly-once delivery

#81

Post by jbergens »

tacitusarc wrote: post #58
theamk wrote: post #37

Earlier quoted context omitted.

We are talking network stack, so there is no actions - just data hand-off to the actual application code. Someone arrives at your house, gives you a package, says "this is order 123". You thank them, they leave, but then they are hit by a car before they can report this. You unpack the package and use it. Next day, someone else arrives at your house, gives you a package, says "this is order 123". You thank them, they…

As this is a semantic debate over the definition of delivery, I asked my very non-technical wife if she thought in the scenario you described, the package was delivered exactly once. She said obviously not, and this discussion is very stupid, and I should stop participating in it. So there’s that.

I don't think the example was perfect which explains your wife's reaction.

Think of it more like the first delivery guy/girl left his/her car outside and wrote 123 on it. Then walked back.

The next one sees the car with a sign saying 123 and won't even ring the door bell or leave a package. Now you haven't gotten the package twice, it has not been delivered twice.

Sure you can complain that there's a car outside your home but in digital system you won't even see it. It would also cost the deliver firm a car for every package but that is not your problem and again, in the digital world the cost is a lot less than a car.

There is an argument that the street would be filled up with delivery vans ans there would be no more room for new deliveries to you or your neighbors but that is a limitation you could talk about. You probably can't handle an infinite number of packages delivered at the same time either and you won't wait an infinite amount of time for any specific package.

Try this version with your wife.

Re: Yes, you can have exactly-once delivery

#82

Post by stackghost »

lisper wrote: post #78
stackghost wrote: post #75

Earlier quoted context omitted.

>> if you have received duplicate messages then by definition you have been delivered that message more than once >Yes. Okay, I'm glad we agree that delivering something multiple times and discarding the duplicates is not exactly-once delivery.

As you have described it, you are right, it's not. But presumably the thing discarding the duplicates then does something with the non-duplicates, like deliver then to something else. That is exactly-once delivery.

You wrote in the post:

>This post was intended to be about human communication more than distributed systems or network protocols

But resorting to technical minutiae as you have done doesn't seem germane to "human communication". Honestly, seeing such mental gymnastics just to avoid losing face on an internet argument is really disappointing.

Re: Yes, you can have exactly-once delivery

#83

Post by dragonwriter »

lisper wrote: post #61
dragonwriter wrote: post #43

Earlier quoted context omitted.

Draw a diagram containing a source system, a destination system, and an unreliable communication channel between them. The destination system also has an output with no unreliable communications channel. Exactly-once delivery means that a message sent from the source system reaches the destination system exactly once, and its result reaches the output channel exactly once as a consequence. Excactly-once processing me…

> That's a little sloppy Yes, that is exactly my point. The only way you can make it non-sloppy is to define "delivery" as being something that happens exclusively upstream of deduping.

No, I'm saying its "sloppy" as a definition because while it addresses the distinction you ask about it, it doesn't fully cover what distinguishes exactly-once from at-most-once.

> The only way you can make it non-sloppy is to define "delivery" as being something that happens exclusively upstream of deduping.

"Deduping" can happen in many places. If it happens anywhere before the destination system end of the unreliable connection it is part of delivery (but also can't get you to exactly-once delivery). If it happens on the destination side of the unreliable communication channel, then yes, it's not part of the delivery guarantee, it is how you get exactly-once processing from at-least-once delivery. This has been well-known for a very long time. (I don't think it was new when I first encountered it in 1999.)

Re: Yes, you can have exactly-once delivery

#84

Post by brigadier132 »

lisper wrote: post #68
brigadier132 wrote: post #22

It seems like this person does not understand what exactly once delivery actually means. It's not about being able to do something exactly once if the author is reading this...

> It seems like this person does not understand what exactly once delivery actually means. Then by all means, enlighten me.

I don't want to get into a debate over semantics with you. You've redefined exactly once delivery to mean that if you have a layer of abstraction that hides that a message was delivered multiple times then it is "exactly once" delivery. That's not what exactly once delivery is. You clearly disagree but taking well understood and researched terms and redefining them to make a point is a complete waste of time for everyone. Nobody cares about what your personal definition of exactly once delivery is, they care about the definition that is used in literally all of distributed systems research.

Re: Yes, you can have exactly-once delivery

#85

Post by lisper »

upeguiborja wrote: post #79
lisper wrote: post #69

Earlier quoted context omitted.

Yes, that's true, but that's often a reasonable assumption.

The main confusion I see is that at-least-once delivery on a 99% reliable channel can be erroneusly called exactly once delivery 99 out of 100 times. It could be reasonable depending on the scale of your communication exercise, but if you have hundreds of thousands of receivers then asumption will surely start to fail.

The kind of reliability I'm talking about where "100% reliable" is a reasonable assumption is inside the boundaries of a single server, i.e. a CPU, memory bus, and I/O peripherals including a NIC and non-volatile storage. Yes, things can and do fail inside that boundary, but if you start taking that into account things get much more complicated. You are also no longer in the realm of distributed systems at that point, which is the context in which the controversy about exactly-once delivery arises.

Re: Yes, you can have exactly-once delivery

#86

Post by lisper »

brigadier132 wrote: post #84
lisper wrote: post #68

Earlier quoted context omitted.

> It seems like this person does not understand what exactly once delivery actually means. Then by all means, enlighten me.

I don't want to get into a debate over semantics with you. You've redefined exactly once delivery to mean that if you have a layer of abstraction that hides that a message was delivered multiple times then it is "exactly once" delivery. That's not what exactly once delivery is. You clearly disagree but taking well understood and researched terms and redefining them to make a point is a complete waste of time for ever…

I'm not asking for a debate. I'm simply asking you to tell me what "exactly once delivery" actually means in a way that isn't circular and doesn't beg the question of whether or not it is possible, i.e. the definition that is used in literally all of distributed systems research. So far no one has been able to do that.

Re: Yes, you can have exactly-once delivery

#88

Post by mlyle »

lisper wrote: post #56
mlyle wrote: post #33

Earlier quoted context omitted.

A 4 year old piece laying out the exact difference as it's understood and how people use the terms: https://blog.bulloak.io/post/20200917-the-impossibility-of-e... I read similar content at least 2 decades ago... > While exactly-once-delivery is not possible, we have a way out: Exactly-once processing. Exactly-once processing is the guarantee that even though we may receive a message multiple times, in the end, we ob…

> A 4 year old piece laying out the exact difference... "Exactly-once delivery guarantee is the guarantee that a message can be delivered to a recipient once, and only once." That seems circular to me. Also, the author's proof is flawed. The 2GP requires more than exactly-once delivery, it requires common knowledge . It is not enough for the first general to know that the message will be received, it is required that…

The piece makes a very simple distinction.

Delivery is the property of a message showing up at a receiver, irrespective of the receiver making state changes.

Processing is making state changes.

You can't dedupe messages without some kind of state change. Your guards, writing down that a given message has been here before, have been delivered the message. An endpoint on a lossy medium has to cope with either (0 or 1) or (1 or many) messages.

Now, can the guards deliver it to you at most once? Well, if there's no lossy medium between, sure. But we already know that we can deliver exactly once when the medium is perfectly reliable. The guards have "processed" the message for this purpose, and the fact that they can deliver it to you over a perfectly reliable channel is moot.

The distinction is between the characteristics of the channel (delivery) versus what the receiver must do to achieve appropriate processing properties.

These might come from some combination of intrinsic idempotency, timers, persisting past messages to disks, establishing a message ordering, etc, etc, etc. These are the mechanisms that you need to cope with "one or many" delivery, and they all shape the state model of your system with respect to messages.

Re: Yes, you can have exactly-once delivery

#89

Post by lisper »

mlyle wrote: post #80
lisper wrote: post #52

Earlier quoted context omitted.

Yes, that is true. But why can't I choose to view "the system itself" as the thing that is on the other side of a de-duplicator? It feels to me like an argument over whether or not humans can fly. An unassisted human cannot fly, but with some technological augmentation, they can. It seems a bit pedantic to deny that someone can fly from LA to New York simply because they have to get into an airplane to do it.

Having a clear system boundary is required for analysis. Knowing that messages will be delivered 1+ times gives us a variety of ways we could choose to deal with this on the endpoint, with different vulnerabilities. (Getting "exactly once" processing usually requires making various kinds of resilience tradeoffs based on timing windows, storage requirements, etc). > It seems a bit pedantic to deny that someone can fly…

> You're calling people out by name

I am? Where?

> Getting "exactly once" processing usually requires making various kinds of resilience tradeoffs based on timing windows, storage requirements, etc

Yes, of course. But that's not the same as "impossible".

Re: Yes, you can have exactly-once delivery

#90

Post by mlyle »

lisper wrote: post #78
stackghost wrote: post #75

Earlier quoted context omitted.

>> if you have received duplicate messages then by definition you have been delivered that message more than once >Yes. Okay, I'm glad we agree that delivering something multiple times and discarding the duplicates is not exactly-once delivery.

As you have described it, you are right, it's not. But presumably the thing discarding the duplicates then does something with the non-duplicates, like deliver then to something else. That is exactly-once delivery.

To discard the duplicates, you need to update your state based on the first message and consult that state for all subsequent messages. Someone must be delivered those messages for this to happen.

If you say that someone can then pass you messages exactly once over a perfectly reliable channel, well.. sure. But the system still has to deal with duplicate delivery and update its state based on that. (And potentially must durably persist this state. And, then, if your application doesn't have exactly the same persistence/transactional boundaries as what's doing the deduping, you have problems...).

Post reply on HN