Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Yes, you can have exactly-once delivery

blog.rongarret.info

Post reply on HN
Print view View original

61–70 of 140 posts

Re: Yes, you can have exactly-once delivery

#61

Post by lisper »

dragonwriter wrote: post #43
lisper wrote: post #25

Earlier quoted context omitted.

It is hard to make the point that "exactly-once delivery" is not a technical term without referring to it. If you think it is a technical term, would you kindly point me to a definition? I'm particularly interested in learning how "exactly-once delivery" is distinguished from "exactly-once processing".

Draw a diagram containing a source system, a destination system, and an unreliable communication channel between them. The destination system also has an output with no unreliable communications channel. Exactly-once delivery means that a message sent from the source system reaches the destination system exactly once, and its result reaches the output channel exactly once as a consequence. Excactly-once processing me…

> That's a little sloppy

Yes, that is exactly my point. The only way you can make it non-sloppy is to define "delivery" as being something that happens exclusively upstream of deduping.

Re: Yes, you can have exactly-once delivery

#62

Post by lisper »

srkiNZ wrote: post #49
lisper wrote: post #42

Earlier quoted context omitted.

> If you have duplicate things, then you've clearly been delivered more than one thing. Yes, that's true. But this doesn't turn on what "delivery" means, it turns on what "you" means. If "you" are downstream of a de-duplication mechanism, then "you" can get exactly once-delivery. Why is that so absurd?

If "you" are "downstream" of a "de-duplication mechanism" how do you ensure "exactly once delivery" between "you" and the "de-duplication mechanism"?

The same way I ensure any behavior in a digital system. There are boundaries inside of which processes are presumed to be reliable, typically consisting of a CPU, memory busses, and attached non-volatile storage. If you don't assume that those are reliable, then you can't guarantee anything.

Re: Yes, you can have exactly-once delivery

#63

Post by lisper »

computerfan494 wrote: post #44
lisper wrote: post #42

Earlier quoted context omitted.

> If you have duplicate things, then you've clearly been delivered more than one thing. Yes, that's true. But this doesn't turn on what "delivery" means, it turns on what "you" means. If "you" are downstream of a de-duplication mechanism, then "you" can get exactly once-delivery. Why is that so absurd?

You're correct, but in my experience the vast majority of code written is not downstream of a usable de-duplication mechanism.

That may well be, but that's a very different question.

Re: Yes, you can have exactly-once delivery

#64

Post by lisper »

two_handfuls wrote: post #50
lisper wrote: post #19

Earlier quoted context omitted.

You apparently missed this part: "Just for the sake of completeness I should point out that removing duplicates at the receiver is a pretty extreme oversimplification of what you would do in practice to provide exactly-once delivery. A complete solution would almost certainly be an example of Greenspun's Tenth Law applied to the TCP protocol rather than Common Lisp."

Can you explain this point? I think you are trying to say the application would include an implementation of TCP? I don't see how this is related to memory.

I was responding to this:

> You can't get exactly once delivery using proposed solution in general case because you can not have infinite memory for dedupe.

That is a specific criticism of a specific and highly oversimplified algorithm for de-duplication that I described purely for illustrative purposes, not as a suggestion for how de-duplication should actually be implemented. Actual implementation of de-duplication is much more complicated, but a solved problem that I didn't think I needed to rehash. A real implementation doesn't require infinite memory.

(Actually, even the naive algorithm doesn't require infinite memory, just unbounded memory. Those are not the same.)

Re: Yes, you can have exactly-once delivery

#65

Post by lisper »

tacitusarc wrote: post #40
lisper wrote: post #14

Author here. Most commenters still seem to be missing the point, despite the fact that I explicitly say this in the opening sentence: "This post is ostensibly about an obscure technical issue in distributed systems, but it's really about human communications." and reiterate it at the end: "This post was intended to be about human communication more than distributed systems or network protocols." I really don't know h…

Right, but the meaning of “delivery” in human communication is still different from how you are using it…

I disagree. If I'm an actual general in a fort with a gate, and I tell the gate guards to inspect messages brought in by couriers and not allow couriers carrying duplicate messages to enter, the ones that the guards let through are still delivering those messages to me.

Re: Yes, you can have exactly-once delivery

#66

Post by stackghost »

lisper wrote: post #42
stackghost wrote: post #24

Earlier quoted context omitted.

Your proposed definition of "delivery" is absurd. If you have duplicate things, then you've clearly been delivered more than one thing. There is no way to deliver something exactly once, and yet the receiver has more than one thing such that they can throw all but one thing away. It's okay to admit you were mistaken.

> If you have duplicate things, then you've clearly been delivered more than one thing. Yes, that's true. But this doesn't turn on what "delivery" means, it turns on what "you" means. If "you" are downstream of a de-duplication mechanism, then "you" can get exactly once-delivery. Why is that so absurd?

>Yes, that's true. But this doesn't turn on what "delivery" means, it turns on what "you" means. If "you" are downstream of a de-duplication mechanism, then "you" can get exactly once-delivery. Why is that so absurd?

So in the case of, say, a network service on server A and a network client B, your solution to "exactly once delivery" is to re-define it as "deliver it from A to B multiple times and have B deduplicate"?

Do you not see how nonsensical that is to call that "exactly once delivery"?

Re: Yes, you can have exactly-once delivery

#67

Post by jumploops »

If I receive multiple hamburgers via DoorDash, but only eat one, it’s not “exactly-once delivery.”

The extra step where I give my neighbor(s), compost, or otherwise discard the N+1 hamburgers is a processing step.

My house can only hold so many hamburgers, and I can only process so many after eating my lucky chosen one.

This is what we (distributed systems thinkers) refer to when we say “delivery” — anything after the DoorDash step is up to us, the consumer, to process.

Yes, this definition is confusing to new programmers, because it makes it hard to reason about everyday systems, but it’s this exact type of definition that we need so that we can build the proper abstractions, as the author has done in his post, to make our applications behave the way we want.

Re: Yes, you can have exactly-once delivery

#68

Post by lisper »

brigadier132 wrote: post #22

It seems like this person does not understand what exactly once delivery actually means. It's not about being able to do something exactly once if the author is reading this...

> It seems like this person does not understand what exactly once delivery actually means.

Then by all means, enlighten me.

Re: Yes, you can have exactly-once delivery

#70

Post by computerfan494 »

lisper wrote: post #63
computerfan494 wrote: post #44

Earlier quoted context omitted.

You're correct, but in my experience the vast majority of code written is not downstream of a usable de-duplication mechanism.

That may well be, but that's a very different question.

It explains why most people seem to disagree with you on what "delivery" and "you" mean in this context. For the majority of contexts, "delivery" means that a system responsible for de-duplication receives the message.
Post reply on HN