Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Yes, you can have exactly-once delivery

blog.rongarret.info

Post reply on HN
Print view View original

71–80 of 140 posts

Re: Yes, you can have exactly-once delivery

#71

Post by srkiNZ »

lisper wrote: post #62
srkiNZ wrote: post #49

Earlier quoted context omitted.

If "you" are "downstream" of a "de-duplication mechanism" how do you ensure "exactly once delivery" between "you" and the "de-duplication mechanism"?

The same way I ensure any behavior in a digital system. There are boundaries inside of which processes are presumed to be reliable, typically consisting of a CPU, memory busses, and attached non-volatile storage. If you don't assume that those are reliable, then you can't guarantee anything.

Great! I agree 100%. We have to assume a "reliable network" within a "boundary" (i.e. a computers CPU, memory, busses etc...). Distributed systems (from which these rules are taken) are specifically systems where anything within one of these "boundaries" is considered a "single node" and treated the same, whether it's a NIC, a kernel module/driver, a user space process or anything else.

In our case if we were to take (for example) that the NIC would de-duplicate the messages for us, anyone writing the producer/sender and a user-space program for the receiver a would need to know that the NIC was doing this (or risk having messages dropped for failure of including a unique id).

This is a pedantic point, but I would strongly stress that the only reason these "delivery rules" are so popular (and evoke such a reaction) is because of the very large number of times that programmers mis-understand them.

Commonly they either assume that:

* the network is reliable

* something else will guarantee "exactly once delivery" for me (when in fact nothing will)

Re: Yes, you can have exactly-once delivery

#72

Post by nhumrich »

jongjong wrote: post #7

Yes of course you can have exactly once delivery if you clearly define who/what the receiver is. You can easily deduplicate messages on the receiving end based on UUIDs created by the sender for example... Since they are duplicates of the exact same message, it doesn't matter which one is 'the original'. It makes sense to worry about this only if you're worried about wasting bandwidth in the event of network instabil…

You seem to be missing the point though. Wanting "exactly once delivery" in practice is like saying, "I shouldn't need to worry about dedupes at the application level". Which is everyone's dream, but the seniors are saying, "yes, you have to handle it at the application level, there is no way around that".

Re: Yes, you can have exactly-once delivery

#73

Post by lisper »

stackghost wrote: post #21

>a simple matter of keeping track of all the delivered messages and removing duplicates Ron, if you have received duplicate messages then by definition you have been delivered that message more than once. I don't have a PhD in computer science so maybe you can explain how this constitutes "exactly once".

> if you have received duplicate messages then by definition you have been delivered that message more than once

Yes. So? If you don't include a mechanism for ensuring exactly-once delivery then you will not be guaranteed exactly-once delivery. But if you do, you will. It doesn't actually require a Ph.D.

Re: Yes, you can have exactly-once delivery

#74

Post by bigstrat2003 »

lisper wrote: post #14

Author here. Most commenters still seem to be missing the point, despite the fact that I explicitly say this in the opening sentence: "This post is ostensibly about an obscure technical issue in distributed systems, but it's really about human communications." and reiterate it at the end: "This post was intended to be about human communication more than distributed systems or network protocols." I really don't know h…

I'm sorry people are being such utter jerks about this. Honestly, the comments on this are an embarrassment. Even if you are wrong, there's absolutely no call for the tone a lot of the comments are taking.

Re: Yes, you can have exactly-once delivery

#75

Post by stackghost »

lisper wrote: post #73
stackghost wrote: post #21

>a simple matter of keeping track of all the delivered messages and removing duplicates Ron, if you have received duplicate messages then by definition you have been delivered that message more than once. I don't have a PhD in computer science so maybe you can explain how this constitutes "exactly once".

> if you have received duplicate messages then by definition you have been delivered that message more than once Yes. So? If you don't include a mechanism for ensuring exactly-once delivery then you will not be guaranteed exactly-once delivery. But if you do, you will. It doesn't actually require a Ph.D.

>> if you have received duplicate messages then by definition you have been delivered that message more than once

>Yes.

Okay, I'm glad we agree that delivering something multiple times and discarding the duplicates is not exactly-once delivery.

Re: Yes, you can have exactly-once delivery

#76

Post by lisper »

computerfan494 wrote: post #70
lisper wrote: post #63

Earlier quoted context omitted.

That may well be, but that's a very different question.

It explains why most people seem to disagree with you on what "delivery" and "you" mean in this context. For the majority of contexts, "delivery" means that a system responsible for de-duplication receives the message.

Yeah, but that just seems like a bizarre definition on which to base the claim that you cannot have exactly-once delivery. Obviously, if you define delivery to preclude de-duplication, then you can't have exactly-once delivery. But you can have something that delivers messages (for some reasonable definition of "delivers") exactly once. It seems weird to define delivery in such a way that such a system does not provide exactly-once delivery.

Re: Yes, you can have exactly-once delivery

#77

Post by lisper »

bigstrat2003 wrote: post #74
lisper wrote: post #14

Author here. Most commenters still seem to be missing the point, despite the fact that I explicitly say this in the opening sentence: "This post is ostensibly about an obscure technical issue in distributed systems, but it's really about human communications." and reiterate it at the end: "This post was intended to be about human communication more than distributed systems or network protocols." I really don't know h…

I'm sorry people are being such utter jerks about this. Honestly, the comments on this are an embarrassment. Even if you are wrong, there's absolutely no call for the tone a lot of the comments are taking.

Thanks.

Re: Yes, you can have exactly-once delivery

#78

Post by lisper »

stackghost wrote: post #75
lisper wrote: post #73

Earlier quoted context omitted.

> if you have received duplicate messages then by definition you have been delivered that message more than once Yes. So? If you don't include a mechanism for ensuring exactly-once delivery then you will not be guaranteed exactly-once delivery. But if you do, you will. It doesn't actually require a Ph.D.

>> if you have received duplicate messages then by definition you have been delivered that message more than once >Yes. Okay, I'm glad we agree that delivering something multiple times and discarding the duplicates is not exactly-once delivery.

As you have described it, you are right, it's not. But presumably the thing discarding the duplicates then does something with the non-duplicates, like deliver then to something else. That is exactly-once delivery.

Re: Yes, you can have exactly-once delivery

#79

Post by upeguiborja »

lisper wrote: post #69
upeguiborja wrote: post #57

It is only possible if the channel between your "abstraction layer" (call it another system, processing, etc) and your receiver is 100% reliable.

Yes, that's true, but that's often a reasonable assumption.

The main confusion I see is that at-least-once delivery on a 99% reliable channel can be erroneusly called exactly once delivery 99 out of 100 times. It could be reasonable depending on the scale of your communication exercise, but if you have hundreds of thousands of receivers then asumption will surely start to fail.

Re: Yes, you can have exactly-once delivery

#80

Post by mlyle »

lisper wrote: post #52
mlyle wrote: post #27

Earlier quoted context omitted.

The thing that is at the end of the lossy medium. It must tolerate (0 or 1) or (1 or more) things being delivered to it.

Yes, that is true. But why can't I choose to view "the system itself" as the thing that is on the other side of a de-duplicator? It feels to me like an argument over whether or not humans can fly. An unassisted human cannot fly, but with some technological augmentation, they can. It seems a bit pedantic to deny that someone can fly from LA to New York simply because they have to get into an airplane to do it.

Having a clear system boundary is required for analysis.

Knowing that messages will be delivered 1+ times gives us a variety of ways we could choose to deal with this on the endpoint, with different vulnerabilities. (Getting "exactly once" processing usually requires making various kinds of resilience tradeoffs based on timing windows, storage requirements, etc).

> It seems a bit pedantic to deny that someone can fly from LA to New York

At this point I question your good faith. You're calling people out by name, and you're going full on "well, aktuallyyyy" and seeming to deliberately misunderstand other peoples' assertions. "People can't breathe underwater" v. "Well, once I was in a tunnel that was under a body of water, and I still breathed!(@!("

If you choose to define words differently than everyone else, you're just sabotaging your own communication to try and feel smart.

Post reply on HN