Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Yes, you can have exactly-once delivery

blog.rongarret.info

Post reply on HN
Print view View original

101–110 of 140 posts

Re: Yes, you can have exactly-once delivery

#101

Post by Filligree »

lisper wrote: post #86
brigadier132 wrote: post #84

Earlier quoted context omitted.

I don't want to get into a debate over semantics with you. You've redefined exactly once delivery to mean that if you have a layer of abstraction that hides that a message was delivered multiple times then it is "exactly once" delivery. That's not what exactly once delivery is. You clearly disagree but taking well understood and researched terms and redefining them to make a point is a complete waste of time for ever…

I'm not asking for a debate. I'm simply asking you to tell me what "exactly once delivery" actually means in a way that isn't circular and doesn't beg the question of whether or not it is possible, i.e. the definition that is used in literally all of distributed systems research. So far no one has been able to do that.

Delivery is what happens when the signals cross from the wire into the network interface card.

Depending on the layer you’re operating on, you might instead say it’s the call to recv, or the DMA transfer. The point is that it’s logically a memcpy with no further processing. Just a memcpy. The physical data transfer.

It’s easy to build a reliable message-passing system on top of that, but any such system will either involve further processing or else be vulnerable to data loss.

Re: Yes, you can have exactly-once delivery

#102

Post by brigadier132 »

lisper wrote: post #86
brigadier132 wrote: post #84

Earlier quoted context omitted.

I don't want to get into a debate over semantics with you. You've redefined exactly once delivery to mean that if you have a layer of abstraction that hides that a message was delivered multiple times then it is "exactly once" delivery. That's not what exactly once delivery is. You clearly disagree but taking well understood and researched terms and redefining them to make a point is a complete waste of time for ever…

I'm not asking for a debate. I'm simply asking you to tell me what "exactly once delivery" actually means in a way that isn't circular and doesn't beg the question of whether or not it is possible, i.e. the definition that is used in literally all of distributed systems research. So far no one has been able to do that.

Exactly once delivery is when a message is delivered to any computer at any layer of abstraction exactly once.

It doesn't matter if TCP abstracts or any other layer abstracts this away, if a message can be delivered more than once it does not have the property of exactly once delivery.

You might disagree with this definition but you would be wrong. There is negative value in redefining exactly once delivery to mean what you think it means.

Re: Yes, you can have exactly-once delivery

#103

Post by [deleted] »

jbergens wrote: post #81
tacitusarc wrote: post #58

Earlier quoted context omitted.

As this is a semantic debate over the definition of delivery, I asked my very non-technical wife if she thought in the scenario you described, the package was delivered exactly once. She said obviously not, and this discussion is very stupid, and I should stop participating in it. So there’s that.

I don't think the example was perfect which explains your wife's reaction. Think of it more like the first delivery guy/girl left his/her car outside and wrote 123 on it. Then walked back. The next one sees the car with a sign saying 123 and won't even ring the door bell or leave a package. Now you haven't gotten the package twice, it has not been delivered twice. Sure you can complain that there's a car outside your…

[deleted]

Re: Yes, you can have exactly-once delivery

#104

Post by two_handfuls »

Fire-Dragon-DoL wrote: post #100

Isn't "exactly once processing" also incorrect, since it's all always "at most once": the system could go down and never come back online, that would result in a missed processing

All these guarantees are of the form "if (...) then: exactly once processing."

You'll find that, indeed, those assumptions must include ruling out permanent link failures.

Re: Yes, you can have exactly-once delivery

#105

Post by computerfan494 »

lisper wrote: post #76
computerfan494 wrote: post #70

Earlier quoted context omitted.

It explains why most people seem to disagree with you on what "delivery" and "you" mean in this context. For the majority of contexts, "delivery" means that a system responsible for de-duplication receives the message.

Yeah, but that just seems like a bizarre definition on which to base the claim that you cannot have exactly-once delivery. Obviously, if you define delivery to preclude de-duplication, then you can't have exactly-once delivery. But you can have something that delivers messages (for some reasonable definition of "delivers") exactly once. It seems weird to define delivery in such a way that such a system does not provi…

I'm not sure how to convince you to get over your hangup around how other people feel about the word "delivery", but this is exactly why others distinguish between "delivery" and "processing". If you think there are better terms than those two to describe "the system that receives a message and must be responsible for de-duplication" and "the system that can rely on messages being already de-duplicated", then feel free to propose them and have people debate, that I suppose. But because of what I noted earlier, this is a very useful distinction to maintain for people working on systems that are responsible for de-duplication (likely most people on this forum), and these words seem to make sense for most individuals.

Re: Yes, you can have exactly-once delivery

#106

Post by [deleted] »

lisper wrote: post #52
mlyle wrote: post #27

Earlier quoted context omitted.

The thing that is at the end of the lossy medium. It must tolerate (0 or 1) or (1 or more) things being delivered to it.

Yes, that is true. But why can't I choose to view "the system itself" as the thing that is on the other side of a de-duplicator? It feels to me like an argument over whether or not humans can fly. An unassisted human cannot fly, but with some technological augmentation, they can. It seems a bit pedantic to deny that someone can fly from LA to New York simply because they have to get into an airplane to do it.

[deleted]

Re: Yes, you can have exactly-once delivery

#107

Post by gbonik »

So it seems it boils down to the difference between "delivery" and "processing".

I think we can make this distinction formally. For a given communication channel C, we can define "delivery via C" as a message showing up successfully at the receiver end of C. This definition seems unambiguous.

Now, we can phrase our "theorem" more carefully:

    For an arbitrary given communication channel C, exactly-once delivery via C is generally impossible.
The important part here is "For an arbitrary given communication channel C". By adding a layer of deduplication on top of C, we would be constructing a new logical communication channel C', via which exactly-once delivery is indeed possible. But that would be a different channel C', not the original channel C that we were given. In this context, we can refer to delivery via C' as "processing" relative to the original channel C.

Re: Yes, you can have exactly-once delivery

#108

Post by lisper »

qaq wrote: post #98
lisper wrote: post #64

Earlier quoted context omitted.

I was responding to this: > You can't get exactly once delivery using proposed solution in general case because you can not have infinite memory for dedupe. That is a specific criticism of a specific and highly oversimplified algorithm for de-duplication that I described purely for illustrative purposes, not as a suggestion for how de-duplication should actually be implemented. Actual implementation of de-duplication…

You certainly do need to rehash it or provide a link to it being solved without constraints on time window, msgs rates and msgs sizes. You also most certainly want to build it and provide it as service because no existing product claims this capability in unbounded case.

> solved without constraints on time window, msgs rates and msgs sizes

But that is not what I am claiming.

Re: Yes, you can have exactly-once delivery

#109

Post by lisper »

Jach wrote: post #99
lisper wrote: post #14

Author here. Most commenters still seem to be missing the point, despite the fact that I explicitly say this in the opening sentence: "This post is ostensibly about an obscure technical issue in distributed systems, but it's really about human communications." and reiterate it at the end: "This post was intended to be about human communication more than distributed systems or network protocols." I really don't know h…

This has been a fun thread to read, though not reflecting highly on HN. For what it's worth, I agree with you that the original adage kicking this off is kind of silly, and basically wrong. For further enjoyment I found this interesting blog post from another PhD that addresses things more comprehensively (and also basically agrees with you): https://www.mydistributed.systems/2021/10/exactly-once-deliv... The opening…

Thank you for that link! You just saved me a lot of time reinventing that wheel.

Re: Yes, you can have exactly-once delivery

#110

Post by Izkata »

jchw wrote: post #41
bhaney wrote: post #11

In case you're someone who actually knows anything about distributed systems and you're not looking forward to slogging through this long article filled with claims like "I have a PhD in AI so I know what I'm talking about" to find where the author made their mistake, let me save you the time. It's the typical conflation of exactly-once delivery with exactly-once processing, which the author acknowledges and then cho…

Honestly, I really do find the traditional nomenclature to be a little pointless. It seems like the classic saying assumes that it's somehow okay to assume infinite time for re-delivery, but not infinite memory for memoization for some reason. On the other hand, in real life there aren't unlimited numbers of messages and you rarely want to accept infinitely stale messages either, so it's a bit moot. I'd go as far as…

> Honestly, I really do find the traditional nomenclature to be a little pointless. It seems like the classic saying assumes that it's somehow okay to assume infinite time for re-delivery, but not infinite memory for memoization for some reason.

This is exactly where the argument is coming from. The same people who will say "you can get at most once or at least once, but not only once" don't realize they're doing the exact same thing as the "you can get only once" people, when they criticize the conflation of delivery and processing. They'll argue "delivery" and "processing" have to be kept separate because of memory/storage/bandwidth/etc it uses up in the retries, which is why "only once delivery" can't exist and they actually mean "only once processing", but if you keep that reasoning in mind, there's also no such thing as "at least once delivery" - you'll run out of something at some point (or even just hit your retry limit) and have to drop the retries, resulting in no delivery.

The people saying you can get "only once delivery" by using "at least once"+idempotency are working under other group's definitions, then getting annoyed when the definitions are changed so this implementation of "only once" isn't allowed.

Post reply on HN