Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Yes, you can have exactly-once delivery

blog.rongarret.info

Post reply on HN
Print view View original

31–40 of 140 posts

Re: Yes, you can have exactly-once delivery

#31

Post by jongjong »

qaq wrote: post #12
jongjong wrote: post #7

Yes of course you can have exactly once delivery if you clearly define who/what the receiver is. You can easily deduplicate messages on the receiving end based on UUIDs created by the sender for example... Since they are duplicates of the exact same message, it doesn't matter which one is 'the original'. It makes sense to worry about this only if you're worried about wasting bandwidth in the event of network instabil…

you can't in general case because you don't have infinite memory for dedupe.

Good point but you don't need infinite memory. You can set expiries to discard, for example. Expiries and timeouts can be defined as part of the protocol. You can impose reasonable constraints per-socket on the buffer size for spam prevention. There is a lot of wiggle room.

On the sender side, you can require an ACK response for each message UUID and rebroadcast only if the ACK is not received within a certain timeframe.

You don't necessarily need a sophisticated algorithm to get a good practical solution which solves real problems.

Re: Yes, you can have exactly-once delivery

#32

Post by EGreg »

Groxx wrote: post #20
bhaney wrote: post #11

In case you're someone who actually knows anything about distributed systems and you're not looking forward to slogging through this long article filled with claims like "I have a PhD in AI so I know what I'm talking about" to find where the author made their mistake, let me save you the time. It's the typical conflation of exactly-once delivery with exactly-once processing, which the author acknowledges and then cho…

Yea... this is kinda inflammatory but I honestly have to agree with it. The post largely summarizes as "you can have exactly-once delivery if you re-define it to be at-least-once processing with idempotency". Those are different things. In fact, that's the entire point behind saying that it's impossible. You can't design a system that is exactly-once at any level, so don't even bother trying. If someone wants you to…

This is sorta like the argument about CAP impossibility theorem while in practice consensus algorithms work 99.999% of the time. Or like Shannon’s information theory showing impossibility of compression, while many compression algorithms work well on actual data.

This seems to me the same. In practical applications, you can indeed have the at-least-once delivery with an idempotency / backpressure system, and work 99% of the time, and be unavailable 1% of the time.

Re: Yes, you can have exactly-once delivery

#33

Post by mlyle »

lisper wrote: post #25
pyrolistical wrote: post #18

Earlier quoted context omitted.

Don’t use technical terms like “exactly-once delivery” if you don’t have want it to be interrupted as technical

It is hard to make the point that "exactly-once delivery" is not a technical term without referring to it. If you think it is a technical term, would you kindly point me to a definition? I'm particularly interested in learning how "exactly-once delivery" is distinguished from "exactly-once processing".

A 4 year old piece laying out the exact difference as it's understood and how people use the terms:

https://blog.bulloak.io/post/20200917-the-impossibility-of-e...

I read similar content at least 2 decades ago...

> While exactly-once-delivery is not possible, we have a way out: Exactly-once processing. Exactly-once processing is the guarantee that even though we may receive a message multiple times, in the end, we observe the effects of a single processing operation. This can be achieved in two ways:

> Deduplication: dropping messages if they are received more than once

> Idempotent processing: applying messages more than once has precisely the same effect as applying it exactly once

(I view deduplication as a special case of idempotency).

Re: Yes, you can have exactly-once delivery

#34

Post by Groxx »

EGreg wrote: post #32
Groxx wrote: post #20

Earlier quoted context omitted.

Yea... this is kinda inflammatory but I honestly have to agree with it. The post largely summarizes as "you can have exactly-once delivery if you re-define it to be at-least-once processing with idempotency". Those are different things. In fact, that's the entire point behind saying that it's impossible. You can't design a system that is exactly-once at any level, so don't even bother trying. If someone wants you to…

This is sorta like the argument about CAP impossibility theorem while in practice consensus algorithms work 99.999% of the time. Or like Shannon’s information theory showing impossibility of compression, while many compression algorithms work well on actual data. This seems to me the same. In practical applications, you can indeed have the at-least-once delivery with an idempotency / backpressure system, and work 99%…

Yep, and for practical applications (i.e. stuff that exists in this universe) that is absolutely good enough. You just have to choose which tradeoffs you can stomach best. With a fancy enough system, those tradeoffs can be driven shockingly low.

But if someone tries to sell you a database that is 100% available and has perfect consistency, you can laugh and walk away. They're a flat-earther trying to sell you a bridge: they're either trying to trick you or they have no idea what they're talking about. Either way you don't want to be involved with them.

Re: Yes, you can have exactly-once delivery

#35

Post by yew »

bhaney wrote: post #11

In case you're someone who actually knows anything about distributed systems and you're not looking forward to slogging through this long article filled with claims like "I have a PhD in AI so I know what I'm talking about" to find where the author made their mistake, let me save you the time. It's the typical conflation of exactly-once delivery with exactly-once processing, which the author acknowledges and then cho…

[flagged]

Re: Yes, you can have exactly-once delivery

#36

Post by Spivak »

bhaney wrote: post #11

In case you're someone who actually knows anything about distributed systems and you're not looking forward to slogging through this long article filled with claims like "I have a PhD in AI so I know what I'm talking about" to find where the author made their mistake, let me save you the time. It's the typical conflation of exactly-once delivery with exactly-once processing, which the author acknowledges and then cho…

This article isn't for you then, this article is for people who have casually heard that exactly once delivery is impossible and take it to mean exactly once processing is impossible. When someone talks about at-least-once and at-most-once in the context of well-known queueing systems they say will say delivery but will mean processing, because as you say, they're the same in practice.

You typically hide the processing bit so that from the perspective of your application code it really is exactly-once.

Re: Yes, you can have exactly-once delivery

#37

Post by theamk »

tacitusarc wrote: post #6

I appreciate the insights here, but I am struggling to understand how “exactly one” can equate to “eliminate duplicates”. Let’s say someone arrived at my house and cut my grass, and I failed to confirm they had done so, so the company sent someone over to cut my grass again, maybe multiple times. It seems silly to claim my grass was cut exactly once, despite it consistently remaining at the same height. Obviously it…

We are talking network stack, so there is no actions - just data hand-off to the actual application code.

Someone arrives at your house, gives you a package, says "this is order 123". You thank them, they leave, but then they are hit by a car before they can report this. You unpack the package and use it.

Next day, someone else arrives at your house, gives you a package, says "this is order 123". You thank them, they leave. You know you've already received order 123, so you throw package away without even taking it into the house.

This happens few more times, but you don't care, your trash can is big.

Done! You now have "exactly once delivery".

Now, some might argue this is "exactly once processing" and you should only count what the delivery person does.. but this depends on where you draw the boundary. I draw it at "I am taking the package into the house", and I've only ever took one package there, so it was exactly-once for me.

The key part here is cost. I am assuming that opening package and using its contents is hard and takes a long time; while answering the door and throwing the package away is easy. This is definitely the case with modern networking stack, which re-transmits stuff all the time, and where the loss rate is very low.

Re: Yes, you can have exactly-once delivery

#38

Post by theamk »

hinkley wrote: post #29
supportengineer wrote: post #2

Over 20 years ago I worked with some specialized commercial software, Cyclone, that did guaranteed delivery of files. Guaranteed as in the legal sense. If the server sent back a ticket number to the sending client, it was a LEGAL assurance that the file was received, because there was a contract in place with financial penalties. The time stamps were particularly important in the legal contract. So, there are a lot o…

But you sometimes delivered the bytes twice, before getting the receipt, surely?

I am sure there were some duplicate packets on the wire - it's a normal part of most network protocols. The important thing is that as far as user was concerned, it was exactly-once.

Re: Yes, you can have exactly-once delivery

#39

Post by bhaney »

Spivak wrote: post #36
bhaney wrote: post #11

In case you're someone who actually knows anything about distributed systems and you're not looking forward to slogging through this long article filled with claims like "I have a PhD in AI so I know what I'm talking about" to find where the author made their mistake, let me save you the time. It's the typical conflation of exactly-once delivery with exactly-once processing, which the author acknowledges and then cho…

This article isn't for you then, this article is for people who have casually heard that exactly once delivery is impossible and take it to mean exactly once processing is impossible. When someone talks about at-least-once and at-most-once in the context of well-known queueing systems they say will say delivery but will mean processing, because as you say, they're the same in practice. You typically hide the processi…

> this article is for people who have casually heard that exactly once delivery is impossible and take it to mean exactly once processing is impossible

Those people would be better served by approximately two sentences clarifying that exactly-once processing is a different thing that can be achieved with at-least-once delivery and idempotency, rather than 20+ rambling paragraphs of redefining formal terms.

Re: Yes, you can have exactly-once delivery

#40

Post by tacitusarc »

lisper wrote: post #14

Author here. Most commenters still seem to be missing the point, despite the fact that I explicitly say this in the opening sentence: "This post is ostensibly about an obscure technical issue in distributed systems, but it's really about human communications." and reiterate it at the end: "This post was intended to be about human communication more than distributed systems or network protocols." I really don't know h…

Right, but the meaning of “delivery” in human communication is still different from how you are using it…
Post reply on HN