Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Yes, you can have exactly-once delivery

blog.rongarret.info

Post reply on HN
Print view View original

51–60 of 140 posts

Re: Yes, you can have exactly-once delivery

#51

Post by deathanatos »

Filligree wrote: post #5
toast0 wrote: post #4

I don't see how you can guarantee at least one delivery, but maybe I've seen too many things disappear off busses, never to be seen again.

You keep sending it until you get an acknowledgement back. As the article points out, this does assume nobody cut the wire.

  SEND -->
  SEND -->
Don't try to "fix" it, either; there's no way to do that.

You make the job idempotent, in some manner: make it such that receiving the message and processing it twice is safe.

(This is what TFA eventually capitulates to, but tries to call it "exactly once delivery", even in cases where deliver is occurring more than once. I don't think this view is pedagogically useful, which is partly why we say "exactly once delivery is impossible.")

Re: Yes, you can have exactly-once delivery

#52

Post by lisper »

mlyle wrote: post #27
lisper wrote: post #26

Earlier quoted context omitted.

> referring to the delivery of the message to the system itself And how do you define "the system itself"?

The thing that is at the end of the lossy medium. It must tolerate (0 or 1) or (1 or more) things being delivered to it.

Yes, that is true. But why can't I choose to view "the system itself" as the thing that is on the other side of a de-duplicator?

It feels to me like an argument over whether or not humans can fly. An unassisted human cannot fly, but with some technological augmentation, they can. It seems a bit pedantic to deny that someone can fly from LA to New York simply because they have to get into an airplane to do it.

Re: Yes, you can have exactly-once delivery

#53

Post by philipswood »

Groxx wrote: post #20
bhaney wrote: post #11

In case you're someone who actually knows anything about distributed systems and you're not looking forward to slogging through this long article filled with claims like "I have a PhD in AI so I know what I'm talking about" to find where the author made their mistake, let me save you the time. It's the typical conflation of exactly-once delivery with exactly-once processing, which the author acknowledges and then cho…

Yea... this is kinda inflammatory but I honestly have to agree with it. The post largely summarizes as "you can have exactly-once delivery if you re-define it to be at-least-once processing with idempotency". Those are different things. In fact, that's the entire point behind saying that it's impossible. You can't design a system that is exactly-once at any level, so don't even bother trying. If someone wants you to…

So if you build systems using messaging middleware you get to choose upfront whether you are going to use XA with exactly-once semantics and pay the performance penalty for that or whether you will implement your business logic idempotently on processing instead.

You can do this across a whole landscape of vendors.

So this whole class of "that's impossible" responses sounds to me (as an ex-brick layer) like "obviously you can't stack bricks next to each other perfectly straightly - so building walls is clearly impossible".

So it feels a bit jarring when several vendors allow you to do this impossible thing.

Google for JMS and exactly-once and you can find documentation with several products on exactly how to do this.

One example: https://www.atomikos.com/Blog/TheSimpleSecretOfExactlyOnceDe...

Re: Yes, you can have exactly-once delivery

#54

Post by mtndew4brkfst »

jongjong wrote: post #31
qaq wrote: post #12

Earlier quoted context omitted.

you can't in general case because you don't have infinite memory for dedupe.

Good point but you don't need infinite memory. You can set expiries to discard, for example. Expiries and timeouts can be defined as part of the protocol. You can impose reasonable constraints per-socket on the buffer size for spam prevention. There is a lot of wiggle room. On the sender side, you can require an ACK response for each message UUID and rebroadcast only if the ACK is not received within a certain timefr…

If lack of a successful ACK means you retry indefinitely until you do get an ACK, then whichever side handles deduplication need to store all observed messages indefinitely or you may deliver the same message more than once without correctly deduplicating it.

If instead you only store them for N days but an ongoing retry loop finally succeeds in N+1 days, you get duplicated delivery.

If a retry loop gives up after finite elapsed time, which sets a ceiling on how long you must retain observations, then you do not in fact achieve at-least-once delivery without using a lesser qualifying statement.

One of those constraints must be compromised on. That doesn't mean a practically useful implementation can't exist, it just means a theoretically ideal one can't without the infinite memory.

Many systems and real world processes would actually suffer negative consequences if a weeks-old payload was delivered-as-new, so hardly anyone balks at this part of the implications. But again, you have to redefine at least one of the claims (or acquire infinite memory) to have a coherent system that fulfills those claims.

Re: Yes, you can have exactly-once delivery

#55

Post by jchw »

warkdarrior wrote: post #48
jchw wrote: post #13

Earlier quoted context omitted.

The point is that it doesn't matter exactly where the deduplication matters. It could happen in your own processing code, or something upstream of it, like a queue library of some kind. That's pretty much what the entire article is saying; it's hard to meaningfully distinguish what part is actually delivery versus processing. e.g. most people would consider the guarantees imparted by the TCP stack are indeed part of…

> The point is that it doesn't matter exactly where the deduplication matters. Actually the point is that once deduplication is done at some layer, the layers above it will have to re-achieve exactly-once delivery. "Yes, the TCP layer did deliver this message only once, but the receiving software crashed right after, so now the sender has to send it again."

Hmmm. Maybe this is the reason why the processing vs delivery distinction matters. Because my thought is, well of course: To fix that you only send the acknowledgement after processing succeeds.

But then again, once you do that, the processing code that is being wrapped really doesn't have to care about being idempotent anymore, as it is being handled a layer up. At that point, all it needs to care about is being atomic.

I'm not sure if it practically matters either way. I'd rather have my processing code be both atomic and idempotent regardless just to make things easier to reason about, as long as it's not too much of a burden. I've always been a fan of concepts like idempotency tokens.

Re: Yes, you can have exactly-once delivery

#56

Post by lisper »

mlyle wrote: post #33
lisper wrote: post #25

Earlier quoted context omitted.

It is hard to make the point that "exactly-once delivery" is not a technical term without referring to it. If you think it is a technical term, would you kindly point me to a definition? I'm particularly interested in learning how "exactly-once delivery" is distinguished from "exactly-once processing".

A 4 year old piece laying out the exact difference as it's understood and how people use the terms: https://blog.bulloak.io/post/20200917-the-impossibility-of-e... I read similar content at least 2 decades ago... > While exactly-once-delivery is not possible, we have a way out: Exactly-once processing. Exactly-once processing is the guarantee that even though we may receive a message multiple times, in the end, we ob…

> A 4 year old piece laying out the exact difference...

"Exactly-once delivery guarantee is the guarantee that a message can be delivered to a recipient once, and only once."

That seems circular to me.

Also, the author's proof is flawed. The 2GP requires more than exactly-once delivery, it requires common knowledge. It is not enough for the first general to know that the message will be received, it is required that the first general knows that the message has been received, and that the second general knows that the first general knows this, and that the first general knows that the second general knows... and so on.

Re: Yes, you can have exactly-once delivery

#58

Post by tacitusarc »

theamk wrote: post #37
tacitusarc wrote: post #6

I appreciate the insights here, but I am struggling to understand how “exactly one” can equate to “eliminate duplicates”. Let’s say someone arrived at my house and cut my grass, and I failed to confirm they had done so, so the company sent someone over to cut my grass again, maybe multiple times. It seems silly to claim my grass was cut exactly once, despite it consistently remaining at the same height. Obviously it…

We are talking network stack, so there is no actions - just data hand-off to the actual application code. Someone arrives at your house, gives you a package, says "this is order 123". You thank them, they leave, but then they are hit by a car before they can report this. You unpack the package and use it. Next day, someone else arrives at your house, gives you a package, says "this is order 123". You thank them, they…

As this is a semantic debate over the definition of delivery, I asked my very non-technical wife if she thought in the scenario you described, the package was delivered exactly once. She said obviously not, and this discussion is very stupid, and I should stop participating in it. So there’s that.

Re: Yes, you can have exactly-once delivery

#59

Post by Groxx »

philipswood wrote: post #53
Groxx wrote: post #20

Earlier quoted context omitted.

Yea... this is kinda inflammatory but I honestly have to agree with it. The post largely summarizes as "you can have exactly-once delivery if you re-define it to be at-least-once processing with idempotency". Those are different things. In fact, that's the entire point behind saying that it's impossible. You can't design a system that is exactly-once at any level, so don't even bother trying. If someone wants you to…

So if you build systems using messaging middleware you get to choose upfront whether you are going to use XA with exactly-once semantics and pay the performance penalty for that or whether you will implement your business logic idempotently on processing instead. You can do this across a whole landscape of vendors. So this whole class of "that's impossible" responses sounds to me (as an ex-brick layer) like "obviousl…

>In this post we will show how [exactly once delivery] can be done, plus how simple it is with Atomikos. ...

>The producer should do its processing and send its message as part of a JTA/XA transaction. This ensures a message is sent if and only if the transaction commits. Any failures will result in rollback of the JTA/XA transaction - and no message will be sent. This means that failures can be safely retried until they succeed, without sending the resulting message more than once.

This is at-least-once (retries) with probably-idempotency: a transaction.

As I said: charlatans.

They're selling you "exactly once" in the headline, but clearly state that it is not exactly once in the legalese below. This is less "buyer beware" and more "blatantly false advertising", in exactly the same way as a free energy machine. The abstraction they're offering may indeed be useful, but "exactly once" is literally lying, and I wouldn't trust them to be honest elsewhere eit

Re: Yes, you can have exactly-once delivery

#60

Post by srkiNZ »

lisper wrote: post #52
mlyle wrote: post #27

Earlier quoted context omitted.

The thing that is at the end of the lossy medium. It must tolerate (0 or 1) or (1 or more) things being delivered to it.

Yes, that is true. But why can't I choose to view "the system itself" as the thing that is on the other side of a de-duplicator? It feels to me like an argument over whether or not humans can fly. An unassisted human cannot fly, but with some technological augmentation, they can. It seems a bit pedantic to deny that someone can fly from LA to New York simply because they have to get into an airplane to do it.

> why can't I choose to view "the system itself" as the thing that is on the other side of a de-duplicator?

because the "de-duplicator" would either:

* be somewhere else on an unreliable network (in which case we have the same problem)

* be on the same machine (or in the same process) as "the system itself" (in which case from a distributed systems perspective makes it the same thing)

> It seems a bit pedantic...

It is pedantic. The only reason that these "delivery" rules are popular is because of how many times programmers have gotten it wrong. Mostly by making assumptions that either:

* the network is reliable

* the message queue (or whatever) will de-duplicate messages for me

Post reply on HN