Delivering Billions of Messages Exactly Once • Hacker News Board

Re: Delivering Billions of Messages Exactly Once

#131

Post by [deleted] » Fri, Jun 30, 2017, 10:33 AM UTC

urbit wrote: post #67

newobj wrote: post #48

I don't want to ever see the phrase "Exactly Once" without several asterisks behind it. It might be exactly once from an "overall" point of view, but the client effectively needs infinitely durable infinite memory to perform the "distributed transaction" of acting on the message and responding to the server. Imagine: - Server delivers message M - Client process event E entailed by message M - Client tries to ack (A)…

Exactly-once messaging is not a hard problem so long as you change the problem a little. (Plug warning: this is the way Urbit does EOM, or EOM * if you prefer.) TLDR, you don't need infinitely durable infinite memory. You just need (a) a single-level store in which every event is a transaction, (b) a message protocol with true end-to-end acks, and (c) a permanent session between every pair of nodes. We don't have sin…

[deleted]

Re: Delivering Billions of Messages Exactly Once

#132

Post by mshenfield » Fri, Jun 30, 2017, 10:53 AM UTC

travisjeffery wrote: post #55

Kafka 0.11 (recently released) has exactly once semantics and transactional messages built-in. - Talk from Kafka Summit: https://www.confluent.io/kafka-summit-nyc17/resource/#exactl... - Proposal: https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+E...

My understanding (noob here) is that it allows producers to retry without fear of duplication. You still have to consider the system feeding the producer though. In Segment's example, clients might deliver their messages more than once to the API. Kafka's mechanism wouldn't detect duplicate messages sent to the producer, just that any given message a producer wants to append to Kafka won't be duplicated.

Re: Delivering Billions of Messages Exactly Once

#133

Post by sobani » Fri, Jun 30, 2017, 11:26 AM UTC

iso-8859-1 wrote: post #70

mamon wrote: post #20

"Exactly once" model of message is theoretically impossible to do in distributed environment with nonzero possibility of failure. If you haven't received acknowledgement from the other side of communication in the specified amount of time you can only do one of two things: 1) do nothing, risking message loss 2) retransmit, risking duplication But of course that's only from messaging system point of view. Deduplicatio…

> nonzero If you send the message with a hash of the previous state on the server, (like with proof-of-work in Bitcoin), since it is so unlikely that it will hash will be the same with and without the message appended, it doesn't really matter if it is strictly nonzero, if it is just small enough.

The nonzero part is about the possibility of network / endpoint failure. No hash is going to prevent the cleaning lady from 'freeing up' the power socket for her vacuum.

Re: Delivering Billions of Messages Exactly Once

#134

Post by ngrilly » Fri, Jun 30, 2017, 12:43 PM UTC

rusanu wrote: post #90

YZF wrote: post #87

Earlier quoted context omitted.

TCP does deliver data more than once though. Sure within a TCP session you are guaranteed not to get the same byte twice out of your socket, bytes are delivered exactly once, in order. Now if your application that uses TCP pulls data out of the socket and then dies the data will need to be delivered again and the TCP protocol is unable to help us there, it's application level logic at that point. So everyone is talki…

Not when your 'socket' is a persisted, durable, transacted medium (ie. a database). Sure, applications can 'pull data out of the socket and then die', but this is a common scenarios on databases which is handled with transaction and post-crash recovery. The application comes back after the crash and find the same state as before the crash (the 'socket' still has the data ready to pull off), it pull again, process, an…

> Whenever the plumbing does not guarantees EOIO but the business demands it, it gets pushed into the app layer where TCP (retries and acks) is reimplemented, to various success levels.

Interesting! You're saying the exact opposite of what Tyler Treat is saying:

> Even with smart middleware, problems still leak out and you have to handle them at the edge—you’re now being taxed twice. This is essentially the end-to-end argument. Push responsibility to the edges, smart endpoints, dumb pipes, etc. It’s the idea that if you need business-level guarantees, build them into the business layer because the infrastructure doesn’t care about them.

(source: http://bravenewgeek.com/smart-endpoints-dumb-pipes/)

What do you think of his argument?

Re: Delivering Billions of Messages Exactly Once

#135

Post by ngrilly » Fri, Jun 30, 2017, 1:02 PM UTC

majidazimi wrote: post #73

What is so exciting about this? There is still possibility of duplicates. You still have to put the engineering effort to deal with duplicates end-to-end. If the code is there to deal with duplicates end-to-end, then does it really matter to have 5 duplicates or 35? Or may be they just did it to add some useful cool-tech in to CV?

> There is still possibility of duplicates.

Where?

Re: Delivering Billions of Messages Exactly Once

#136

Post by ngrilly » Fri, Jun 30, 2017, 1:05 PM UTC

pfarnsworth wrote: post #122

Sounds very cool. A couple of questions I had: 1) What happens if they lose their rocksdb with all of the messageIds? 2) Is their kafka atleast-once delivery? How do they guarantee that kafka doesn't reject their write? Also, assuming they have set up their kafka for at least once delivery, doesn't that make the output topic susceptible to duplicates due to retries, etc? 3) >Instead of searching a central database fo…

> What happens if they lose their rocksdb with all of the messageIds?

I'm wondering the same.

Re: Delivering Billions of Messages Exactly Once

#137

Post by v1shnu » Fri, Jun 30, 2017, 4:01 PM UTC

bmsatierf wrote: post #38

In terms of connectivity, we deal with a similar problem here at CloudWalk to process payment transactions from POS terminals, where most of them rely on GPRS connections. Our network issues are nearly 6 times higher (~3.5%) due to GPRS, and we solved the duplication problem with an approach involving both client and server side. Clients would always ensure that all the information sent by the server was successfully…

Why not just send the data too along with the UUID? It'll save another roundtrip in case of scenario B right? Or do you have data to prove that scenario B is a lot less likely to occur, making it sensible to save bandwidth by not re-transmitting the data?

Re: Delivering Billions of Messages Exactly Once

#138

Post by vannevar » Fri, Jun 30, 2017, 4:56 PM UTC

StreamBright wrote: post #6

"The single requirement of all data pipelines is that they cannot lose data." Unless the business value of data is derived after applying some summary statistics, than even sampling the data works, and you can lose events in an event stream, while not changing the insight gained. Originally Kafka was designed to be a high throughput data bus for analytical pipeline where losing messages was ok. More recently they are…

Yeah, this was a major overstatement. There are lots of data pipelines where it's ok to lose some data. Consider a sensor that sends measurements hundreds of time a second to an app that operates on a 1-second timeframe. And UDP is used all the time on the internet, yet carries no delivery guarantee.

Re: Delivering Billions of Messages Exactly Once

#139

Post by goodplay » Sat, Jul 01, 2017, 2:21 AM UTC

stratosgear wrote: post #54

disconnected wrote: post #24

Earlier quoted context omitted.

Check your ad blocker/hosts file. In here, uMatrix just blocks the site with the message: > uMatrix has prevented the following page from loading: > https://segment.com/blog/exactly-once-delivery/ I checked, and one of my uMatrix hosts files includes 'www.segment.com'.

Yep. Disabling AdAway seems to do the trick. Thanks for the heads up

You can also add an exception for a blocked site by marking it green and clicking on the opened padlock in the umatrix panel.

No need to nuke a city to get a fly.

Re: Delivering Billions of Messages Exactly Once

#140

Post by vacri » Sat, Jul 01, 2017, 4:07 AM UTC

maffydub wrote: post #130

vacri wrote: post #129

Earlier quoted context omitted.

There comes a point in the chain where both sides have got the initial message, and confirmation that the other side has confirmed the message. message 1 -> attack! message 2 -> sure! message 3 -> I got your sure! At this point, general A knows the message got through message 4 -> I got your confirmation! At this point, general B knows that general A got the reply message 5 -> I got your confirmation confirmation! At…

The point is that it's not just about knowledge - it's about agreement. A and B must not attack alone, i.e. A must not attack if B is not going to attack, and B must not attack if A is not going to attack. Now, message 1 (from A to B) says "attack!". So B knows that A wants B to attack. ...but A mustn't attack alone, so A won't attack unless A receives B's confirmation. So B sends message 2 ("sure!"). ...but B mustn'…

"Dear diary. I have just sent the 537th messenger confirming that we are attacking together tomorrow morning at 9. I'm just waiting on confirmation."

You have that confirmation of agreement in the third round of messages. Receipt of the fourth message proves that both sides have knowledge and have agreed, aware of the other's intent. That's the message you have to prove has arrived for surety. Once you've done enough messages to prove message #4, you're good to go.