Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Delivering Billions of Messages Exactly Once

segment.com

Post reply on HN
Print view View original

51–60 of 141 posts

Re: Delivering Billions of Messages Exactly Once

#51

Post by lostcolony »

idunno246 wrote: post #21
psadauskas wrote: post #15

Earlier quoted context omitted.

SQS is not "exactly once", so might not meet their requirements.

sqs has fifo queues which claim to be exactly once

SQS queues deduplicate over a 5 minute window. This is claiming a much larger window.

Either way your listener(s) still has to have its own deduplication. Ensuring a message ends up on the queue only once, and ensuring it's processed exactly once, are two different problems that require separate handling (and, the former is what most out of the box systems claim to solve, while the latter is more important, and, frankly, completely negates the need of the former).

Re: Delivering Billions of Messages Exactly Once

#52

Post by jmaygarden »

gsmethells wrote: post #43

Why do I get the feeling this is repeating TCP features at the Message level? There must a protocol that can hide this exactly once need away. TCP doesn't create downloads, generally, that are bad and fail their checksum test, hence packets that make up the file are not duplicated.

How would you use TCP sockets to de-duplicate Kafka streams with a many-to-many communication pattern? Surely there is a valid scalability reason for why AWS IoT only provides "at least once" guarantees in their MQTT broker even when TCP is the underlying transport [1].

[1] http://docs.aws.amazon.com/iot/latest/developerguide/protoco...

Re: Delivering Billions of Messages Exactly Once

#53

Post by caust1c »

falcolas wrote: post #34

So, a combination of a best effort "at least once" messaging with deduplication near the receiving edge. Fairly standard, honestly. There is still a potential for problems in the message delivery to the endpoints (malformed messages, Kafka errors, messages not being consumed fast enough and lost), or duplication at that level (restart a listener on the Kafka stream with the wrong message ID) as well. This is based on…

This is generally better, but we're delivering these messages to integrations which don't necessarily take idempotent actions.

Re: Delivering Billions of Messages Exactly Once

#54

Post by stratosgear »

disconnected wrote: post #24
stratosgear wrote: post #8

Earlier quoted context omitted.

Hmm weird. I get: This site can’t be reached segment.com refused to connect. Try: Checking the connection ERR_CONNECTION_REFUSED Maybe something local to me only?

Check your ad blocker/hosts file. In here, uMatrix just blocks the site with the message: > uMatrix has prevented the following page from loading: > https://segment.com/blog/exactly-once-delivery/ I checked, and one of my uMatrix hosts files includes 'www.segment.com'.

Yep. Disabling AdAway seems to do the trick. Thanks for the heads up

Re: Delivering Billions of Messages Exactly Once

#56

Post by StavrosK »

ecesena wrote: post #30
mamon wrote: post #20

"Exactly once" model of message is theoretically impossible to do in distributed environment with nonzero possibility of failure. If you haven't received acknowledgement from the other side of communication in the specified amount of time you can only do one of two things: 1) do nothing, risking message loss 2) retransmit, risking duplication But of course that's only from messaging system point of view. Deduplicatio…

I think "exactly once" doesn't imply "non-reprocessing" (sorry for the double negative). Meaning, you want "exactly once" and you don't want duplicates, yes. But you allow for reprocessing, provided that you have a way for deduplicating. You want a guarantee that if the producer (at the top of your data processing pipeline) sends a message, then this message eventually corresponds to exactly 1 record in your final st…

So basically what's called "at least once with deduplicating". The parent comment addresses that.

Re: Delivering Billions of Messages Exactly Once

#58

Post by jkarneges »

gsmethells wrote: post #43

Why do I get the feeling this is repeating TCP features at the Message level? There must a protocol that can hide this exactly once need away. TCP doesn't create downloads, generally, that are bad and fail their checksum test, hence packets that make up the file are not duplicated.

Yes there is some duplication of TCP capability here.

The problem with relying on TCP for reliability is that its state is in memory, associated with a particular peer IP address, and acknowledgements passed back to the sender only indicate that the receiver has the data in local memory, not that the data has been processed.

A file download over TCP can fail, for example due to a network problem. Ensuring reliable delivery requires additional measures outside of TCP, such as retrying the download using a new connection.

In practice, this means that TCP is primarily useful for providing flow control and offering a streaming interface (no worry about packet sizes). Less so as a complete solution for transmission reliability.

Re: Delivering Billions of Messages Exactly Once

#59

Post by sethammons »

"Exactly Once"

Over a window of time that changes depending on the amount of ingested events.

Basically, they read from a kafka stream and have a deduplication layer in rocks db that produces to another kafka stream. They process about 2.2 billion events through it per day.

While this will reduce duplicates and get closer to Exactly Once (helping reduce the two generals problem on incoming requests and potentially work inside their data center), they still have to face the same problem again when they push data out to their partners. Some packet loss, and they will be sending out duplicate to the partner.

Not to downplay what they have done as we are doing a similar thing near our exit nodes to do our best to prevent duplicate events making it out of our system.

Post reply on HN