Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Delivering Billions of Messages Exactly Once

segment.com

Post reply on HN
Print view View original

11–20 of 141 posts

Re: Delivering Billions of Messages Exactly Once

#11

Post by psadauskas »

stratosgear wrote: post #8
Exuma wrote: post #4

Earlier quoted context omitted.

It's not down

Hmm weird. I get: This site can’t be reached segment.com refused to connect. Try: Checking the connection ERR_CONNECTION_REFUSED Maybe something local to me only?

Its probably an ad- or tracker-blocker. I had to disable mine to be able to load segment.com, since they're pretty much a tracking company.

Re: Delivering Billions of Messages Exactly Once

#12

Post by caust1c »

ggcampinho wrote: post #5

Isn't the new feature of Kafka about this? https://issues.apache.org/jira/browse/KAFKA-4815

It would help, but messages are sent to the API first. We aren't sending messages directly to Kafka from the Internet, of course.

So we can get duplicate API submissions regardless of whether or not we enabled transactional productions into kafka from a producer.

Re: Delivering Billions of Messages Exactly Once

#13

Post by fouadmatin »

stratosgear wrote: post #8
Exuma wrote: post #4

Earlier quoted context omitted.

It's not down

Hmm weird. I get: This site can’t be reached segment.com refused to connect. Try: Checking the connection ERR_CONNECTION_REFUSED Maybe something local to me only?

Might be an adblock issue? (some blockers treat direct access to a website and loading an anlytics snippet as the same)

Re: Delivering Billions of Messages Exactly Once

#15

Post by psadauskas »

wonderwonder wrote: post #9

Would something like AWS SQS not scale for something like this? We currently push about 25k daily transactions over SQS, obviously no where near the scale of this, just wondering about what limitations we will bump into potentially.

SQS is not "exactly once", so might not meet their requirements.

Re: Delivering Billions of Messages Exactly Once

#16

Post by asadlionpk »

stratosgear wrote: post #8
Exuma wrote: post #4

Earlier quoted context omitted.

It's not down

Hmm weird. I get: This site can’t be reached segment.com refused to connect. Try: Checking the connection ERR_CONNECTION_REFUSED Maybe something local to me only?

You are probably blocking this site in your hosts file. Since it's a tracking tool.

Re: Delivering Billions of Messages Exactly Once

#17

Post by iampims »

If the OP doesn't mind expanding a little on this bit, I'd be grateful.

> If the dedupe worker crashes for any reason or encounters an error from Kafka, when it re-starts it will first consult the “source of truth” for whether an event was published: the output topic.

Does this mean that "on worker crash" the worker replays the entire output topic and compare it to the rocksdb dataset?

Also, how do you handle scaling up or down the number of workers/partitions?

Re: Delivering Billions of Messages Exactly Once

#19

Post by eropple »

wonderwonder wrote: post #9

Would something like AWS SQS not scale for something like this? We currently push about 25k daily transactions over SQS, obviously no where near the scale of this, just wondering about what limitations we will bump into potentially.

(edit: incorrect, my bad, see thread)

Re: Delivering Billions of Messages Exactly Once

#20

Post by mamon »

"Exactly once" model of message is theoretically impossible to do in distributed environment with nonzero possibility of failure. If you haven't received acknowledgement from the other side of communication in the specified amount of time you can only do one of two things:

1) do nothing, risking message loss

2) retransmit, risking duplication

But of course that's only from messaging system point of view. Deduplication at receiver end can help reduce problem, but itself can fail (there is no foolproof way of implementing that pseudocode's "has_seen(message.id)" method)

Post reply on HN