Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Delivering Billions of Messages Exactly Once

segment.com

Post reply on HN
Print view View original

111–120 of 141 posts

Re: Delivering Billions of Messages Exactly Once

#111

Post by ryanworl »

tedmiston wrote: post #109
ryanworl wrote: post #23

Earlier quoted context omitted.

I'm not the OP, but changing the number of Kafka partitions isn't a super graceful operation. You would be wise to add as many as you could reasonably need assuming one consumer thread per partition. But not too many because each one is at least two files on disk!

I haven't used Kafka yet, but a Kafka partition is roughly the same as a Kinesis shard, right?

Yes, but you have to set them up upfront and there is no API to split and merge partitions. If you add more partitions, it doesn't automatically re-shard for you either. I don't think it should automatically re-shard as it would cause a ton of disk and network IO, but just something to be aware of.

Re: Delivering Billions of Messages Exactly Once

#112

Post by alexandercrohde »

Here's a radical solution. Instead of becoming a scala pro akka stream 200k engineer with a cluster of kafka nodes that costs your company over $100,000 of engineering time, technical debt, opportunity cost, and server costs, just put it all in bigtable, with deduping by id....

Enough of resume-driven-engineering, why does every need to reinvent the wheel?

Re: Delivering Billions of Messages Exactly Once

#113

Post by azernik »

alexandercrohde wrote: post #112

Here's a radical solution. Instead of becoming a scala pro akka stream 200k engineer with a cluster of kafka nodes that costs your company over $100,000 of engineering time, technical debt, opportunity cost, and server costs, just put it all in bigtable, with deduping by id.... Enough of resume-driven-engineering, why does every need to reinvent the wheel?

Yup. Databases, whether relational or not, have been designed to solve all these problems in a much more "bulletproof" way than your piddly [1] several-dozen-engineer team could ever manage, no matter how genius they are.

[1] No disrespect meant - just a description of size. Source: running a piddly 2-person engineering team.

Re: Delivering Billions of Messages Exactly Once

#114

Post by wenc »

Spearchucker wrote: post #96
redy wrote: post #61

Earlier quoted context omitted.

Everybody knows "exactly once" means deduplication. This is not exactly a new problem. That said it's still a difficult problem and I actually wish people would stop trying to roll their own schemes. For example, this scheme relies on examining a Kafka outbound topic to resolve in-doubt outbound messages. But what happens if the outbound message + commit is still "in flight" when the system recovers so the system ret…

The protocol is in fact quite simple: 1. Sender sends message. 2. If no acknowledgement from recipient is returned to sender, resend message until an acknowledgement is received from recipient. 3. If recipient receives message, check store to see if it's been received before. 4. If it's not in the store, store it, acknowledge receipt to sender, and process it. 5. If it's already in the recipient's store, acknowledge…

What if the "acknowledge receipt to sender" message gets lost?

Re: Delivering Billions of Messages Exactly Once

#115

Post by openasocket »

siliconc0w wrote: post #88

Was thinking a 'reverse bloom filter' could be cool to possibly avoid the RocksDB for situations like this- turns out it already exists: https://github.com/jmhodges/opposite_of_a_bloom_filter I love it when that happens.

It should be noted it's impossible to have a 'reverse bloom filter' with the same properties as a regular bloom filter. That is, a constant predefined size gives you a particular false negative rate. The thing you linked to is really just a cache. It has to store an entire entry (not just set some bits based on the hash) and doesn't have a predictable false negative rate based on its size. For more info: https://cstheory.stackexchange.com/questions/6596/a-probabil...

Re: Delivering Billions of Messages Exactly Once

#116

Post by eternalban »

wbl wrote: post #105
rusanu wrote: post #99

Earlier quoted context omitted.

> it's still something that has to be dealt with somewhere Database programmers have the means to deal with it off-the-shelf: BEGIN TRANSACTION ... COMMIT. When your queues are in the database, this becomes trivial. Even without the system I'm talking about (Service Broker) that has the queues stored in the database, most regular messaging systems do support enrolling into a distributed transaction and achieve an ato…

But the application still has to deal with processing the same message twice if it dies before ACK.

He did say XA.

The transaction boundary defined in your consumer covers the interactions of that consumer with other XA aware nodes who all participate in a "distributed transaction". So you can process this message N times without committing, and thus possibly N times telling other systems to do M' side-effect of that message, but until you commit the world has not changed.

https://en.wikipedia.org/wiki/X/Open_XA

Re: Delivering Billions of Messages Exactly Once

#117

Post by bluejekyll »

ploxiln wrote: post #104
rusanu wrote: post #76

Earlier quoted context omitted.

Having spent 7 years of my life working with Pat Helland in implementing Exactly Once In Order messaging with SQL Server Service Broker[0] I can assure you that practical EOIO messaging is possible, exists, and works as advertised. Delivering data EOIO is not rocket science, TCP has been doing it for decades. Extending the TCP paradigms (basically retries and acks) to messaging is not hard if you buy into transacted…

> Extending the TCP paradigms (basically retries and acks) to messaging is not hard > Just ack after you commit locally. wait ... what is this "retry" and "ack"? is this so that the sender knows not to send the message again? that sounds suspiciously familiar ... I mean, come on, you're just describing a framework you wrote that handles "at least once delivery" plus "idempotent operations" for the programmer. That's…

"At least once", "at most once" and "exactly once" are fairly well established terms in this area.

Idempotency is a restriction/requirement that you may put in place in a distributed system to make the above types of guarantees. Deduping of messages like the article mentions means that it is using "at least once" message delivery, with some notion of unique message IDs and/or messages to then dedup the message.

TCP is a great analogy, where the Windows of the protocol are effectively maintained by Kafka. Anyway, "exactly once" is very hard and requires putting limits into the system such that deduping or something similar is possible. But I'd agree that in all cases anything that claims "exactly once" behavior is in fact implementing that on top of protocols with "at least once" primitives.

Re: Delivering Billions of Messages Exactly Once

#118

Post by ztorkelson »

wbl wrote: post #105
rusanu wrote: post #99

Earlier quoted context omitted.

> it's still something that has to be dealt with somewhere Database programmers have the means to deal with it off-the-shelf: BEGIN TRANSACTION ... COMMIT. When your queues are in the database, this becomes trivial. Even without the system I'm talking about (Service Broker) that has the queues stored in the database, most regular messaging systems do support enrolling into a distributed transaction and achieve an ato…

But the application still has to deal with processing the same message twice if it dies before ACK.

Right. The idea is that by having your database, message queue, and application all share the same transactional context, "reprocessing" the message twice doesn't matter, because the effects are only ever committed exactly once.

It's true that this doesn't work if your processing touches external systems or otherwise escapes the transaction context, but in those cases you do still get at-least-once delivery (or at-most-once, if you choose to commit the receipt before processing the message).

It really is a powerful technology and when leveraged can absolutely reduce the level of effort and cognitive burden to building correct asynchronous systems.

Re: Delivering Billions of Messages Exactly Once

#119

Post by ztorkelson »

alexbeloi wrote: post #100
rusanu wrote: post #85

Earlier quoted context omitted.

Yes, I'm talking about distributed systems and I am aware of the CAP theorem. Hence my choice of the word 'practical'. As I said, users had cases when the plumbing (messaging system) recovered and delivered messages after +40 days of network partitioning. Correctly written apps completed the business process associated with those messages as normal, no special case. Humans can identify and fix outages and databases c…

I'm not really versed in this topic, but it seems like using a database for a socket makes the system entirely centralized around that database. Is there something I'm missing?

ServiceBroker, at least, had the capability of (transactionally) sending messages between databases. So, if you drank the kool-aid (I did; it wasn't so bad), there needn't be "the centralized database". You can separate your databases and decompose your services, and indeed it's easier to do so correctly and with confidence because the technology eliminates a lot of hairy edge cases.

Re: Delivering Billions of Messages Exactly Once

#120

Post by derefr »

ploxiln wrote: post #104
rusanu wrote: post #76

Earlier quoted context omitted.

Having spent 7 years of my life working with Pat Helland in implementing Exactly Once In Order messaging with SQL Server Service Broker[0] I can assure you that practical EOIO messaging is possible, exists, and works as advertised. Delivering data EOIO is not rocket science, TCP has been doing it for decades. Extending the TCP paradigms (basically retries and acks) to messaging is not hard if you buy into transacted…

> Extending the TCP paradigms (basically retries and acks) to messaging is not hard > Just ack after you commit locally. wait ... what is this "retry" and "ack"? is this so that the sender knows not to send the message again? that sounds suspiciously familiar ... I mean, come on, you're just describing a framework you wrote that handles "at least once delivery" plus "idempotent operations" for the programmer. That's…

The point of throwing a message broker at a problem is not to get away from those things; the point is to package non-idempotent operations up into idempotent containers "on the wire", so that the at-least-once semantics of the message being brokered translate to exactly-once semantics for the message-body delivered to the consumer.
Post reply on HN