31–40 of 141 posts
Site seems to be down. Any ideas how big these HN hugs of death usually are? How big of a traffic spike brings these servers down?
"Exactly once" model of message is theoretically impossible to do in distributed environment with nonzero possibility of failure. If you haven't received acknowledgement from the other side of communication in the specified amount of time you can only do one of two things: 1) do nothing, risking message loss 2) retransmit, risking duplication But of course that's only from messaging system point of view. Deduplicatio…
Wait why? Just because you'd have to store the list of seen messages theoretically indefinitely?
Would something like AWS SQS not scale for something like this? We currently push about 25k daily transactions over SQS, obviously no where near the scale of this, just wondering about what limitations we will bump into potentially.
The limitations are most likely on price. For the 200B messages they've already processed in the last 3 months, that would be $100,000 total on just the SQS FIFO queue, or $33,333 per month. And that's not counting data transfer.
There is still a potential for problems in the message delivery to the endpoints (malformed messages, Kafka errors, messages not being consumed fast enough and lost), or duplication at that level (restart a listener on the Kafka stream with the wrong message ID) as well.
This is based on my own pains with Kinesis and Lambda (which, I know, isn't Kafka).
In my experience, better to just allow raw "at least once" messaging and perform idempotant actions based off the messages. It's not always possible (and harder when it is possible), but its tradeoffs mean you're less likely to lose messages.
Earlier quoted context omitted.
(edit: incorrect, my bad, see thread)
This is wrong. There is a type of SQS queue that indeed does exactly-once. It costs more, and not the default option, but it is there.
"In the past three months we’ve built an entirely new de-duplication system to get as close as possible to exactly-once delivery"
What's annoying is that they do not get precise and formal about what they want out of their new model. Also, their numbers only speak to performance, not correctness.
On the plus side, I think it's awesome to see bloom filters successfully used in production. That sort of thing is easy to implement, but not easy to get right for every use case.
The scale is smaller (about 10k rpm), but the basic idea is the same (store a message ID in a key-value store after each successful send).
I like the idea of invalidating records by overall size, we hadn't thought of that. We just use a fixed 24-hour TTL.
Our network issues are nearly 6 times higher (~3.5%) due to GPRS, and we solved the duplication problem with an approach involving both client and server side.
Clients would always ensure that all the information sent by the server was successfully received. If something goes wrong, instead of retrying (sending the payment again), the client sends just the transaction UUID to the server, and the server might either respond with: A. the corresponding response for the transaction or B. not found.
In the scenario A, the POS terminal managed to properly send all the information to the server but failed to receive the response.
In the scenario B, the POS terminal didn't even manage to properly send the information to the server, so the POS can safely retry.
"Exactly once" model of message is theoretically impossible to do in distributed environment with nonzero possibility of failure. If you haven't received acknowledgement from the other side of communication in the specified amount of time you can only do one of two things: 1) do nothing, risking message loss 2) retransmit, risking duplication But of course that's only from messaging system point of view. Deduplicatio…
> there is no foolproof way of implementing that pseudocode's "has_seen(message.id)" method Wait why? Just because you'd have to store the list of seen messages theoretically indefinitely?
but if you can persist a monotonic sequence number, thats gets you pretty far. we use tcp all the time even though its has no magic answer to distributed consensus (and uses a super-weak checksum). 2pc doesnt guarantee progress and/or consistency either and its pretty effective.
[0] https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+E...