Kafka at the low end: how bad can it get?

Post by enether » Sun, Feb 23, 2025, 10:00 PM UTC

agallego wrote: post #136

enether wrote: post #134

Earlier quoted context omitted.

It's just my polite way of saying it's safe enough for most use cases and that you're wrong. The fsync thing is complete FUD by RedPanda. They later introduce write caching[1] and call it an innovation[2]. I notice you also work for them. Nevertheless, those that are super concerned with safety usually run with an RF of 5 (e.g banks). And you can configure Kafka to fsync as often as you want[3] 1 - https://www.redpan…

It is not FUD. It is deterministic. Reproducible on your laptop. Out of all the banks I work with only a handful of use cases use rf=5. Defaults matter, because most people do not change them.

Post by rcombatwombat » Mon, Feb 24, 2025, 7:26 PM UTC

relistan wrote: post #76

Joel_Mckay wrote: post #47

Earlier quoted context omitted.

Actually, I used RabbitMQ static routes to feed per-cpu-core single thread bound consumers that restart their process every k transactions, or watchdog process timeout after w seconds. This prevents cross contamination of memory spaces, and slow fragmentation when the parsers get hammered hard. RabbitMQ/Erlang on OTP is probably one of the most solid solutions I've deployed over the years (low service cycle demands.)…

Agree. RabbitMQ is a Swiss Army knife that has a lot of available patterns, scales fairly well, and is very robust. If you don’t know what to choose, start with Rabbit. It will let you figure out which patterns you need and you probably won’t scale out of it. Pay someone to host it. On the other hand, if you know what you need to do and it’s supported by it, NATS is IME the way to go (particularly JetStream).