21–30 of 44 posts
I dont understand why almost nobody wants to understand the difference between message brokers and a distributed log and its implications.
Can you give a tldr/eli5?
I treat Redis a bit like I treat PostgreSQL. It's what I use when I don't know what I should use. (In fact these days I might even be tempted to start with a simple PostgreSQL based queue and only swap to Redis later if it becomes clear that's what's needed). I guess a better approach might be to carefully analyse requirements up front but if those requirements aren't known at the time you start the project it's usef…
I dont understand why almost nobody wants to understand the difference between message brokers and a distributed log and its implications.
I dont understand why almost nobody wants to understand the difference between message brokers and a distributed log and its implications.
Can you give a tldr/eli5?
I dont understand why almost nobody wants to understand the difference between message brokers and a distributed log and its implications.
Because almost everyone refers to Kafka as a queue. And Kafka isn’t a queue.
Especially if you want one-to-many connections, exactly once delivery and push/ pull consumers, it is worth checking out nats [0, 1]. It is very performant. There are clients for Javascript, Python, Rust and Go [2]. (I am not affiliated with them) [0] https://nats.io/ [1] https://github.com/nats-io/nats-server [2] https://nats.io/download/#nats-clients
that's just not true. NATS does have neither at-least-once nor exactly-once. NATS Jetstream has at-least-once, but it does not have flexible topics anymore like NATS. It's two completely different things.
Is anyone else considering using DyanmoDB for event sourcing instead of Kafka? We have a project at work where we store events into an event table. We then had a stream that invokes a distributor lambda. The distributor lambda lookups subscribers to a given event type. For each subscriber for a given event, we place a copy of the event to a SQS queue for the event subscriber. Each subscriber lambda can process events…
Earlier quoted context omitted.
Because almost everyone refers to Kafka as a queue. And Kafka isn’t a queue.
Doesn't confluent offer a JMS client for kafka? So certainly you can use kafka as a queue.
We use Kafka as a queue because we understand it very well, but it has a lot of limitations compared to "purpose-built" queuing services.
I treat Redis a bit like I treat PostgreSQL. It's what I use when I don't know what I should use. (In fact these days I might even be tempted to start with a simple PostgreSQL based queue and only swap to Redis later if it becomes clear that's what's needed). I guess a better approach might be to carefully analyse requirements up front but if those requirements aren't known at the time you start the project it's usef…
Same. Kafka might be better for larger datasets/throughput but a much harder to maintain and manage. Redis is much simpler if you know you aren’t going to scale that much
Just a fun question: is there a comparison that compares how well a simple PostgreSQL-based queue[0] fares against these specialized solutions? [0] https://blog.crunchydata.com/blog/message-queuing-using-nati...
https://www.psycopg.org/docs/advanced.html#asynchronous-noti...