Kafka at the low end: how bad can it get?
1–10 of 144 posts
Re: Kafka at the low end: how bad can it get?
#2thankfully early access for KIP-932 is coming in 1-3 weeks as the 4.0.0 release gets published
Re: Kafka at the low end: how bad can it get?
#3What do people recommend?
Especially for low levels of load, that doesn't require that the dispatcher and consumer are written in the same language.
Re: Kafka at the low end: how bad can it get?
#4What do people recommend? Especially for low levels of load, that doesn't require that the dispatcher and consumer are written in the same language.
Redis, SQLite or even a traditional DB like Postgres or MySQL can all do a better job than that.
Re: Kafka at the low end: how bad can it get?
#5What do people recommend? Especially for low levels of load, that doesn't require that the dispatcher and consumer are written in the same language.
(Author here)
RabbitMQ or AWS SQS are probably good choices.
Re: Kafka at the low end: how bad can it get?
#6What do people recommend? Especially for low levels of load, that doesn't require that the dispatcher and consumer are written in the same language.
We use RabbitMQ, and workers simply pull whatever is next in the queue after they finish processing their previous jobs. I’ve never witnessed jobs piling up for a single consumer.
Re: Kafka at the low end: how bad can it get?
#7What do people recommend? Especially for low levels of load, that doesn't require that the dispatcher and consumer are written in the same language.
Re: Kafka at the low end: how bad can it get?
#8What do people recommend? Especially for low levels of load, that doesn't require that the dispatcher and consumer are written in the same language.
[deleted]
Re: Kafka at the low end: how bad can it get?
#9What do people recommend? Especially for low levels of load, that doesn't require that the dispatcher and consumer are written in the same language.
Until you hit scale, the database you're already using is fine. If that's Postgres, look up SELECT FOR UPDATE SKIP LOCKED. The major convenience here - aside from operational simplicity - is transactional task enqueueing.
For hosted, SQS or Google Cloud Tasks. Google's approach is push-based (as opposed to pull-based) and is far and above easier to use than any other queueing system.
Re: Kafka at the low end: how bad can it get?
#10What do people recommend? Especially for low levels of load, that doesn't require that the dispatcher and consumer are written in the same language.
NATS https://docs.nats.io/nats-concepts/overview/compare-nats
NATS/WebSockets are good for 1 publisher -> many consumer (pubsub)
RabbitMQ is good for 1 producer -> 1 consumer with ack/nack
Right?