What do people recommend? Especially for low levels of load, that doesn't require that the dispatcher and consumer are written in the same language.
Sqs- easy and cheap.
Kafka at the low end: how bad can it get?
21–30 of 144 posts
Re: Kafka at the low end: how bad can it get?
#22Re: Kafka at the low end: how bad can it get?
#23Earlier quoted context omitted.
Until you hit scale, the database you're already using is fine. If that's Postgres, look up SELECT FOR UPDATE SKIP LOCKED. The major convenience here - aside from operational simplicity - is transactional task enqueueing. For hosted, SQS or Google Cloud Tasks. Google's approach is push-based (as opposed to pull-based) and is far and above easier to use than any other queueing system.
Famious last words. There are database as a queue antipattern warnings about this.
Can you elaborate? I guess it has to do with connection pooling?
Re: Kafka at the low end: how bad can it get?
#24What do people recommend? Especially for low levels of load, that doesn't require that the dispatcher and consumer are written in the same language.
NATS https://docs.nats.io/nats-concepts/overview/compare-nats
I'd wish NATS were more popular. It feels it lacks some real big sponsors $$$.
Re: Kafka at the low end: how bad can it get?
#25Re: Kafka at the low end: how bad can it get?
#26Earlier quoted context omitted.
Until you hit scale, the database you're already using is fine. If that's Postgres, look up SELECT FOR UPDATE SKIP LOCKED. The major convenience here - aside from operational simplicity - is transactional task enqueueing. For hosted, SQS or Google Cloud Tasks. Google's approach is push-based (as opposed to pull-based) and is far and above easier to use than any other queueing system.
Famious last words. There are database as a queue antipattern warnings about this.
I suppose you are referring to this:
https://mikehadlow.blogspot.com/2012/04/database-as-queue-an...
The main complaint seems to be that it's not optimal...but then, the frame of the discussion was "Until you hit scale", so IMHO convenience and simpler infra trumps having the absolute most efficient tool at that stage.
Re: Kafka at the low end: how bad can it get?
#27For a small load queueing system, I had great success with Apache ActiveMQ back in the days. I designed and implemented a system with the goal of triggering SMS for paid content. This was in 2012.
Ultimately, the system was fast enough that the telco company emailed us and asked to slow down our requests because their API was not keeping up.
In short: we had two Apache Camel based apps: one to look at the database for paid content schedule, and queue up the messages (phone number and content). Then, another for triggering the telco company API.
Re: Kafka at the low end: how bad can it get?
#28Kafka for small message volumes is one of those distinct resume-padding architectural vibes.
Or, as mentioned in the article, you've already got Kafka in place handling a lot of other things but need a small queue as well and were hoping to avoid adding a new technology stack into the mix.
Re: Kafka at the low end: how bad can it get?
#29What do people recommend? Especially for low levels of load, that doesn't require that the dispatcher and consumer are written in the same language.
Kafka. If your load is low enough for the problem described in the article to happen, your load is low enough that it's not an issue.
Re: Kafka at the low end: how bad can it get?
#30Kafka for small message volumes is one of those distinct resume-padding architectural vibes.
You haven't seen the worst of it. We had to implement a whole kafka module for a SCADA system because Target already had unrelated kafka infrastructure. Instead of REST API or anything else sane (which was available), ultra low volume messaging is now done by JSON objects wrapped in kafka. Peak incompetence.