91–100 of 144 posts
What do people recommend? Especially for low levels of load, that doesn't require that the dispatcher and consumer are written in the same language.
Until you hit scale, the database you're already using is fine. If that's Postgres, look up SELECT FOR UPDATE SKIP LOCKED. The major convenience here - aside from operational simplicity - is transactional task enqueueing. For hosted, SQS or Google Cloud Tasks. Google's approach is push-based (as opposed to pull-based) and is far and above easier to use than any other queueing system.
What do people recommend? Especially for low levels of load, that doesn't require that the dispatcher and consumer are written in the same language.
Kafka. If your load is low enough for the problem described in the article to happen, your load is low enough that it's not an issue.
Earlier quoted context omitted.
Why does everybody keep missing this point? I don’t know.
There's a wonderful Kafka Children's book that I always suggest every team I work with read: https://www.gentlydownthe.stream/ The way I describe Kafka is, "an event has transpired... sometimes you care, and choose to take an action based on that event" The way I describe RabbitMQ is, "there's a new ticket in the lineup... it needs to be grabbed for action or left in the lineup... or discarded" Definitely not perfect…
Kafka for small message volumes is one of those distinct resume-padding architectural vibes.
If anybody knows of a simpler way to accomplish this, please do let me know.
The kafka protocol is a distributed write ahead log. If you want a job queue you need to build something on top of that, it’s a pretty low level primative.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A...
Earlier quoted context omitted.
You haven't seen the worst of it. We had to implement a whole kafka module for a SCADA system because Target already had unrelated kafka infrastructure. Instead of REST API or anything else sane (which was available), ultra low volume messaging is now done by JSON objects wrapped in kafka. Peak incompetence.
We did something similar using RabbitMQ with bson over AMQP, and static message routing. Anecdotally, the design has been very reliable for over 6 years with very little maintenance on that part of the system, handles high-latency connection outage reconciliation, and new instances are cycled into service all the time. Mostly people that ruminate on naive choices like REST/HTTP2/MQTT will have zero clue how the probl…
Honestly from reading this it seems like you’re the one who is too smart to work with any team.
Earlier quoted context omitted.
Until you hit scale, the database you're already using is fine. If that's Postgres, look up SELECT FOR UPDATE SKIP LOCKED. The major convenience here - aside from operational simplicity - is transactional task enqueueing. For hosted, SQS or Google Cloud Tasks. Google's approach is push-based (as opposed to pull-based) and is far and above easier to use than any other queueing system.
How could I solve the problem of in-order processing based on a key using skip locked? Basically all records having the key to be processed one after other.
SELECT
*
FROM jobs
WHERE partition_key = (
SELECT partition_key
FROM jobs
ORDER BY partition_key
LIMIT 1
SKIP LOCKED
)
ORDER BY submitted_at
FOR UPDATE SKIP LOCKED;What do people recommend? Especially for low levels of load, that doesn't require that the dispatcher and consumer are written in the same language.
Pulsar. Works extremely well as both a job queue and a data bus. We have been using it in this application for half a decade now with no serious issues. I don't understand why it doesn't get more popular attention.
thankfully early access for KIP-932 is coming in 1-3 weeks as the 4.0.0 release gets published
First time I've heard of KIP-932 and it looks very good. The two biggest issues IMO are finding a good Kafka client in the language you need (even for ruby this is a challenge) and easy at-least-once workers. You can over partition and make at-least-once workers happen (if you have a good Kafka client), or you use an http gateway and give up safe at-least-once. Hopefully this will make it easier to build an at-least-…
Uber wrote a Kafka push gateway years ago, when it was considerably harder to do well: https://www.uber.com/blog/kafka-async-queuing-with-consumer-...