RabbitMQ 4.0
81–90 of 120 posts
Re: RabbitMQ 4.0
#82I've regarded RabbitMQ as a secret weapon in plain sight for years. The killer reason people don't use it more is it "doesn't scale" to truly enormous sizes, but for anyone with less than a million users it's great. Too many people end up with their own half rolled pubsub via things like grpc, and they'd be far better off using this, particularly in the early stages of development.
Also want to give a shoutout to BeanStalkd: https://github.com/beanstalkd/beanstalkd If you are looking at RabbitMQ with "Maybe this is too much". Beanstalkd likely has features you need with almost none of the setup. Just don't web expose it ;)
Be careful with it, it will segfault randomly and there hasn't been a fix.
After hitting my own posts on the Google groups while sleepily debugging the segfault at wee hours of the morning and getting falsely excited about the possibility of a fix, I gave up and wrote a replacement: https://github.com/chronomq/chronomq
Have been running it for years without it falling over, submillisecond operations on average, and has processed billions of messages without failing.
Re: RabbitMQ 4.0
#83I've regarded RabbitMQ as a secret weapon in plain sight for years. The killer reason people don't use it more is it "doesn't scale" to truly enormous sizes, but for anyone with less than a million users it's great. Too many people end up with their own half rolled pubsub via things like grpc, and they'd be far better off using this, particularly in the early stages of development.
It isn't more popular because it's not easy to use it properly. I haven't touched it in years so I can't expand, but when I did, I had to write so many wrappers and add extra logic to use it properly.
I have found it to be more trouble than it is worth, as well
Re: RabbitMQ 4.0
#84The linked github page gives me 404. But here are the release notes: https://github.com/rabbitmq/rabbitmq-server/blob/main/releas...
Looks like 4.0.0 was pulled and replaced with a 4.0.1 release.
Re: RabbitMQ 4.0
#85I've regarded RabbitMQ as a secret weapon in plain sight for years. The killer reason people don't use it more is it "doesn't scale" to truly enormous sizes, but for anyone with less than a million users it's great. Too many people end up with their own half rolled pubsub via things like grpc, and they'd be far better off using this, particularly in the early stages of development.
I think people don't use it more because people don't really know what it is. From their website:
> RabbitMQ is a reliable and mature messaging and streaming broker, which is easy to deploy on cloud environments, on-premises, and on your local machine.
What does that mean? "Messaging and streaming broker"? I understand the need for worker queues to process videos, images, emails and such but I can't easily tell if that's what this is.
Also, what are the benefits of this over just processing incomplete records straight out of my database? i.e. using MySQL as a queue.
Re: RabbitMQ 4.0
#86I've regarded RabbitMQ as a secret weapon in plain sight for years. The killer reason people don't use it more is it "doesn't scale" to truly enormous sizes, but for anyone with less than a million users it's great. Too many people end up with their own half rolled pubsub via things like grpc, and they'd be far better off using this, particularly in the early stages of development.
If you’re not worried about scale, I’d just use a database backed queue.
Re: RabbitMQ 4.0
#87I've regarded RabbitMQ as a secret weapon in plain sight for years. The killer reason people don't use it more is it "doesn't scale" to truly enormous sizes, but for anyone with less than a million users it's great. Too many people end up with their own half rolled pubsub via things like grpc, and they'd be far better off using this, particularly in the early stages of development.
I think people don't use it more because people don't really know what it is. From their website: > RabbitMQ is a reliable and mature messaging and streaming broker, which is easy to deploy on cloud environments, on-premises, and on your local machine. What does that mean? "Messaging and streaming broker"? I understand the need for worker queues to process videos, images, emails and such but I can't easily tell if th…
[dead]
Re: RabbitMQ 4.0
#88I've regarded RabbitMQ as a secret weapon in plain sight for years. The killer reason people don't use it more is it "doesn't scale" to truly enormous sizes, but for anyone with less than a million users it's great. Too many people end up with their own half rolled pubsub via things like grpc, and they'd be far better off using this, particularly in the early stages of development.
For me the killer feature of Kafka is that topics are persistent until the data expires. Meaning different readers can be working at different offsets. And you can rewind or fast forward the offset you are reading, which can really be a life saver when things go sideways. Does RabbitMQ have equivalent features?
Yes, a stream queue type [0] is available where you can set retention, and replay messages.
Re: RabbitMQ 4.0
#89Earlier quoted context omitted.
When I read this message I think this an absolutely terrible waste of time for a startup/small company. I want to spend my time building features not infrastructure.
This stuff is so trivially easy these days. 20 years ago it was hard to deploy a clustered message queue application, but now we've got so much open source tooling. Are you telling me you can't deploy RabbitMQ in an afternoon? Give it a day or two and you've got a monitoring stack and some swanky GitOps. Now your OpEx has been reduced 80x for hardly doing anything. As for continual maintenance, us-east-1 has gone dow…
Who wants to monitor and be oncall for a critical rabbitMQ service that someone only runs to save $15/month? Even if there's one outage it's already worth just paying for it.
Re: RabbitMQ 4.0
#90I've regarded RabbitMQ as a secret weapon in plain sight for years. The killer reason people don't use it more is it "doesn't scale" to truly enormous sizes, but for anyone with less than a million users it's great. Too many people end up with their own half rolled pubsub via things like grpc, and they'd be far better off using this, particularly in the early stages of development.
I think people don't use it more because people don't really know what it is. From their website: > RabbitMQ is a reliable and mature messaging and streaming broker, which is easy to deploy on cloud environments, on-premises, and on your local machine. What does that mean? "Messaging and streaming broker"? I understand the need for worker queues to process videos, images, emails and such but I can't easily tell if th…
> what are the benefits of this over just processing incomplete records straight out of my database? i.e. using MySQL as a queue
Mainly throughput and latency. I haven’t used MySQL recently so some of this may apply more to Postgres.
Postgres has LISTEN/NOTIFY which helps with latency. I don’t think MySQL has LISTEN/NOTIFY, which means you’d have to resort to polling.
You have to use the `SELECT … FOR UPDATE SKIP LOCKED LIMIT 1` features to grab a message from the queue table, so multiple consumers don’t pull the same message.
The biggest issue, if you’re trying to achieve decent throughput, is dealing with bloat (empty pages still sitting on the disk that haven’t been cleaned up yet). You can run vacuum but an online vacuum will only mark the pages as available for reuse (doesn’t free up the disk space). And if you run a full vacuum (which will free the disk space) it locks the entire database while it runs the vacuum. This can compound if you’re using indexes.
One way of dealing with this is setting up partitioning by message timestamp, so that as old messages roll out, you can just drop those partitions and not have to deal with vacuum.
It can work if your queue needs are low throughput or can tolerate higher latency, but there are some things to manage, and realistically setting up a Redis instance is probably less complex than trying to work around the database-specific quirks, unless you’re already very familiar with your database’s inner workings.