Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I’m leaving Redis for SolidQueue

simplethread.com

Post reply on HN
Print view View original

121–130 of 146 posts

Re: I’m leaving Redis for SolidQueue

#121

Post by bgentry »

parthdesai wrote: post #120
bgentry wrote: post #106

Earlier quoted context omitted.

This is largely because LISTEN/NOTIFY has an implementation which uses a global lock. At high volume this obviously breaks down: https://www.recall.ai/blog/postgres-listen-notify-does-not-s... None of that means Oban or similar queues don't/can't scale—it just means a high volume of NOTIFY doesn't scale, hence the alternative notifiers and the fact that most of its job processing doesn't depend on notifications at al…

> None of that means Oban or similar queues don't/can't scale—it just means a high volume of NOTIFY doesn't scale Given the context of this post, it really does mean the same thing though?

No, I don't think so. Oban does not rely on a large volume of NOTIFY in order to process a large volume of jobs. The insert notifications are simply a latency optimization for lower volume environments, and for inserts can be fully disabled such that they're mainly used for control flow (canceling jobs, pausing queues, etc) and gossip among workers.

River for example also uses LISTEN/NOTIFY for some stuff, but we definitely do not emit a NOTIFY for every single job that's inserted; instead there's a debouncing setup where each client notifies at most once per fetch period, and you don't need notifications at all in order to process with extremely high throughput.

In short, the fact that high volume NOTIFY is a bottleneck does not mean these systems cannot scale, because they do not rely on a high volume of NOTIFY or even require it at all.

Re: I’m leaving Redis for SolidQueue

#122

Post by solid_fuel »

parthdesai wrote: post #69
victorbjorklund wrote: post #9

For people that does not think it scales. A similar implementation in Elixir is Oban. Their benchmark shows a million jobs per minute on a single node (and I am sure it could be increased further with more optimizations). I bet 99,99999% of apps have less than a million background jobs per minute. https://oban.pro/articles/one-million-jobs-a-minute-with-oba...

Funny you mention Oban, we do use it at work as well, and first thing Oban tells you is to either use Redis as a notifier or resort to polling for jobs and just not notify. https://hexdocs.pm/oban/scaling.html

Not quite, I used it at work too - the first thing that page suggests is using `Oban.Notifiers.PG` which uses distributed erlang's Process Group implementation, not Redis. You only really need Redis if you're not running with erlang clustering, but doing that rules out several other great elixir features.

Re: I’m leaving Redis for SolidQueue

#123

Post by solid_fuel »

yandrypozo wrote: post #109
ivolimmen wrote: post #65

Exactly what https://www.amazingcto.com/postgres-for-everything/ says; keep it simpel and use PostgreSQL.

I love the idea of PG for everything, but every time I suggest it I get the same answer "When you're a hammer, everything looks like a nail" which makes sense to me, but not sure how to give a good answer to that phrase :(

I mean, that's just a truism - it's not really engineering advice. Maybe Postgres is just a hammer, but when you're building a house there's a lot of nails.

If you've got to store 5 GB videos, maybe reach for object store instead of postgres. But for most uses postgres is a solid choice.

Re: I’m leaving Redis for SolidQueue

#124

Post by yawboakye »

bgentry wrote: post #102
runako wrote: post #75

Earlier quoted context omitted.

SolidQueue uses its own db configuration. > it shouldn’t be the same as the production database This is highly dependent on the application (scale, usage, phase of lifecycle, etc.)

Yeah, River generally recommends this pattern as well (River co-author here :) To get the benefits of transactional enqueueing you generally need to commit the jobs transactionally with other database changes. https://riverqueue.com/docs/transactional-enqueueing It does not scale forever, and as you grow in throughput and job table size you will probably need to do some tuning to keep things running smoothly. But aft…

state machines to the rescue, ie i think the nature of asynchronous processing requires that we design for good/safe intermediate states.

Re: I’m leaving Redis for SolidQueue

#125

Post by yawboakye »

nick__m wrote: post #48
yawboakye wrote: post #38

Earlier quoted context omitted.

got it. is it necessary, then, to couple queue db with app db? if answer is no then we can’t make a necessity argument here, unfortunately.

Frequently you have to couple the transactional state of the queue db and the app db, colocating them is the simplest way to achieve that without resorting to distributed transactions or patterns that involve orchestrated compensation actions.

that’s setting yourself up for trouble, imo. intermediate states solve this problem, and economically. for mature production system see temporal[0]. their magic sauce is good intermediate states.

[0]: https://temporal.io/

Re: I’m leaving Redis for SolidQueue

#126

Post by stock_toaster »

rajaravivarma_r wrote: post #15

The one use case where a DB backed queue will fail for sure is when the payload is large. For example, you queue a large JSON payload to be picked up by a worker and process it, then the DB writing overhead itself makes a background worker useless. I've benchmarked Redis (Sidekiq), Postgres (using GoodJob) and SQLite (SolidQueue), Redis beats everything else for the above usecase. SolidQueue backed by SQLite may be g…

> The one use case where a DB backed queue will fail for sure is when the payload is large. For example, you queue a large JSON payload to be picked up by a worker and process it, then the DB writing overhead itself makes a background worker useless.

redis would suffer from the same issue. Possibly even more severely due to being memory constrained?

I'd probably just stuff the "large data" in s3 or something like that, and just include the reference/location of the data in the actual job itself, if it was big enough to cause problems.

Re: I’m leaving Redis for SolidQueue

#127

Post by parthdesai »

bgentry wrote: post #121
parthdesai wrote: post #120

Earlier quoted context omitted.

> None of that means Oban or similar queues don't/can't scale—it just means a high volume of NOTIFY doesn't scale Given the context of this post, it really does mean the same thing though?

No, I don't think so. Oban does not rely on a large volume of NOTIFY in order to process a large volume of jobs. The insert notifications are simply a latency optimization for lower volume environments, and for inserts can be fully disabled such that they're mainly used for control flow (canceling jobs, pausing queues, etc) and gossip among workers. River for example also uses LISTEN/NOTIFY for some stuff, but we def…

Does River without any extra configuration run into scaling issues at a certain point? If the answer is yes, then River doesn’t scale without optimization (Redis/Clustering in Oban’s case).

While the root cause might not be River/Oban, them not being scalable still holds true. It’s of extra importance given the context of this post is moving away from redis and to strictly a database for a queue system.

Re: I’m leaving Redis for SolidQueue

#128

Post by dns_snek »

cle wrote: post #55
dns_snek wrote: post #32

Earlier quoted context omitted.

Facing issues with 83 jobs per second (5k/min) sounds like an extreme misconfiguration. That's not high throughput at all and it shouldn't create any appreciable load on any database.

This comes up every time this conversation occurs. Yes, PG can theoretically handle just about anything with the right configuration, schema, architecture, etc. Finding that right configuration is not trivial. Even dedicated frameworks like Graphile struggle with it. My startup had the exact same struggles with PG and did the same migration to BullMQ bc we were sick of fiddling with it instead of solving business pro…

The issue is that "83 per second" is multiple orders of magnitude off the expected level of performance on any RDBMS running on anything resembling modern hardware.

I haven't worked with Graphile but this just doesn't pass the sniff test unless those 83 jobs per second are somehow translating into thousands of write transactions per second.

Their documentation has a performance section with a benchmark that claims to process 10k jobs per second on a pretty modest machine, as an indication.

Re: I’m leaving Redis for SolidQueue

#129

Post by NARKOZ »

robertlagrant wrote: post #60
jacob-s-son wrote: post #14

Every author of the free software obviously has rights to full control of the scope of their project. That being said, I regret that we have switched from good_job ( https://github.com/bensheldon/good_job ). The thing is - Basecamp is a MySQL shop and their policy is not to accept RDMS engine specific queries. You can see in their issues in Github that they try to stick "universal" SQL and are personally mostly conce…

> their policy is not to accept RDMS engine specific queries Why? Is it so they can switch in future?

Earlier Rails avoided database specific features so apps could stay portable using only ActiveRecord. Since then Rails has added much better PostgreSQL support: JSON/JSONB, hstore, array columns, GIN/GiST indexes.

Re: I’m leaving Redis for SolidQueue

#130

Post by skunkworker »

ecshafer wrote: post #110
otterley wrote: post #98

Earlier quoted context omitted.

> Redis is fundamentally the wrong storage system for a job queue when you have an RDBMS handy One could go one step further and say an RDBMS is fundamentally the wrong storage system for a job queue when you have a persistent, purpose-built message queue handy. Honestly, for most people, I'd recommend they just use their cloud provider's native message queue offering. On AWS, SQS is cheap, reliable, easy to start wi…

Rails shops seem to not like to use SQS/PubSub/Kafka/RabbitMQ for some reason. They seem to really like these worker tasks like SideKiq or SolidQueue. When I compare this with Java, C# or Python who all seem much more likely to use a separate message queue then have that handle the job queue.

Rails shops running on normal CRuby, have difficult in effectively scaling out multithreading due to the GVL lock. It's much easier to "scale" ruby using forking with sidekiq or multi process, and to have it consume data from a Redis list. It is possible to get around the GVL using JRuby, but that poses a different set of constraints and issues.

There is some definite blending of async messaging in the Ruby world though. I've seen connectors which take protobufs on a kafka topic and use sidekiq to fan out the work. With Redis (looking at sidekiq specifically) it becomes trivial to maintain the "current" working set with items popped out of the queue, with atomic commands like BLMOVE (formerly BRPOPLPUSH).

Kafka is taking an interesting turn however with the KIP-932 "Queues for Kafka" initiative. I personally believe it could eat RabbitMQ's lunch if done effectively. Allowing for multiple consumers, a "working set" of unack'ed data, without having to worry as much about the topic partition count.

Post reply on HN