Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I’m leaving Redis for SolidQueue

simplethread.com

Post reply on HN
Print view View original

111–120 of 146 posts

Re: I’m leaving Redis for SolidQueue

#111

Post by dynamicentropy »

Django is slowly catching up with Rails by adding support for a unified task interface in Django 6.0, but less feature rich than Rails' ActiveJob.

There are already a few implementations, and the reference one (django-tasks), even has a database-backed task backend that also uses FOR UPDATE SKIP LOCKED to control concurrency. With django-tasks and a few extra packages you can already get quite far compared to what Solid Queue provides, except maybe for features like concurrency controls and using a separate database for the queues.

I really enjoyed learning about the internals of Solid Queue, to the point that I decided to port it to Django [1]. It provides all of Solid Queue's features, except for retries on errors which is something that IMHO should be provided by the Django task interface, like Active Job does.

[1]: https://github.com/knifecake/steady-queue

Re: I’m leaving Redis for SolidQueue

#112

Post by pm90 »

One other concern: if you ever have to deploy in another cloud, there are all kinds of issues with authentication and version support. e.g. azure doesn’t support the latest redis version, GCP MemoryStore forbids password only login for Redis Clusters etc. The infrastructure complexity can be high (albeit manageable).

Re: I’m leaving Redis for SolidQueue

#113

Post by riffraff »

cortesoft wrote: post #108
jacob-s-son wrote: post #14

Every author of the free software obviously has rights to full control of the scope of their project. That being said, I regret that we have switched from good_job ( https://github.com/bensheldon/good_job ). The thing is - Basecamp is a MySQL shop and their policy is not to accept RDMS engine specific queries. You can see in their issues in Github that they try to stick "universal" SQL and are personally mostly conce…

Don't you think the officially supported Rails modules should work with all the RDMS engines that Rails supports? What would a MySQL based Rails app use if the official supported module didn't support it?

I think the suggestion is that one can have rdbms-specific optimizations while still keeping a standards-compliant base implementation.

Both MySQL and Postgresql could get their own optimizations.

Re: I’m leaving Redis for SolidQueue

#114

Post by another_twist »

antirez wrote: post #4

Every time some production environment can be simplified, it is good news in my opinion. The ideal situation with Rails would be if there is a simple way to switch back to Redis, so that you can start simple, and as soon as you hit some fundamental issue with using SolidQueue (mostly scalability, I guess, in environments where the queue is truly stressed -- and you don't want to have a Postgres scalability problem be…

Since you're here - https://redis.io/docs/latest/operate/oss_and_stack/managemen...

In AOF mode does Redis write all changes to a WAL ? Is this paired with periodic snapshotting to prevent the log from growing too large ? Does this work in distributed mode or is this single node thing ?

Re: I’m leaving Redis for SolidQueue

#115

Post by cortesoft »

riffraff wrote: post #113
cortesoft wrote: post #108

Earlier quoted context omitted.

Don't you think the officially supported Rails modules should work with all the RDMS engines that Rails supports? What would a MySQL based Rails app use if the official supported module didn't support it?

I think the suggestion is that one can have rdbms-specific optimizations while still keeping a standards-compliant base implementation. Both MySQL and Postgresql could get their own optimizations.

Oh, weird, they won't even allow functionally equivalent optimizations? That seems silly.

I was responding to Rails not officially supporting good_job, though, which appears to be a Postgres-only tool.

Re: I’m leaving Redis for SolidQueue

#116

Post by vjerancrnjak »

Not sure how that helps. They mention SKIP LOCKED but then show a job with 15 minute duration.

How will you hold an open transaction for 15 minutes without seriously compromising the performance of the database?

Allowing people to do this easily will just result in an antipattern with horrible performance and reliability once network starts to randomly end transactions. Pretty sure, just like Python can’t figure out connection to the db was closed, so can’t Rails.

Once people add transaction pinning proxies, and try to actually get most performance from db, these kind of locking mechanisms that require a long running open transaction start falling apart.

Edit: I must have misunderstood and it is a lease.

Re: I’m leaving Redis for SolidQueue

#117

Post by byroot »

watercolorblind wrote: post #41
antirez wrote: post #4

Every time some production environment can be simplified, it is good news in my opinion. The ideal situation with Rails would be if there is a simple way to switch back to Redis, so that you can start simple, and as soon as you hit some fundamental issue with using SolidQueue (mostly scalability, I guess, in environments where the queue is truly stressed -- and you don't want to have a Postgres scalability problem be…

The primary pain point I see here is if devs lean into transactions such that their job is only created together with the everything else that happened. Losing that guarantee can make the eventual migration harder, even if that migration is to a different postgres instance than the primary db.

That's also something Rails helps abstract away by automatically deferring enqueues to after the transaction completed.

Even SolidQueue behave that way by default.

https://github.com/rails/rails/pull/51426

Re: I’m leaving Redis for SolidQueue

#118

Post by vjerancrnjak »

dns_snek wrote: post #32
ashniu123 wrote: post #6

For Node.js, my startup used to use [Graphile Worker]( https://github.com/graphile/worker ) which utilised the same "SKIP LOCKED" mechanism under the hood. We ran into some serious issues in high throughput scenarios (~2k jobs/min currently, and ~5k job/min during peak hours) and switched to Redis+BullMQ and have never looked back ever since. Our bottleneck was Postgres performance. I wonder if SolidQueue runs into s…

Facing issues with 83 jobs per second (5k/min) sounds like an extreme misconfiguration. That's not high throughput at all and it shouldn't create any appreciable load on any database.

They probably did not batch. It’s realistic they will have issues if code is written to handle 1 job at a time and needs to make several roundtrips to the same db inside the same locking transaction.

Leases exist for a reason.

Re: I’m leaving Redis for SolidQueue

#119

Post by otterley »

ecshafer wrote: post #110
otterley wrote: post #98

Earlier quoted context omitted.

> Redis is fundamentally the wrong storage system for a job queue when you have an RDBMS handy One could go one step further and say an RDBMS is fundamentally the wrong storage system for a job queue when you have a persistent, purpose-built message queue handy. Honestly, for most people, I'd recommend they just use their cloud provider's native message queue offering. On AWS, SQS is cheap, reliable, easy to start wi…

Rails shops seem to not like to use SQS/PubSub/Kafka/RabbitMQ for some reason. They seem to really like these worker tasks like SideKiq or SolidQueue. When I compare this with Java, C# or Python who all seem much more likely to use a separate message queue then have that handle the job queue.

I've also noticed that they conflate the notion of workers, queues, and message busses. A worker handles asynchronous tasks, but the means by which they communicate might be best served by either a queue or a message bus, depending on the specific needs. Tight coupling might be good for knocking out PoCs quickly, but once you have production-grade needs, the model begins to show its weaknesses.

Re: I’m leaving Redis for SolidQueue

#120

Post by parthdesai »

bgentry wrote: post #106
parthdesai wrote: post #69

Earlier quoted context omitted.

Funny you mention Oban, we do use it at work as well, and first thing Oban tells you is to either use Redis as a notifier or resort to polling for jobs and just not notify. https://hexdocs.pm/oban/scaling.html

This is largely because LISTEN/NOTIFY has an implementation which uses a global lock. At high volume this obviously breaks down: https://www.recall.ai/blog/postgres-listen-notify-does-not-s... None of that means Oban or similar queues don't/can't scale—it just means a high volume of NOTIFY doesn't scale, hence the alternative notifiers and the fact that most of its job processing doesn't depend on notifications at al…

> None of that means Oban or similar queues don't/can't scale—it just means a high volume of NOTIFY doesn't scale

Given the context of this post, it really does mean the same thing though?

Post reply on HN