Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I’m leaving Redis for SolidQueue

simplethread.com

Post reply on HN
Print view View original

101–110 of 146 posts

Re: I’m leaving Redis for SolidQueue

#101

Post by lta »

jacob-s-son wrote: post #14

Every author of the free software obviously has rights to full control of the scope of their project. That being said, I regret that we have switched from good_job ( https://github.com/bensheldon/good_job ). The thing is - Basecamp is a MySQL shop and their policy is not to accept RDMS engine specific queries. You can see in their issues in Github that they try to stick "universal" SQL and are personally mostly conce…

I have no opinion whatsoever yet on SolidQueue, but I'm having a blast with good job. Stuff works pretty well.

Re: I’m leaving Redis for SolidQueue

#102

Post by bgentry »

runako wrote: post #75
yawboakye wrote: post #11

Earlier quoted context omitted.

the problem i see here is that we end up treating the background job/task processor as part of the production system (e.g. the server that responds to requests, in the case of a web application) instead of a separate standalone thing. rails doesn’t make this distinction clear enough. it’s okay to back your tasks processor with a pg database (e.g. river[0]) but, as you indirectly pointed out, it shouldn’t be the same…

SolidQueue uses its own db configuration. > it shouldn’t be the same as the production database This is highly dependent on the application (scale, usage, phase of lifecycle, etc.)

Yeah, River generally recommends this pattern as well (River co-author here :)

To get the benefits of transactional enqueueing you generally need to commit the jobs transactionally with other database changes. https://riverqueue.com/docs/transactional-enqueueing

It does not scale forever, and as you grow in throughput and job table size you will probably need to do some tuning to keep things running smoothly. But after the amount of time I've spent in my career tracking down those numerous distributed systems issues arising from a non-transactional queue, I've come to believe this model is the right starting point for the vast majority of applications. That's especially true given how high the performance ceiling is on newer / more modern job queues and hardware relative to where things were 10+ years ago.

If you are lucky enough to grow into the range of many thousands of jobs per second then you can start thinking about putting in all that extra work to build a robust multi-datastore queueing system, or even just move specific high-volume jobs into a dedicated system. Most apps will never hit this point, but if you do you'll have deferred a ton of complexity and pain until it's truly justified.

Re: I’m leaving Redis for SolidQueue

#103

Post by parthdesai »

victorbjorklund wrote: post #88
parthdesai wrote: post #69

Earlier quoted context omitted.

Funny you mention Oban, we do use it at work as well, and first thing Oban tells you is to either use Redis as a notifier or resort to polling for jobs and just not notify. https://hexdocs.pm/oban/scaling.html

I don't think that Oban is telling you to always use Redis. I think what they're saying is if you reach a certain scale where you're feeling the pain of the default notifier you could use Oban.Notifiers.PG as long as your application is running as a cluster. If you don't run it as a cluster, then you might have to reach for Redis. But then it's more about not running a cluster.

> For people that does not think it scales

You started your comment with that

Re: I’m leaving Redis for SolidQueue

#104

Post by perfmode »

kid64 wrote: post #97
jrochkind1 wrote: post #72

Earlier quoted context omitted.

It's reasonable for basecamp, but the complaint of GP is that basecamp controls what is the Rails standard/default solution intended to be useful for multiple rdbms, without being willing to put rdbms-specific logic in rdbms-specific adapters.

Ooh. That's a dealbreaker, ladies!

I thought I was the only one who remembers this one.

Re: I’m leaving Redis for SolidQueue

#105

Post by erikpukinskis »

formerly_proven wrote: post #26
victorbjorklund wrote: post #9

For people that does not think it scales. A similar implementation in Elixir is Oban. Their benchmark shows a million jobs per minute on a single node (and I am sure it could be increased further with more optimizations). I bet 99,99999% of apps have less than a million background jobs per minute. https://oban.pro/articles/one-million-jobs-a-minute-with-oba...

This benchmark is probably as far removed from how applications use task queues as it could possibly be. The headline is "1 million jobs per minute", which is true. However... - this is achieved by queuing batches of 5000 jobs , so on the queue side this is actually not 1 million TPS, but rather 200 TPS. I've never seen any significant batching of background job creation. - the dispatch is also batched to a few hundr…

Also worth noting that it’s often not single-node performance that caps throughput… it’s replication.

Databases are pretty good at quickly adding and removing lots of rows. But even if you can keep up with churning through 1000 rows/second, with batching or whatever, you still need to replicate 1000 rows/second do your failover nodes.

That’s the big win for queues over a relational db here: queues have ways to efficiently replicate without copying the entire work queue across instances.

Re: I’m leaving Redis for SolidQueue

#106

Post by bgentry »

parthdesai wrote: post #69
victorbjorklund wrote: post #9

For people that does not think it scales. A similar implementation in Elixir is Oban. Their benchmark shows a million jobs per minute on a single node (and I am sure it could be increased further with more optimizations). I bet 99,99999% of apps have less than a million background jobs per minute. https://oban.pro/articles/one-million-jobs-a-minute-with-oba...

Funny you mention Oban, we do use it at work as well, and first thing Oban tells you is to either use Redis as a notifier or resort to polling for jobs and just not notify. https://hexdocs.pm/oban/scaling.html

This is largely because LISTEN/NOTIFY has an implementation which uses a global lock. At high volume this obviously breaks down: https://www.recall.ai/blog/postgres-listen-notify-does-not-s...

None of that means Oban or similar queues don't/can't scale—it just means a high volume of NOTIFY doesn't scale, hence the alternative notifiers and the fact that most of its job processing doesn't depend on notifications at all.

There are other reasons Oban recommends a different notifier per the doc link above:

> That keeps notifications out of the db, reduces total queries, and allows larger messages, with the tradeoff that notifications from within a database transaction may be sent even if the transaction is rolled back

Re: I’m leaving Redis for SolidQueue

#107

Post by film42 »

jacob-s-son wrote: post #14

Every author of the free software obviously has rights to full control of the scope of their project. That being said, I regret that we have switched from good_job ( https://github.com/bensheldon/good_job ). The thing is - Basecamp is a MySQL shop and their policy is not to accept RDMS engine specific queries. You can see in their issues in Github that they try to stick "universal" SQL and are personally mostly conce…

I made the switch on a new project and I don't regret it but it's still early days software despite the marketing. Concurrency control is fantastic, but it doesn't always work. I've woken up to see all threads occupied with a job that should be concurrency of 1.

I've also run into issues where a db connection pool is filled up and solid queue silently fails. No error or logs, just stops polling forever until manual restart. Far from ideal.

But, I can live with it. I am going for minimal maintenance, and the ability to run solid queue under puma inside rails on cloud run is just so easy. Having ~3 solid queue related issues a year is acceptable for my use case, but that doesn't mean it will be ok for others.

Re: I’m leaving Redis for SolidQueue

#108

Post by cortesoft »

jacob-s-son wrote: post #14

Every author of the free software obviously has rights to full control of the scope of their project. That being said, I regret that we have switched from good_job ( https://github.com/bensheldon/good_job ). The thing is - Basecamp is a MySQL shop and their policy is not to accept RDMS engine specific queries. You can see in their issues in Github that they try to stick "universal" SQL and are personally mostly conce…

Don't you think the officially supported Rails modules should work with all the RDMS engines that Rails supports? What would a MySQL based Rails app use if the official supported module didn't support it?

Re: I’m leaving Redis for SolidQueue

#109

Post by yandrypozo »

ivolimmen wrote: post #65

Exactly what https://www.amazingcto.com/postgres-for-everything/ says; keep it simpel and use PostgreSQL.

I love the idea of PG for everything, but every time I suggest it I get the same answer "When you're a hammer, everything looks like a nail" which makes sense to me, but not sure how to give a good answer to that phrase :(

Re: I’m leaving Redis for SolidQueue

#110

Post by ecshafer »

otterley wrote: post #98
skywhopper wrote: post #52

Redis is fundamentally the wrong storage system for a job queue when you have an RDBMS handy. This is not new information. You still might want to split the job queue onto its own DB server when things start getting busy, though. For caching, though, I wouldn’t drop Redis so fast. As a in-memory cache, the ops overhead of running Redis is a lot lower. You can even ignore HA for most use cases. Source: I helped design…

> Redis is fundamentally the wrong storage system for a job queue when you have an RDBMS handy One could go one step further and say an RDBMS is fundamentally the wrong storage system for a job queue when you have a persistent, purpose-built message queue handy. Honestly, for most people, I'd recommend they just use their cloud provider's native message queue offering. On AWS, SQS is cheap, reliable, easy to start wi…

Rails shops seem to not like to use SQS/PubSub/Kafka/RabbitMQ for some reason. They seem to really like these worker tasks like SideKiq or SolidQueue. When I compare this with Java, C# or Python who all seem much more likely to use a separate message queue then have that handle the job queue.
Post reply on HN