Earlier quoted context omitted.
You're in luck, the article speaks about that at length!
Sorry, I went full typical HN commenter stereotype :-)
I do it all the time too.
I’m leaving Redis for SolidQueue
91–100 of 146 posts
Re: I’m leaving Redis for SolidQueue
#92Every author of the free software obviously has rights to full control of the scope of their project. That being said, I regret that we have switched from good_job ( https://github.com/bensheldon/good_job ). The thing is - Basecamp is a MySQL shop and their policy is not to accept RDMS engine specific queries. You can see in their issues in Github that they try to stick "universal" SQL and are personally mostly conce…
Can you be more specific about the issues you have run into that make you advise GoodJob over SolidQueue? I am (and have been for a while, not in a hurry) considering them each as a move off resque. The main blocker for me with GoodJob is that it uses certain pg-specific features in a way that makes it incompatible with transaction-mode in pgbounder -- that is, it requires persistent sessions. Which is annoying, and…
The first one that jumps out at me when I've evaluated it are batches (a Sidekiq Pro feature, though there are some Sidekiq plugins that support the same)
Re: I’m leaving Redis for SolidQueue
#93Earlier quoted context omitted.
Can you be more specific about the issues you have run into that make you advise GoodJob over SolidQueue? I am (and have been for a while, not in a hurry) considering them each as a move off resque. The main blocker for me with GoodJob is that it uses certain pg-specific features in a way that makes it incompatible with transaction-mode in pgbounder -- that is, it requires persistent sessions. Which is annoying, and…
The first one that jumps out at me when I've evaluated it are batches (a Sidekiq Pro feature, though there are some Sidekiq plugins that support the same)
Ah neat, I didn't realize GoodJob had batches, great.
Re: I’m leaving Redis for SolidQueue
#94Every time some production environment can be simplified, it is good news in my opinion. The ideal situation with Rails would be if there is a simple way to switch back to Redis, so that you can start simple, and as soon as you hit some fundamental issue with using SolidQueue (mostly scalability, I guess, in environments where the queue is truly stressed -- and you don't want to have a Postgres scalability problem be…
The primary pain point I see here is if devs lean into transactions such that their job is only created together with the everything else that happened. Losing that guarantee can make the eventual migration harder, even if that migration is to a different postgres instance than the primary db.
You can look at it both ways.
Using the database as a queue, you no longer need to setup transaction triggers to fire your tasks, you can have atomic guarantees that the data and the task were created successfully, or nothing was created.
Re: I’m leaving Redis for SolidQueue
#95Ignoring how it works, there are a a solid handful of great features you get out of the box with Solid + Active Job that don't exist w/just using Sidekiq, even through the Active Job adapter.
Re: I’m leaving Redis for SolidQueue
#96Earlier quoted context omitted.
> it shouldn’t be the same as the production database Why is that?
Here's an example from the circleci incident https://status.circleci.com/incidents/hr0mm9xmm3x6 and a good analysis by a flicker engineer who ran into similar issues https://blog.mihasya.com/2015/07/19/thoughts-evoked-by-circl...
CircleCI and Flickr are both pretty big systems. There are tons of businesses that will never operate at that scale.
Re: I’m leaving Redis for SolidQueue
#97Earlier quoted context omitted.
If you’re tied so tight to MySQL that you’re labeled a “MySQL shop” then it seems logical to use MySQL specific features. I must be missing something.
It's reasonable for basecamp, but the complaint of GP is that basecamp controls what is the Rails standard/default solution intended to be useful for multiple rdbms, without being willing to put rdbms-specific logic in rdbms-specific adapters.
Ooh. That's a dealbreaker, ladies!
Re: I’m leaving Redis for SolidQueue
#98Redis is fundamentally the wrong storage system for a job queue when you have an RDBMS handy. This is not new information. You still might want to split the job queue onto its own DB server when things start getting busy, though. For caching, though, I wouldn’t drop Redis so fast. As a in-memory cache, the ops overhead of running Redis is a lot lower. You can even ignore HA for most use cases. Source: I helped design…
> Redis is fundamentally the wrong storage system for a job queue when you have an RDBMS handy
One could go one step further and say an RDBMS is fundamentally the wrong storage system for a job queue when you have a persistent, purpose-built message queue handy.
Honestly, for most people, I'd recommend they just use their cloud provider's native message queue offering. On AWS, SQS is cheap, reliable, easy to start with, and gives you plenty of room to grow. GCP PubSub and Azure Storage Queues are probably similar in these regards.
Unless managing queues is your business, I wouldn't make it your problem. Hand that undifferentiated heavy lifting off.
Re: I’m leaving Redis for SolidQueue
#99Earlier quoted context omitted.
Here's an example from the circleci incident https://status.circleci.com/incidents/hr0mm9xmm3x6 and a good analysis by a flicker engineer who ran into similar issues https://blog.mihasya.com/2015/07/19/thoughts-evoked-by-circl...
CircleCI and Flickr are both pretty big systems. There are tons of businesses that will never operate at that scale.
I don't disagree with that call out. However, we've been through these discussions many times over the years. The solid queue of yesteryear was delayed_job which was originally created by Shopify's CEO.
https://github.com/tobi/delayed_job
Shopify however grew (as many others) and we saw a host of blog posts and talks about moving away from DB queues to Redis, RabbitMQ, Kafka etc. We saw posts about moving from Resque to SideKiq etc. All this to day storing a task queue in the db has always been the naive approach. Engineers absolutely shouldn't be shocked that approach isn't viable at higher workloads.
Re: I’m leaving Redis for SolidQueue
#100I'm not sure how similar they are internally (I suspect: quite), but I use Django-Q2's database broker to similar effect. More simple = better!