Would be more useful as a report back with the switch a couple months behind, than as a "This is what I'm going to do"!
I’m leaving Redis for SolidQueue
71–80 of 146 posts
Re: I’m leaving Redis for SolidQueue
#72Every author of the free software obviously has rights to full control of the scope of their project. That being said, I regret that we have switched from good_job ( https://github.com/bensheldon/good_job ). The thing is - Basecamp is a MySQL shop and their policy is not to accept RDMS engine specific queries. You can see in their issues in Github that they try to stick "universal" SQL and are personally mostly conce…
If you’re tied so tight to MySQL that you’re labeled a “MySQL shop” then it seems logical to use MySQL specific features. I must be missing something.
It's reasonable for basecamp, but the complaint of GP is that basecamp controls what is the Rails standard/default solution intended to be useful for multiple rdbms, without being willing to put rdbms-specific logic in rdbms-specific adapters.
Re: I’m leaving Redis for SolidQueue
#73Exactly what https://www.amazingcto.com/postgres-for-everything/ says; keep it simpel and use PostgreSQL.
Isn't Redis just a lot less relevant these days since enterprise NVME storage is so ridiculously fast? How much latency could you really be saving versus introducing complexity? But I am not a storage/backend engineer, so maybe I don't understand the target use of Redis.
You'll be amazed on what the new breed of engineers are using Redis for. I personally saw an entire backend database using Redis with RDB+AOF on. If you redis-cli into the server, you can't understand anything because you need to know the schema to make sense of it all.
Re: I’m leaving Redis for SolidQueue
#74Every author of the free software obviously has rights to full control of the scope of their project. That being said, I regret that we have switched from good_job ( https://github.com/bensheldon/good_job ). The thing is - Basecamp is a MySQL shop and their policy is not to accept RDMS engine specific queries. You can see in their issues in Github that they try to stick "universal" SQL and are personally mostly conce…
Can you be more specific about the issues you have run into that make you advise GoodJob over SolidQueue?
I am (and have been for a while, not in a hurry) considering them each as a move off resque.
The main blocker for me with GoodJob is that it uses certain pg-specific features in a way that makes it incompatible with transaction-mode in pgbounder -- that is, it requires persistent sessions. Which is annoying, and is done to get some upper-end performance improvements that I don't think matter for my or most scales. Otherwise, I much prefer GoodJob's development model, trust the maintainer's judgement more, find the code more readable, etc. -- but that's a big But for me.
Re: I’m leaving Redis for SolidQueue
#75Every time some production environment can be simplified, it is good news in my opinion. The ideal situation with Rails would be if there is a simple way to switch back to Redis, so that you can start simple, and as soon as you hit some fundamental issue with using SolidQueue (mostly scalability, I guess, in environments where the queue is truly stressed -- and you don't want to have a Postgres scalability problem be…
the problem i see here is that we end up treating the background job/task processor as part of the production system (e.g. the server that responds to requests, in the case of a web application) instead of a separate standalone thing. rails doesn’t make this distinction clear enough. it’s okay to back your tasks processor with a pg database (e.g. river[0]) but, as you indirectly pointed out, it shouldn’t be the same…
SolidQueue uses its own db configuration.
> it shouldn’t be the same as the production database
This is highly dependent on the application (scale, usage, phase of lifecycle, etc.)
Re: I’m leaving Redis for SolidQueue
#76Earlier quoted context omitted.
It’s not necessary to separate queue db from application db.
got it. is it necessary, then, to couple queue db with app db? if answer is no then we can’t make a necessity argument here, unfortunately.
solid_queue by default prefers you use a different db than app db, and will generate that out of the box (also by default with sqlite3, which, separate discussion) but makes it possible, and fairly smooth, to configure to use the same db.
Personally, I prefer the same db unless I were at a traffic scale where splitting them is necessary for load.
One advantage of same db is you can use db transaction control over enqueing jobs and app logic too, when they are dependent. But that's not the main advantage to me, I don't actually need that. I just prefer the simplicity, and as someone else said above, prefer not having to reconcile app db state with queue state if they are separate and only ONE goes down. Fewer moving parts are better in the apps I work on which are relatively small-scale, often "enterprise", etc.
Re: I’m leaving Redis for SolidQueue
#77Exactly what https://www.amazingcto.com/postgres-for-everything/ says; keep it simpel and use PostgreSQL.
Isn't Redis just a lot less relevant these days since enterprise NVME storage is so ridiculously fast? How much latency could you really be saving versus introducing complexity? But I am not a storage/backend engineer, so maybe I don't understand the target use of Redis.
> But I am not a storage/backend engineer, so maybe I don't understand the target use of Redis.
We use it to broadcast messages across horizontally scaled services.
Works fine, probably a better tool out there for the job with better delivery guarantees, but the decision was taken many years ago, and no point in changing something that just works.
It's also language agnostic, which really helps.
We use ElasticCache (Valkey i suppose), so most of the articles points are moot for our use.
Were we to implement it from scratch today, we might look for better delivery guarantees, or we might just use what we already know works.
Re: I’m leaving Redis for SolidQueue
#78Not a ruby shop here so it's not directly comparable, but I'm very happy with beanstalkd as a minimalistic job queue. We're on mysql for historical reasons, and it didn't support SKIP LOCKED at the time, so we had to add another tool.
I pulled beanstalkd into a legacy PHP/MySQL application several years back and was very pleased with it. It's probably not the right choice for a modern Rails application, but if you already don't have a framework, it's a straightforward solution to drop in.
Re: I’m leaving Redis for SolidQueue
#79Earlier quoted context omitted.
In my experience you want job parameters to be one, maybe two ids. Do you have a real world example where that is not the case?
I'm guessing you're with that adding indirection for what you're actually processing, in that case? So I guess the counter-case would be when you don't want/need that indirection. If I understand what you're saying, is that you'll instead of doing: - Create job with payload (maybe big) > Put in queue > Let worker take from queue > Done You're suggesting: - Create job with ID of payload (stored elsewhere) > Put in que…
> I can definitively see use cases for both
Me too, I was just wondering if you have any real world examples of a project with a large payload.
Re: I’m leaving Redis for SolidQueue
#80Postgres will eat the world
At least until people - in a couple of years - figure out that the "Postgres for everything" fad was just as much of a bad idea as "MongoDB for everything" and "Put Redis into everything".
It's not "Postgres for everything", it's "Postgres by default". Nobody is saying you should replace your billion-message-per-second Kafka cluster (or whatever) with Postgres, but plenty of people are saying "don't start with a Kafka cluster when you have two messages a day", which is a much better idea than "MongoDB by default".