Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I’m leaving Redis for SolidQueue

simplethread.com

Post reply on HN
Print view View original

51–60 of 146 posts

Re: I’m leaving Redis for SolidQueue

#51

Post by speleding »

We've been storing jobs in the DB long before SolidQueue appeared. One major advantage is that we can snapshot the state of the system (or one customer account) to our dev environment and get to see it exactly as it is in production.

We still keep rate limiters in Redis though, it would be pretty easy for some scanner to overload the DB if every rogue request would need a round trip to the DB before being processed. Because we only store ephemeral data in Redis it does not need backups.

Re: I’m leaving Redis for SolidQueue

#52

Post by skywhopper »

Redis is fundamentally the wrong storage system for a job queue when you have an RDBMS handy. This is not new information. You still might want to split the job queue onto its own DB server when things start getting busy, though.

For caching, though, I wouldn’t drop Redis so fast. As a in-memory cache, the ops overhead of running Redis is a lot lower. You can even ignore HA for most use cases.

Source: I helped design and run a multi-tiered Redis caching architecture for a Rails-based SaaS serving millions of daily users, coordinating shared data across hundreds of database clusters and thousands of app servers across a dozen AWS regions, with separate per-host, per-cluster, per-region, and global cache layers.

We used Postgres for the job queues, though. Entirely separate from the primary app DBs.

Re: I’m leaving Redis for SolidQueue

#53

Post by patwolf »

I've been looking at DBOS for queuing and other scheduling tasks in a nodejs app. However, it only works with Postgres, and that means I can't use it in web or mobile with sqlite. I like that SolidQueue works with multiple databases. Too bad it needs rails.

Re: I’m leaving Redis for SolidQueue

#54

Post by brightball »

jacob-s-son wrote: post #14

Every author of the free software obviously has rights to full control of the scope of their project. That being said, I regret that we have switched from good_job ( https://github.com/bensheldon/good_job ). The thing is - Basecamp is a MySQL shop and their policy is not to accept RDMS engine specific queries. You can see in their issues in Github that they try to stick "universal" SQL and are personally mostly conce…

Agreed. good_job is the ideal approach to a PG backed queue.

Re: I’m leaving Redis for SolidQueue

#55

Post by cle »

dns_snek wrote: post #32
ashniu123 wrote: post #6

For Node.js, my startup used to use [Graphile Worker]( https://github.com/graphile/worker ) which utilised the same "SKIP LOCKED" mechanism under the hood. We ran into some serious issues in high throughput scenarios (~2k jobs/min currently, and ~5k job/min during peak hours) and switched to Redis+BullMQ and have never looked back ever since. Our bottleneck was Postgres performance. I wonder if SolidQueue runs into s…

Facing issues with 83 jobs per second (5k/min) sounds like an extreme misconfiguration. That's not high throughput at all and it shouldn't create any appreciable load on any database.

This comes up every time this conversation occurs.

Yes, PG can theoretically handle just about anything with the right configuration, schema, architecture, etc.

Finding that right configuration is not trivial. Even dedicated frameworks like Graphile struggle with it.

My startup had the exact same struggles with PG and did the same migration to BullMQ bc we were sick of fiddling with it instead of solving business problems. We are very glad we migrated off of PG for our work queues.

Re: I’m leaving Redis for SolidQueue

#56

Post by saberd »

cies wrote: post #29
dependency_2x wrote: post #5

Postgres will eat the world

I use PGQM and PG_CRON now... Not looking back. The MySQL + Redis + AWS' elasti-cron (or whatever) was a ghetto compared to Postgres.

We use pgmq with the pgmq-go client, and it has clients in many different languages, it's amazing. The queues persist on disk and visualizations of queues can easily be made with grafana or just pure sql requests. The fact that the queues lives in the same database as all the other data is also a huge benefit if the 5-15ms time penalty is not an issue.

Re: I’m leaving Redis for SolidQueue

#57

Post by dzonga »

zihotki wrote: post #20
rajaravivarma_r wrote: post #15

The one use case where a DB backed queue will fail for sure is when the payload is large. For example, you queue a large JSON payload to be picked up by a worker and process it, then the DB writing overhead itself makes a background worker useless. I've benchmarked Redis (Sidekiq), Postgres (using GoodJob) and SQLite (SolidQueue), Redis beats everything else for the above usecase. SolidQueue backed by SQLite may be g…

Using Redis to store large queue payloads is usually a bad practice. Redis memory is finite.

this!! 100%.

pass around ID's

Re: I’m leaving Redis for SolidQueue

#59

Post by erispoe »

yawboakye wrote: post #11
antirez wrote: post #4

Every time some production environment can be simplified, it is good news in my opinion. The ideal situation with Rails would be if there is a simple way to switch back to Redis, so that you can start simple, and as soon as you hit some fundamental issue with using SolidQueue (mostly scalability, I guess, in environments where the queue is truly stressed -- and you don't want to have a Postgres scalability problem be…

the problem i see here is that we end up treating the background job/task processor as part of the production system (e.g. the server that responds to requests, in the case of a web application) instead of a separate standalone thing. rails doesn’t make this distinction clear enough. it’s okay to back your tasks processor with a pg database (e.g. river[0]) but, as you indirectly pointed out, it shouldn’t be the same…

> it shouldn’t be the same as the production database

Why is that?

Re: I’m leaving Redis for SolidQueue

#60

Post by robertlagrant »

jacob-s-son wrote: post #14

Every author of the free software obviously has rights to full control of the scope of their project. That being said, I regret that we have switched from good_job ( https://github.com/bensheldon/good_job ). The thing is - Basecamp is a MySQL shop and their policy is not to accept RDMS engine specific queries. You can see in their issues in Github that they try to stick "universal" SQL and are personally mostly conce…

> their policy is not to accept RDMS engine specific queries

Why? Is it so they can switch in future?

Post reply on HN