Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I’m leaving Redis for SolidQueue

simplethread.com

Post reply on HN
Print view View original

31–40 of 146 posts

Re: I’m leaving Redis for SolidQueue

#31

Post by ddorian43 »

rajaravivarma_r wrote: post #15

The one use case where a DB backed queue will fail for sure is when the payload is large. For example, you queue a large JSON payload to be picked up by a worker and process it, then the DB writing overhead itself makes a background worker useless. I've benchmarked Redis (Sidekiq), Postgres (using GoodJob) and SQLite (SolidQueue), Redis beats everything else for the above usecase. SolidQueue backed by SQLite may be g…

> Redis beats everything else for the above usecase.

Reminds me of Antirez blog post that when Redis is configured for durability it becomes like/slower than postgresql http://oldblog.antirez.com/post/redis-persistence-demystifie...

Re: I’m leaving Redis for SolidQueue

#32

Post by dns_snek »

ashniu123 wrote: post #6

For Node.js, my startup used to use [Graphile Worker]( https://github.com/graphile/worker ) which utilised the same "SKIP LOCKED" mechanism under the hood. We ran into some serious issues in high throughput scenarios (~2k jobs/min currently, and ~5k job/min during peak hours) and switched to Redis+BullMQ and have never looked back ever since. Our bottleneck was Postgres performance. I wonder if SolidQueue runs into s…

Facing issues with 83 jobs per second (5k/min) sounds like an extreme misconfiguration. That's not high throughput at all and it shouldn't create any appreciable load on any database.

Re: I’m leaving Redis for SolidQueue

#33

Post by michaelbuckbee »

rajaravivarma_r wrote: post #15

The one use case where a DB backed queue will fail for sure is when the payload is large. For example, you queue a large JSON payload to be picked up by a worker and process it, then the DB writing overhead itself makes a background worker useless. I've benchmarked Redis (Sidekiq), Postgres (using GoodJob) and SQLite (SolidQueue), Redis beats everything else for the above usecase. SolidQueue backed by SQLite may be g…

FWIW, Sidekiq docs strongly suggest only passing around primary keys or identifiers for jobs.

Re: I’m leaving Redis for SolidQueue

#34

Post by touisteur »

croes wrote: post #21
ckbkr10 wrote: post #17

Comparing Redis to SQL is kinda off topic. Sure you can replace the one with the other but then we are talking about completely different concepts aren't we? When all we are talking about is "good enough" the bar is set at a whole different level.

Maybe Redis is just overkill

I wish you'd have expanded on that. I almost always learn about some interesting lower-level tech through people trying to avoid a full-featured heavy-for-their-use-case tool or system.

Re: I’m leaving Redis for SolidQueue

#35

Post by michaelbuckbee »

ckbkr10 wrote: post #17

Comparing Redis to SQL is kinda off topic. Sure you can replace the one with the other but then we are talking about completely different concepts aren't we? When all we are talking about is "good enough" the bar is set at a whole different level.

I wrote this article about migrating from Redis to SQLite for a particular scenario and the tradeoffs involved.

To be clear, I think the most important thing is understanding the performance characteristics of each technology enough that you can make good choices for your particular scenario.

https://wafris.org/blog/rearchitecting-for-sqlite

Re: I’m leaving Redis for SolidQueue

#36

Post by michaelbuckbee »

jjgreen wrote: post #3

Nice article, I'm just productionising a Rails 8 app and was wondering whether it was worth switching from SolidQueue (which has given me no stress in dev) to Redis ... maybe not.

Unless you hit a performance wall with Postgres or absolutely need Batch capability you've probably got a very large runway with SolidQueue.

Re: I’m leaving Redis for SolidQueue

#37

Post by speed_spread »

KolmogorovComp wrote: post #23

> Job latency under 1ms is critical to your business. This is a real and pressing concern for real-time bidding, high frequency trading (HFT), and other applications in the same ilk. From TFA. Are there really people using Rails for HFT?

Trading engine will not run Rails for sure but the web UI to monitor and control trades might do.

Re: I’m leaving Redis for SolidQueue

#38

Post by yawboakye »

andrewstuart wrote: post #28
yawboakye wrote: post #11

Earlier quoted context omitted.

the problem i see here is that we end up treating the background job/task processor as part of the production system (e.g. the server that responds to requests, in the case of a web application) instead of a separate standalone thing. rails doesn’t make this distinction clear enough. it’s okay to back your tasks processor with a pg database (e.g. river[0]) but, as you indirectly pointed out, it shouldn’t be the same…

It’s not necessary to separate queue db from application db.

got it. is it necessary, then, to couple queue db with app db? if answer is no then we can’t make a necessity argument here, unfortunately.

Re: I’m leaving Redis for SolidQueue

#39

Post by epolanski »

ddorian43 wrote: post #31
rajaravivarma_r wrote: post #15

The one use case where a DB backed queue will fail for sure is when the payload is large. For example, you queue a large JSON payload to be picked up by a worker and process it, then the DB writing overhead itself makes a background worker useless. I've benchmarked Redis (Sidekiq), Postgres (using GoodJob) and SQLite (SolidQueue), Redis beats everything else for the above usecase. SolidQueue backed by SQLite may be g…

> Redis beats everything else for the above usecase. Reminds me of Antirez blog post that when Redis is configured for durability it becomes like/slower than postgresql http://oldblog.antirez.com/post/redis-persistence-demystifie...

There's been 6 major releases and countless improvements on Redis since then, I don't think we can say whether it's still relevant.

Also, Antirez has always been very opinionated on not comparing or benchmarking Redis against other dbs for a decade.

Re: I’m leaving Redis for SolidQueue

#40

Post by uep »

formerly_proven wrote: post #26
victorbjorklund wrote: post #9

For people that does not think it scales. A similar implementation in Elixir is Oban. Their benchmark shows a million jobs per minute on a single node (and I am sure it could be increased further with more optimizations). I bet 99,99999% of apps have less than a million background jobs per minute. https://oban.pro/articles/one-million-jobs-a-minute-with-oba...

This benchmark is probably as far removed from how applications use task queues as it could possibly be. The headline is "1 million jobs per minute", which is true. However... - this is achieved by queuing batches of 5000 jobs , so on the queue side this is actually not 1 million TPS, but rather 200 TPS. I've never seen any significant batching of background job creation. - the dispatch is also batched to a few hundr…

This isn't my area, but wouldn't this still be quite effective if it automatically grouped and batched those jobs for you? At low throughput levels, it doesn't need giant batches, and could just timeout after a very short time, and submit smaller batches. At high throughput, they would be full batches. Either way, it seems like this would still serve the purpose, wouldn't it?
Post reply on HN