Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I’m leaving Redis for SolidQueue

simplethread.com

Post reply on HN
Print view View original

141–146 of 146 posts

Re: I’m leaving Redis for SolidQueue

#141

Post by otterley »

skunkworker wrote: post #130
ecshafer wrote: post #110

Earlier quoted context omitted.

Rails shops seem to not like to use SQS/PubSub/Kafka/RabbitMQ for some reason. They seem to really like these worker tasks like SideKiq or SolidQueue. When I compare this with Java, C# or Python who all seem much more likely to use a separate message queue then have that handle the job queue.

Rails shops running on normal CRuby, have difficult in effectively scaling out multithreading due to the GVL lock. It's much easier to "scale" ruby using forking with sidekiq or multi process, and to have it consume data from a Redis list. It is possible to get around the GVL using JRuby, but that poses a different set of constraints and issues. There is some definite blending of async messaging in the Ruby world tho…

> Rails shops running on normal CRuby, have difficult in effectively scaling out multithreading due to the GVL lock. It's much easier to "scale" ruby using forking with sidekiq or multi process, and to have it consume data from a Redis list.

This isn't cloud-native at all. In a cloud-native world, these workers would be running in hosted functions (e.g. Lambda) and be consuming from a work queue. I assume this is possible in Rails, but the startup overhead might be considerable.

Re: I’m leaving Redis for SolidQueue

#142

Post by zarzavat »

stavros wrote: post #66
zarzavat wrote: post #63

Earlier quoted context omitted.

If you need to restore the production database do you also want to restore the task database? If your task is to send an email, do you want to send it again? Probably not.

It's not like I'll get a choice between the task database going down and not going down. If my task database goes down, I'm either losing jobs or duplicating jobs, and I have to pick which one I want. Whether the downtime is at the same time as the production database or not is irrelevant. In fact, I'd rather it did happen at the same time as production, so I don't have to reconcile a bunch of data on top of the task…

Right, I was referring to logical databases rather than the database server itself.

Re: I’m leaving Redis for SolidQueue

#143

Post by stavros »

zarzavat wrote: post #142
stavros wrote: post #66

Earlier quoted context omitted.

It's not like I'll get a choice between the task database going down and not going down. If my task database goes down, I'm either losing jobs or duplicating jobs, and I have to pick which one I want. Whether the downtime is at the same time as the production database or not is irrelevant. In fact, I'd rather it did happen at the same time as production, so I don't have to reconcile a bunch of data on top of the task…

Right, I was referring to logical databases rather than the database server itself.

But even for the logical databases, if I want to revert to an earlier state of the database, why wouldn't I want the tasks as well? If I have a bunch of update tasks in flight at that point, wouldn't I want them to actually run? They are a part of the overall state of the system.

Re: I’m leaving Redis for SolidQueue

#144

Post by cle »

dns_snek wrote: post #128
cle wrote: post #55

Earlier quoted context omitted.

This comes up every time this conversation occurs. Yes, PG can theoretically handle just about anything with the right configuration, schema, architecture, etc. Finding that right configuration is not trivial. Even dedicated frameworks like Graphile struggle with it. My startup had the exact same struggles with PG and did the same migration to BullMQ bc we were sick of fiddling with it instead of solving business pro…

The issue is that "83 per second" is multiple orders of magnitude off the expected level of performance on any RDBMS running on anything resembling modern hardware. I haven't worked with Graphile but this just doesn't pass the sniff test unless those 83 jobs per second are somehow translating into thousands of write transactions per second. Their documentation has a performance section with a benchmark that claims to…

> The issue is that "83 per second" is multiple orders of magnitude off the expected level of performance on any RDBMS running on anything resembling modern hardware.

This is just not true, there are so many scenarios where 83/sec would be the limit. That number by itself is almost meaningless, similar to benchmarks which also make a bunch of assumptions about workloads and runtime environments.

As a simple example if your queue has a large backlog, you have a large worker fleet aggressively pulling work to minimize latency, your payloads are large, you have not optimized indexing, and/or you have many jobs scheduled for the future, every acquire can be an expensive table scan.

(This is a specific example because this is one of many failure scenarios I’ve encountered with Graphile that can cause your DB to meltdown. The same workload in Redis barely causes a blip in Redis CPU, without having to fiddle with indexes and auto vacuuming and worker backoffs.)

Re: I’m leaving Redis for SolidQueue

#145

Post by rajaravivarma_r »

Manfred wrote: post #18
rajaravivarma_r wrote: post #15

The one use case where a DB backed queue will fail for sure is when the payload is large. For example, you queue a large JSON payload to be picked up by a worker and process it, then the DB writing overhead itself makes a background worker useless. I've benchmarked Redis (Sidekiq), Postgres (using GoodJob) and SQLite (SolidQueue), Redis beats everything else for the above usecase. SolidQueue backed by SQLite may be g…

In my experience you want job parameters to be one, maybe two ids. Do you have a real world example where that is not the case?

I have been doing this for at least a decade now and it is a great pattern, but think of an ETL pipeline where you fetch a huge JSON payload, store it in the database and then transform it and load it in another model. I had an use case where I wanted to process the JSON payload and pass it down the pipeline before storing it in the useful model. I didn't want to store the intermediate JSON anywhere. I benchmarked it for this specific use case.

Re: I’m leaving Redis for SolidQueue

#146

Post by rajaravivarma_r »

ddorian43 wrote: post #31
rajaravivarma_r wrote: post #15

The one use case where a DB backed queue will fail for sure is when the payload is large. For example, you queue a large JSON payload to be picked up by a worker and process it, then the DB writing overhead itself makes a background worker useless. I've benchmarked Redis (Sidekiq), Postgres (using GoodJob) and SQLite (SolidQueue), Redis beats everything else for the above usecase. SolidQueue backed by SQLite may be g…

> Redis beats everything else for the above usecase. Reminds me of Antirez blog post that when Redis is configured for durability it becomes like/slower than postgresql http://oldblog.antirez.com/post/redis-persistence-demystifie...

May be, but over 6 years of using Redis with bare minimum setup, I have never lost any data and my use case happens to be queuing intermediate results, so durability won't be an issue.
Post reply on HN