Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I’m leaving Redis for SolidQueue

simplethread.com

Post reply on HN
Print view View original

61–70 of 146 posts

Re: I’m leaving Redis for SolidQueue

#61

Post by cl0ckt0wer »

robertlagrant wrote: post #60
jacob-s-son wrote: post #14

Every author of the free software obviously has rights to full control of the scope of their project. That being said, I regret that we have switched from good_job ( https://github.com/bensheldon/good_job ). The thing is - Basecamp is a MySQL shop and their policy is not to accept RDMS engine specific queries. You can see in their issues in Github that they try to stick "universal" SQL and are personally mostly conce…

> their policy is not to accept RDMS engine specific queries Why? Is it so they can switch in future?

Then they don't have to troubleshoot advanced queries.

Re: I’m leaving Redis for SolidQueue

#62

Post by chasd00 »

jacob-s-son wrote: post #14

Every author of the free software obviously has rights to full control of the scope of their project. That being said, I regret that we have switched from good_job ( https://github.com/bensheldon/good_job ). The thing is - Basecamp is a MySQL shop and their policy is not to accept RDMS engine specific queries. You can see in their issues in Github that they try to stick "universal" SQL and are personally mostly conce…

If you’re tied so tight to MySQL that you’re labeled a “MySQL shop” then it seems logical to use MySQL specific features. I must be missing something.

Re: I’m leaving Redis for SolidQueue

#63

Post by zarzavat »

erispoe wrote: post #59
yawboakye wrote: post #11

Earlier quoted context omitted.

the problem i see here is that we end up treating the background job/task processor as part of the production system (e.g. the server that responds to requests, in the case of a web application) instead of a separate standalone thing. rails doesn’t make this distinction clear enough. it’s okay to back your tasks processor with a pg database (e.g. river[0]) but, as you indirectly pointed out, it shouldn’t be the same…

> it shouldn’t be the same as the production database Why is that?

If you need to restore the production database do you also want to restore the task database?

If your task is to send an email, do you want to send it again? Probably not.

Re: I’m leaving Redis for SolidQueue

#66

Post by stavros »

zarzavat wrote: post #63
erispoe wrote: post #59

Earlier quoted context omitted.

> it shouldn’t be the same as the production database Why is that?

If you need to restore the production database do you also want to restore the task database? If your task is to send an email, do you want to send it again? Probably not.

It's not like I'll get a choice between the task database going down and not going down. If my task database goes down, I'm either losing jobs or duplicating jobs, and I have to pick which one I want. Whether the downtime is at the same time as the production database or not is irrelevant.

In fact, I'd rather it did happen at the same time as production, so I don't have to reconcile a bunch of data on top of the tasks.

Re: I’m leaving Redis for SolidQueue

#67

Post by cess11 »

formerly_proven wrote: post #26
victorbjorklund wrote: post #9

For people that does not think it scales. A similar implementation in Elixir is Oban. Their benchmark shows a million jobs per minute on a single node (and I am sure it could be increased further with more optimizations). I bet 99,99999% of apps have less than a million background jobs per minute. https://oban.pro/articles/one-million-jobs-a-minute-with-oba...

This benchmark is probably as far removed from how applications use task queues as it could possibly be. The headline is "1 million jobs per minute", which is true. However... - this is achieved by queuing batches of 5000 jobs , so on the queue side this is actually not 1 million TPS, but rather 200 TPS. I've never seen any significant batching of background job creation. - the dispatch is also batched to a few hundr…

The 5k batching is done when inserting the jobs into the database. It's not like they exert some special control over the performance of the database engine, and this isn't what they're trying to measure in the article.

They spend some time explaining how to tune the job runners to double the 17k jobs/s. The article is kind of old, Elixir 1.14 was a while ago, and it is basically a write-up on how they managed a bit of performance increase by using new features of this language version.

Re: I’m leaving Redis for SolidQueue

#68

Post by antisthenes »

ivolimmen wrote: post #65

Exactly what https://www.amazingcto.com/postgres-for-everything/ says; keep it simpel and use PostgreSQL.

Isn't Redis just a lot less relevant these days since enterprise NVME storage is so ridiculously fast?

How much latency could you really be saving versus introducing complexity?

But I am not a storage/backend engineer, so maybe I don't understand the target use of Redis.

Re: I’m leaving Redis for SolidQueue

#69

Post by parthdesai »

victorbjorklund wrote: post #9

For people that does not think it scales. A similar implementation in Elixir is Oban. Their benchmark shows a million jobs per minute on a single node (and I am sure it could be increased further with more optimizations). I bet 99,99999% of apps have less than a million background jobs per minute. https://oban.pro/articles/one-million-jobs-a-minute-with-oba...

Funny you mention Oban, we do use it at work as well, and first thing Oban tells you is to either use Redis as a notifier or resort to polling for jobs and just not notify.

https://hexdocs.pm/oban/scaling.html

Re: I’m leaving Redis for SolidQueue

#70

Post by byroot »

antirez wrote: post #4

Every time some production environment can be simplified, it is good news in my opinion. The ideal situation with Rails would be if there is a simple way to switch back to Redis, so that you can start simple, and as soon as you hit some fundamental issue with using SolidQueue (mostly scalability, I guess, in environments where the queue is truly stressed -- and you don't want to have a Postgres scalability problem be…

> The ideal situation with Rails would be if there is a simple way to switch back to Redis

That's largely the case.

Rails provide an abstracted API for jobs (Active Job). Of course some application do depend on queue implementation specific features, but for the general case, you just need to update your config to switch over (and of course handle draining the old queue).

Post reply on HN