Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I’m leaving Redis for SolidQueue

simplethread.com

Post reply on HN
Print view View original

41–50 of 146 posts

Re: I’m leaving Redis for SolidQueue

#41

Post by watercolorblind »

antirez wrote: post #4

Every time some production environment can be simplified, it is good news in my opinion. The ideal situation with Rails would be if there is a simple way to switch back to Redis, so that you can start simple, and as soon as you hit some fundamental issue with using SolidQueue (mostly scalability, I guess, in environments where the queue is truly stressed -- and you don't want to have a Postgres scalability problem be…

The primary pain point I see here is if devs lean into transactions such that their job is only created together with the everything else that happened.

Losing that guarantee can make the eventual migration harder, even if that migration is to a different postgres instance than the primary db.

Re: I’m leaving Redis for SolidQueue

#42

Post by EugeneOZ »

Chapter "The True Cost of Redis" surprised me.

> Deploy, version, patch, and monitor the server software

And with PostgreSQL you don't need it?

> Configure a persistence strategy. Do you choose RDB snapshots, AOF logs, or both?

It's a one-time decision. You don't need to do it daily.

> Sustain network connectivity, including firewall rules, between Rails and Redis

And for a PostgreSQL DB you don't need it?

> Authenticate your Redis clients

And your PostgreSQL works without that?

> Build and care for a high availability (HA) Redis cluster

If you want a cluster of PostgreSQL databases, perhaps you will do that too.

Re: I’m leaving Redis for SolidQueue

#43

Post by adamors »

KolmogorovComp wrote: post #23

> Job latency under 1ms is critical to your business. This is a real and pressing concern for real-time bidding, high frequency trading (HFT), and other applications in the same ilk. From TFA. Are there really people using Rails for HFT?

Of course not, and the company whose blog we're reading isn't doing anything similar either https://www.simplethread.com/case-studies/ Rather funny IMO

Re: I’m leaving Redis for SolidQueue

#44

Post by hahahahhaah »

ckbkr10 wrote: post #17

Comparing Redis to SQL is kinda off topic. Sure you can replace the one with the other but then we are talking about completely different concepts aren't we? When all we are talking about is "good enough" the bar is set at a whole different level.

Well they move from one thing not designed for queues to another not designed for queues. Maybe use a queue!

Re: I’m leaving Redis for SolidQueue

#45

Post by madethemcry »

DHH also famously describe why and how they are leaving the cloud https://world.hey.com/dhh/why-we-re-leaving-the-cloud-654b47...

I'm not a fan boy of DHH but I really like his critical thinking about the status quo. I'm not able to leave the cloud or I better phrase it as it's too comfortable right now. I really wanted to leave redis behind me as it's mostly a hidden part of Rails nothing I use directly but often I have to pay for it "in the cloud"

I quickly hit an issue with the family of Solid features: Documentation doesn't really cover the case "inside your existing application" (at least when I looked into it shortly after Rails 8 was released). Being in the cloud (render.com, fly.io and friends) I had to create multiple DBs, one for each Solid feature. That was not acceptable as you usually pay per service/DB not per usage - similar how you have to pay for Redis.

This was a great motivation to research the cloud space once again and then I found Railway. You pay per usage. So I've right now multiple DBs, one for each Solid feature. And on top multiple environments multiplying those DBs and I pay like cents for that part of the app while it's not really filled. Of course in this setup I would also pay cents for Redis but it's still good to see a less complex landscape in my deployment environment.

Long story short, while try to integrate SolidQueue myself I found Railway. Deployment are fun again with that! Maybe that helps someone today as well.

Re: I’m leaving Redis for SolidQueue

#46

Post by downsplat »

EugeneOZ wrote: post #42

Chapter "The True Cost of Redis" surprised me. > Deploy, version, patch, and monitor the server software And with PostgreSQL you don't need it? > Configure a persistence strategy. Do you choose RDB snapshots, AOF logs, or both? It's a one-time decision. You don't need to do it daily. > Sustain network connectivity, including firewall rules, between Rails and Redis And for a PostgreSQL DB you don't need it? > Authenti…

I guess the point is that you're already doing it for postgres. You alrrady need persistent storage for your app, and the same engine can handle your queuing needs.

Re: I’m leaving Redis for SolidQueue

#47

Post by downsplat »

Not a ruby shop here so it's not directly comparable, but I'm very happy with beanstalkd as a minimalistic job queue. We're on mysql for historical reasons, and it didn't support SKIP LOCKED at the time, so we had to add another tool.

Re: I’m leaving Redis for SolidQueue

#48

Post by nick__m »

yawboakye wrote: post #38
andrewstuart wrote: post #28

Earlier quoted context omitted.

It’s not necessary to separate queue db from application db.

got it. is it necessary, then, to couple queue db with app db? if answer is no then we can’t make a necessity argument here, unfortunately.

Frequently you have to couple the transactional state of the queue db and the app db, colocating them is the simplest way to achieve that without resorting to distributed transactions or patterns that involve orchestrated compensation actions.

Re: I’m leaving Redis for SolidQueue

#49

Post by downsplat »

jacob-s-son wrote: post #14

Every author of the free software obviously has rights to full control of the scope of their project. That being said, I regret that we have switched from good_job ( https://github.com/bensheldon/good_job ). The thing is - Basecamp is a MySQL shop and their policy is not to accept RDMS engine specific queries. You can see in their issues in Github that they try to stick "universal" SQL and are personally mostly conce…

That sounds like the worst of possible worlds! At $WORK we're also on mysql, but I don't know what I would do without engine-specific queries. For one, on complex JOINs, mysql sometimes gets the query plan spectacularly wrong, and even if it doesn't now, you can't be sure it won't in the future. So for many important queries I put the tables in the intended order and add a STRAIGHT_JOIN to future-proof it and skip query planner complexity.

Re: I’m leaving Redis for SolidQueue

#50

Post by heartbreak »

downsplat wrote: post #46
EugeneOZ wrote: post #42

Chapter "The True Cost of Redis" surprised me. > Deploy, version, patch, and monitor the server software And with PostgreSQL you don't need it? > Configure a persistence strategy. Do you choose RDB snapshots, AOF logs, or both? It's a one-time decision. You don't need to do it daily. > Sustain network connectivity, including firewall rules, between Rails and Redis And for a PostgreSQL DB you don't need it? > Authenti…

I guess the point is that you're already doing it for postgres. You alrrady need persistent storage for your app, and the same engine can handle your queuing needs.

Exactly, if you’re already doing it for Postgres and Postgres can do the job well enough to meet your requirements, you’re only adding more cost and complexity by deploying Redis too.
Post reply on HN