Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I’m leaving Redis for SolidQueue

simplethread.com

Post reply on HN
Print view View original

131–140 of 146 posts

Re: I’m leaving Redis for SolidQueue

#131

Post by victorbjorklund »

parthdesai wrote: post #103
victorbjorklund wrote: post #88

Earlier quoted context omitted.

I don't think that Oban is telling you to always use Redis. I think what they're saying is if you reach a certain scale where you're feeling the pain of the default notifier you could use Oban.Notifiers.PG as long as your application is running as a cluster. If you don't run it as a cluster, then you might have to reach for Redis. But then it's more about not running a cluster.

> For people that does not think it scales You started your comment with that

It does scale. It is your companies choice not to cluster your application. Redis is not needed. It is a choice if you don’t want to cluster your app.

Re: I’m leaving Redis for SolidQueue

#132

Post by victorbjorklund »

bgentry wrote: post #106
parthdesai wrote: post #69

Earlier quoted context omitted.

Funny you mention Oban, we do use it at work as well, and first thing Oban tells you is to either use Redis as a notifier or resort to polling for jobs and just not notify. https://hexdocs.pm/oban/scaling.html

This is largely because LISTEN/NOTIFY has an implementation which uses a global lock. At high volume this obviously breaks down: https://www.recall.ai/blog/postgres-listen-notify-does-not-s... None of that means Oban or similar queues don't/can't scale—it just means a high volume of NOTIFY doesn't scale, hence the alternative notifiers and the fact that most of its job processing doesn't depend on notifications at al…

Yup. I wasn’t talking about notify in particular but about using Postgres in general.

Re: I’m leaving Redis for SolidQueue

#133

Post by dns_snek »

vjerancrnjak wrote: post #118
dns_snek wrote: post #32

Earlier quoted context omitted.

Facing issues with 83 jobs per second (5k/min) sounds like an extreme misconfiguration. That's not high throughput at all and it shouldn't create any appreciable load on any database.

They probably did not batch. It’s realistic they will have issues if code is written to handle 1 job at a time and needs to make several roundtrips to the same db inside the same locking transaction. Leases exist for a reason.

> if code is written to handle 1 job at a time and needs to make several roundtrips to the same db inside the same locking transaction.

Do you mean the application code? The worker itself causing the bottleneck is definitely one possibility however if that were the case the issue wouldn't have resolved itself when they switched to a different job queue.

Re: I’m leaving Redis for SolidQueue

#134

Post by [deleted] »

jacob-s-son wrote: post #14

Every author of the free software obviously has rights to full control of the scope of their project. That being said, I regret that we have switched from good_job ( https://github.com/bensheldon/good_job ). The thing is - Basecamp is a MySQL shop and their policy is not to accept RDMS engine specific queries. You can see in their issues in Github that they try to stick "universal" SQL and are personally mostly conce…

[deleted]

Re: I’m leaving Redis for SolidQueue

#135

Post by pas »

Manfred wrote: post #18
rajaravivarma_r wrote: post #15

The one use case where a DB backed queue will fail for sure is when the payload is large. For example, you queue a large JSON payload to be picked up by a worker and process it, then the DB writing overhead itself makes a background worker useless. I've benchmarked Redis (Sidekiq), Postgres (using GoodJob) and SQLite (SolidQueue), Redis beats everything else for the above usecase. SolidQueue backed by SQLite may be g…

In my experience you want job parameters to be one, maybe two ids. Do you have a real world example where that is not the case?

...well, that's good for scaling the queue, but this means the worker needs to load all relevant state/context from some DB (which might be sped up with a cache, but then things are getting really complex)

ideally you pass the context that's required for the job (let's say it's less than 100Kbytes), but I don't think that counts as large JSON, but request rate (load) can make even 512byte too much, therefore "it depends"

but in general passing around large JSONs on the network/memory is not really slow compared to writing them to a DB (WAL + fsync + MVCC management)

Re: I’m leaving Redis for SolidQueue

#136

Post by vjerancrnjak »

dns_snek wrote: post #133
vjerancrnjak wrote: post #118

Earlier quoted context omitted.

They probably did not batch. It’s realistic they will have issues if code is written to handle 1 job at a time and needs to make several roundtrips to the same db inside the same locking transaction. Leases exist for a reason.

> if code is written to handle 1 job at a time and needs to make several roundtrips to the same db inside the same locking transaction. Do you mean the application code? The worker itself causing the bottleneck is definitely one possibility however if that were the case the issue wouldn't have resolved itself when they switched to a different job queue.

Well, you would no longer have thousands of open transactions maintaining the locks.

Of course, if q on top of psql is reasonably implemented (lease), what they say makes no sense.

Re: I’m leaving Redis for SolidQueue

#137

Post by pas »

andersonklando wrote: post #83
azuanrb wrote: post #25

Earlier quoted context omitted.

If we take webhook for example. - Persist payload in db > Queue with id > Process via worker. Push the payload directly to queue can be tricky. Any queue system usually will have limits on the payload size, for good reasons. Plus if you already commit to db, you can guarantee the data is not lost and can be process again however you want later. But if your queue is having issue, or it failed to queue, you might lost…

> Push the payload directly to queue can be tricky. Any queue system usually will have limits on the payload size, for good reasons. Is that how microservice messages work? They push the whole data so the other systems can consume it and take it from there?

yes and no, as the sibling comment mentions sometimes a message bus is used (Kafka, for example), but Netflix is (was?) all-in with HTTP (low-latency gRPC, HTTP/3, wrapped in nice type-safe SDK packages)

but ideally you don't break the glass and reach for a microservices architecture if you don't need the scalability afforded by very deep decoupling

which means ideally you have separate databases (and DB schema and even likely different kind of data store), and through the magic of having minimally overlapping "bounded contexts" you don't need a lot of data to be sent over (the client SDK will pick what it needs for example)

... of course serving a content recommendation request (which results in a cascade of requests that go to various microservices, eg. profile, rights management data, CDN availability, and metadata for the results, image URLs, etc) for a Netflix user doesn't need durability, so no Kafka (or other message bus), but when the user changes their profile it might be something that gets "broadcasted"

(and durable "replayable" queues help, because then services can be put to read-only mode to serve traffic, while new instances are starting up, and they will catch up. and of course it's useful for debugging too, at least compared to HTTP logs, which usually don't have the body/payload logged.)

Re: I’m leaving Redis for SolidQueue

#138

Post by imtringued »

jacob-s-son wrote: post #14

Every author of the free software obviously has rights to full control of the scope of their project. That being said, I regret that we have switched from good_job ( https://github.com/bensheldon/good_job ). The thing is - Basecamp is a MySQL shop and their policy is not to accept RDMS engine specific queries. You can see in their issues in Github that they try to stick "universal" SQL and are personally mostly conce…

In the SQL world even simple things like booleans are RDMS engine specific so I have no idea how that is supposed to work.

Re: I’m leaving Redis for SolidQueue

#139

Post by imtringued »

yawboakye wrote: post #125
nick__m wrote: post #48

Earlier quoted context omitted.

Frequently you have to couple the transactional state of the queue db and the app db, colocating them is the simplest way to achieve that without resorting to distributed transactions or patterns that involve orchestrated compensation actions.

that’s setting yourself up for trouble, imo. intermediate states solve this problem, and economically. for mature production system see temporal[0]. their magic sauce is good intermediate states. [0]: https://temporal.io/

This feels like an ad because you explained absolutely nothing.

It's one thing to explain to people what a screw driver is and you just happen to sell screw drivers to people who might need them.

It's a wholly different thing to sell the screw driver first and then let people figure out why they need it.

Re: I’m leaving Redis for SolidQueue

#140

Post by yawboakye »

imtringued wrote: post #139
yawboakye wrote: post #125

Earlier quoted context omitted.

that’s setting yourself up for trouble, imo. intermediate states solve this problem, and economically. for mature production system see temporal[0]. their magic sauce is good intermediate states. [0]: https://temporal.io/

This feels like an ad because you explained absolutely nothing. It's one thing to explain to people what a screw driver is and you just happen to sell screw drivers to people who might need them. It's a wholly different thing to sell the screw driver first and then let people figure out why they need it.

i’m not associated with temporal, nor does the link above have any referrer nonsense in there. i don’t profit from referring to it here. in fact it may well be a household name in the hn community. that out of the way, it’s not wrong to point to a proper resource that can explain and demonstrate my argument better than a couple of words could. temporal is open source[0] so maybe a github link would have been more palatable?

[0]: https://github.com/temporalio/temporal

Post reply on HN