81–90 of 146 posts
Earlier quoted context omitted.
Maybe Redis is just overkill
I wish you'd have expanded on that. I almost always learn about some interesting lower-level tech through people trying to avoid a full-featured heavy-for-their-use-case tool or system.
Earlier quoted context omitted.
I'm guessing you're with that adding indirection for what you're actually processing, in that case? So I guess the counter-case would be when you don't want/need that indirection. If I understand what you're saying, is that you'll instead of doing: - Create job with payload (maybe big) > Put in queue > Let worker take from queue > Done You're suggesting: - Create job with ID of payload (stored elsewhere) > Put in que…
If we take webhook for example. - Persist payload in db > Queue with id > Process via worker. Push the payload directly to queue can be tricky. Any queue system usually will have limits on the payload size, for good reasons. Plus if you already commit to db, you can guarantee the data is not lost and can be process again however you want later. But if your queue is having issue, or it failed to queue, you might lost…
Is that how microservice messages work? They push the whole data so the other systems can consume it and take it from there?
- No reason to switch to SolidQueue or GoodJob if you have no issue with Sidekiq. Only do it if you want to remove the Redis infra, no other big benefits other than that imo. - For new projects, I might be more biased towards GoodJob. They're more matured, great community and have more features. - One thing I don't like about SolidQueue is the lack of solid UI. Compared to GoodJob or Sidekiq, it's pretty basic. When I tried it last time, the main page would hang due to unoptimized indexes. Only happens when your data reaches certain threshold. Might have been fixed though.
Another consideration with using RDBMS instead of Redis is that you might need to allocate proper connection pool now. Depends on your database setup. It's nothing big, but that's one additional "cost" as you never really had to consider when you're using Redis.
Earlier quoted context omitted.
I wish you'd have expanded on that. I almost always learn about some interesting lower-level tech through people trying to avoid a full-featured heavy-for-their-use-case tool or system.
You're in luck, the article speaks about that at length!
Exactly what https://www.amazingcto.com/postgres-for-everything/ says; keep it simpel and use PostgreSQL.
Isn't Redis just a lot less relevant these days since enterprise NVME storage is so ridiculously fast? How much latency could you really be saving versus introducing complexity? But I am not a storage/backend engineer, so maybe I don't understand the target use of Redis.
Redis also scales horizontally much, much easier because of the lack of relational schemas. Keys can be owned by a node without any consensus within the cluster beyond which node owns the key. Distributed SQL needs consensus around things like "does the record this foreign key references exist?", which also has to take into account other updates occurring simultaneously.
It's why you see something like Redis caching DB queries pretty often. It's way, way easier to make your Redis cluster 100x as fast than it is to make your DB 100x as fast. I think it's also cheaper in terms of hardware, but I haven't done much beyond napkin math to validate that.
Earlier quoted context omitted.
If we take webhook for example. - Persist payload in db > Queue with id > Process via worker. Push the payload directly to queue can be tricky. Any queue system usually will have limits on the payload size, for good reasons. Plus if you already commit to db, you can guarantee the data is not lost and can be process again however you want later. But if your queue is having issue, or it failed to queue, you might lost…
> Push the payload directly to queue can be tricky. Any queue system usually will have limits on the payload size, for good reasons. Is that how microservice messages work? They push the whole data so the other systems can consume it and take it from there?
For people that does not think it scales. A similar implementation in Elixir is Oban. Their benchmark shows a million jobs per minute on a single node (and I am sure it could be increased further with more optimizations). I bet 99,99999% of apps have less than a million background jobs per minute. https://oban.pro/articles/one-million-jobs-a-minute-with-oba...
Funny you mention Oban, we do use it at work as well, and first thing Oban tells you is to either use Redis as a notifier or resort to polling for jobs and just not notify. https://hexdocs.pm/oban/scaling.html
For people that does not think it scales. A similar implementation in Elixir is Oban. Their benchmark shows a million jobs per minute on a single node (and I am sure it could be increased further with more optimizations). I bet 99,99999% of apps have less than a million background jobs per minute. https://oban.pro/articles/one-million-jobs-a-minute-with-oba...
This benchmark is probably as far removed from how applications use task queues as it could possibly be. The headline is "1 million jobs per minute", which is true. However... - this is achieved by queuing batches of 5000 jobs , so on the queue side this is actually not 1 million TPS, but rather 200 TPS. I've never seen any significant batching of background job creation. - the dispatch is also batched to a few hundr…
Earlier quoted context omitted.
the problem i see here is that we end up treating the background job/task processor as part of the production system (e.g. the server that responds to requests, in the case of a web application) instead of a separate standalone thing. rails doesn’t make this distinction clear enough. it’s okay to back your tasks processor with a pg database (e.g. river[0]) but, as you indirectly pointed out, it shouldn’t be the same…
> it shouldn’t be the same as the production database Why is that?
https://status.circleci.com/incidents/hr0mm9xmm3x6
and a good analysis by a flicker engineer who ran into similar issues
https://blog.mihasya.com/2015/07/19/thoughts-evoked-by-circl...