71–80 of 112 posts
Earlier quoted context omitted.
At the bottom of the page on riverqueue.com it appears there's a screenshot of a UI. But I can't seem to find any docs about it. Am I missing something or is it just not available yet?
The UI isn't quite ready for outside consumption yet but it is being worked on. I would love to hear more about what you'd like to see in it if you want to share.
- Use FOR NO KEY UPDATE instead of FOR UPDATE so you don't block inserts into tables with a foreign key relationship with the job table. [1]
- We parallelize worker by tenant_id but process a single tenant sequentially. I didn't see anything in the docs about that use case; might be worth some design time.
[1]: https://www.migops.com/blog/select-for-update-and-its-behavi...
The number of features lifted directly from Oban[1] is astounding, considering there isn't any attribution in the announcement post or the repo. Starting with the project's tagline, "Robust job processing in Elixir", let's see what else: - The same job states, including the British spelling for `cancelled` - Snoozing and cancelling jobs inline - The prioritization system - Tracking where jobs were attempted in an att…
Hi Parker, I'm genuinely sorry it comes across as though we lifted this stuff directly from Oban. I do mean it when I say that Oban has been a huge inspiration, particularly around its technical design and clean UX. Some of what you've mentioned are cases where we surveyed a variety of our favorite job engines and concluded that we thought Oban's way was superior, whereas others we cycled through a few different impl…
JFC One line of code you don't even have to test
Hi HN, I'm one of the authors of River along with Brandur. We've been working on this library for a few months and thought it was about time we get it out into the world. Transactional job queues have been a recurring theme throughout my career as a backend and distributed systems engineer at Heroku, Opendoor, and Mux. Despite the problems with non-transactional queues being well understood I keep encountering these…
Earlier quoted context omitted.
No, this is a fairly common pattern called having an 'outbox' where the emission/enquing of your event/message/job is tied to the transaction completion of the relevant domain data. We use this to ensure Kafka events are only emitted when a process succeeds, this is very similar.
So when the the business data transaction commit a notify event is raised and a job row is inserted. Out of bound job broker listens to a notify event of the job-table or polls the table skipping rows and takes work for processing?
For our particular use case, I think we're actually not using notify events. We just insert rows into the outbox table and the poller re-emits as kafka events and deletes successfully emitted events from the table.
The number of features lifted directly from Oban[1] is astounding, considering there isn't any attribution in the announcement post or the repo. Starting with the project's tagline, "Robust job processing in Elixir", let's see what else: - The same job states, including the British spelling for `cancelled` - Snoozing and cancelling jobs inline - The prioritization system - Tracking where jobs were attempted in an att…
Hi Parker, I'm genuinely sorry it comes across as though we lifted this stuff directly from Oban. I do mean it when I say that Oban has been a huge inspiration, particularly around its technical design and clean UX. Some of what you've mentioned are cases where we surveyed a variety of our favorite job engines and concluded that we thought Oban's way was superior, whereas others we cycled through a few different impl…
Earlier quoted context omitted.
No, this is a fairly common pattern called having an 'outbox' where the emission/enquing of your event/message/job is tied to the transaction completion of the relevant domain data. We use this to ensure Kafka events are only emitted when a process succeeds, this is very similar.
So when the the business data transaction commit a notify event is raised and a job row is inserted. Out of bound job broker listens to a notify event of the job-table or polls the table skipping rows and takes work for processing?
Unlike https://github.com/tembo-io/pgmq a project we've been working on at Tembo, many queue projects still require you to run and manage a process external to the database, like a background worker. Or they ship as a client library and live in your application, which will limit the languages you can chose to work with. PGMQ is a pure SQL API, so any language that can connect to Postgres can use it.
This looks like a great effort and I am looking forward to trying it out. I am a bit confused by the choice of the LGPL 3.0 license. It requires one to dynamically link the library to avoid GPL's virality, but in a language like Go that statically links everything, it becomes impossible to satisfy the requirements of the license, unless we ignore what it says and focus just on its spirit. I see that was discussed pre…