there seems to be a big hype to adapt pg into any infra. I love PG but this seems not be right thing.
I use it as a job queue. Yes, it has it's cons, but not dealing with another moving piece in the big picture is totally worth it.
Show HN: PgQueuer – Transform PostgreSQL into a Job Queue
11–20 of 140 posts
Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue
#12Does the celery SQLAlchemy broker support PostgreSQL's LISTEN/NOTIFY features?
Similar support in SQLite would simplify testing applications built with celery.
How to add table event messages to SQLite so that the SQLite broker has the same features as AMQP? Could a vtable facade send messages on tablet events?
Are there sqlite Triggers?
Celery > Backends and Brokers: https://docs.celeryq.dev/en/stable/getting-started/backends-...
/? sqlalchemy listen notify: https://www.google.com/search?q=sqlalchemy+listen+notify :
asyncpg.Connection.add_listener
sqlalchemy.event.listen, @listen_for
psychopg2 conn.poll(), while connection.notifies
psychopg2 > docs > advanced > Advanced notifications: https://www.psycopg.org/docs/advanced.html#asynchronous-noti...
PgQueuer.db, PgQueuer.listeners.add_listener; asyncpg add_listener: https://github.com/janbjorge/PgQueuer/blob/main/src/PgQueuer...
asyncpg/tests/test_listeners.py: https://github.com/MagicStack/asyncpg/blob/master/tests/test...
/? sqlite LISTEN NOTIFY: https://www.google.com/search?q=sqlite+listen+notify
sqlite3 update_hook: https://www.sqlite.org/c3ref/update_hook.html
Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue
#13I’ve been thinking about the potential for PostgreSQL-backed job queue libraries to share a common schema. For instance, I’m a big fan of Oban in Elixir: https://github.com/sorentwo/oban Given that there are many Sidekiq-compatible libraries across various languages, it might be beneficial to have a similar approach for PostgreSQL-based job queues. This could allow for job processing in different languages while main…
This would be so immensely useful. I’d estimate that there are so many cases where the producer is Node or Rails and the consumer is Python.
Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue
#14BTW: Good PostgresFM episode on implementing queues in Postgres, various caveats etc: https://www.youtube.com/watch?v=mW5z5NYpGeA .
Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue
#15I’ve been thinking about the potential for PostgreSQL-backed job queue libraries to share a common schema. For instance, I’m a big fan of Oban in Elixir: https://github.com/sorentwo/oban Given that there are many Sidekiq-compatible libraries across various languages, it might be beneficial to have a similar approach for PostgreSQL-based job queues. This could allow for job processing in different languages while main…
Qless "solves" this problem (in redis) by having all core logic written as lua and executed in redis.
You could take a similar approach for pg: define a series of procedures that provide all the required functionality, and then language bindings are all just thin wrappers (to handle language native stuff) around calls to execute a given procedure with the correct arguments.
Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue
#16You can make anything that stores data into a job queue.
Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue
#17There is also Procrastinate: https://procrastinate.readthedocs.io/en/stable/index.html
Procrastinate also uses PostgreSQL's LISTEN/NOTIFY (but can optionally be turned off and use polling). It also supports many features (and more are planned), like sync and async jobs (it uses asyncio under the hood), periodic tasks, retries, task locks, priorities, job cancellation/aborting, Django integration (optional).
DISCLAIMER: I am a co-maintainer of Procrastinate.
Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue
#18there seems to be a big hype to adapt pg into any infra. I love PG but this seems not be right thing.
At low-medium scale, this will be fine. Even at higher scale, so long as you monitor autovacuum performance on the queue table.
At some point it may become practical to bring a dedicated queue system into the stack, sure, but this can massively simplify things when you don’t need or want the additional complexity.
Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue
#19I’ve been thinking about the potential for PostgreSQL-backed job queue libraries to share a common schema. For instance, I’m a big fan of Oban in Elixir: https://github.com/sorentwo/oban Given that there are many Sidekiq-compatible libraries across various languages, it might be beneficial to have a similar approach for PostgreSQL-based job queues. This could allow for job processing in different languages while main…
If you want a generic queue that can be consumed in any runtime, you can just build it directly into postgres via extensions like https://github.com/tembo-io/pgmq.
Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue
#20Any suggestions for something like this for dotnet?