Could this work with Turso, the SQLite rust rewrite?
Honker – Durable queues, streams, pub/sub, and cron scheduler in a SQLite file
21–30 of 69 posts
Re: Honker – Durable queues, streams, pub/sub, and cron scheduler in a SQLite file
#22Almost feels like someone is trying to joke about similar postgres application .
To make it look even more absurd . SQLite is not concurrent and you’ll have tons of problems using it practically .
Re: Honker – Durable queues, streams, pub/sub, and cron scheduler in a SQLite file
#23I can’t see any benchmarks or performance stats.
I’d like to see messages per second.
Re: Honker – Durable queues, streams, pub/sub, and cron scheduler in a SQLite file
#24Earlier quoted context omitted.
Yeah, I had the same instinct - this feels very much like a "nice idea" but the execution falls short. I mean - busily banging on sqlite like this? Shit at that point just use Redis.
For what it's worth, Kine (software that k3s uses to replace etcd with SQL databases) implements etcd watches on SQLite through polling[1]. The reason being that SQLite does not offer NOTIFY/LISTEN like MySQL and Postgres do. Ironically, Honkey attempts implementing NOTIFY/LISTEN through polling. k3s has been running on my home server for about three years now (using the default SQLite backend), and there doesn't see…
With SQLite, you're basically funneled towards a single-writer / single-process design anyway ... in which case why not use a more traditional condvar + mutex rather than polling?
Re: Honker – Durable queues, streams, pub/sub, and cron scheduler in a SQLite file
#25At the end it says: "pg-boss and Oban are the Postgres-side gold standards" -- but Oban supports SQLite now too https://github.com/oban-bg/oban
There's also Graphile Worker. https://github.com/graphile/worker
Re: Honker – Durable queues, streams, pub/sub, and cron scheduler in a SQLite file
#26[flagged]
Re: Honker – Durable queues, streams, pub/sub, and cron scheduler in a SQLite file
#27Suggestion for the author wind back the polling to once a second when nothing is happening.
Re: Honker – Durable queues, streams, pub/sub, and cron scheduler in a SQLite file
#28Earlier quoted context omitted.
[flagged]
Hold on -- if it really is "one lightweight SELECT per millisecond", and you're saying a select is "a couple hundred microseconds", say generously 200us?, then you're spending 200us out of every 1000us just selecting. That's a lot of polling!
I mean only in the same sense that you spend 1 second per second doing something. Time is probably not the best way to evaluate the resources this consumes and I doubt it takes much of anything else either.
It does seem weird though even for sqlite. I wonder how oban does it. I also wonder if OP knows oban can run on sqlite.
Re: Honker – Durable queues, streams, pub/sub, and cron scheduler in a SQLite file
#29"Idle cost is that one lightweight SELECT per millisecond per database — no page-cache pressure, no writer-lock contention, no kernel file watcher in the mix." I think (respectfully) the LLM that probably wrote this overshot the mark here because busy-polling a select does not actually sound better to me than a "kernel file watcher".
If you're not making any changes to the database, does the SELECT "kill" you? And if you are making changes, don't you have to poll regardless after the file watcher wakes you? For WAL mode, SQLite can probably satisfy this query just by inspecting some shared memory. But it is busy waiting, sure.
[dead]
Re: Honker – Durable queues, streams, pub/sub, and cron scheduler in a SQLite file
#30I've implemented something similar in the past, but using inotify. You need to watch the -wal file for IN_MODIFY. To make it work reliably I found I had to run:
BEGIN IMMEDIATE TRANSACTION; ROLLBACK;