Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Honker – Durable queues, streams, pub/sub, and cron scheduler in a SQLite file

honker.dev

Post reply on HN
Print view View original

41–50 of 69 posts

Re: Honker – Durable queues, streams, pub/sub, and cron scheduler in a SQLite file

#41

Post by sroussey »

koito17 wrote: post #18
d1l wrote: post #10

Earlier quoted context omitted.

Yeah, I had the same instinct - this feels very much like a "nice idea" but the execution falls short. I mean - busily banging on sqlite like this? Shit at that point just use Redis.

For what it's worth, Kine (software that k3s uses to replace etcd with SQL databases) implements etcd watches on SQLite through polling[1]. The reason being that SQLite does not offer NOTIFY/LISTEN like MySQL and Postgres do. Ironically, Honkey attempts implementing NOTIFY/LISTEN through polling. k3s has been running on my home server for about three years now (using the default SQLite backend), and there doesn't see…

Are you trying to avoid sleep?

Re: Honker – Durable queues, streams, pub/sub, and cron scheduler in a SQLite file

#42

Post by russellthehippo »

Author here - previously posted here: https://news.ycombinator.com/item?id=47874647

Key difference vs SQL polling is that we’re touching metadata instead of data pages. I have work in process to make this work without any polling (innotify, kqueue, mmap’d shm file check) after the original stat(2) direction proved unreliable if lightweight.

Would love your feedback and or contributions in the repo - still figuring out the end shape.

Re: Honker – Durable queues, streams, pub/sub, and cron scheduler in a SQLite file

#43

Post by russellthehippo »

tptacek wrote: post #5

"Idle cost is that one lightweight SELECT per millisecond per database — no page-cache pressure, no writer-lock contention, no kernel file watcher in the mix." I think (respectfully) the LLM that probably wrote this overshot the mark here because busy-polling a select does not actually sound better to me than a "kernel file watcher".

Respectfully (thanks haha) - yeah probably right. Original intent was to use inotify type thing but i avoided per-platform differences at the outset. this was definitely a for fun project that blew up unintentionally and am working to harden/improve.

Love Fly.

Re: Honker – Durable queues, streams, pub/sub, and cron scheduler in a SQLite file

#44

Post by russellthehippo »

8note wrote: post #32
tptacek wrote: post #5

"Idle cost is that one lightweight SELECT per millisecond per database — no page-cache pressure, no writer-lock contention, no kernel file watcher in the mix." I think (respectfully) the LLM that probably wrote this overshot the mark here because busy-polling a select does not actually sound better to me than a "kernel file watcher".

to me it sounds like they asked it to not make a kernel file watcher, and now it writes that into every comment everywhere, despite not even being in the implementation

Yup

Re: Honker – Durable queues, streams, pub/sub, and cron scheduler in a SQLite file

#46

Post by nine_k »

booi wrote: post #31
andai wrote: post #12

Earlier quoted context omitted.

What's the CPU usage? Like 2%? I had a manual fs polling thing a while back. It was ugly (low time budget, didn't wanna mess with the native watchers), just scanned the whole thing once per second. It averaged out to like 0.3% CPU. Not elegant, but acceptable for my purposes! (Small-ish directory, and "ping me within a second or two" was realtime enough for this use case.)

i mean, technically this is once per millisecond , so this would happen 1000x more. In your case due to the kernel overhead you would likely not even be able to do it (300% CPU?). Either way this does seem like a very large overhead due to the fact that there's just no other way to do it without a deeper kernel integration which might be outside the scope of what sqlite is trying to do.

If the fs tree scanned once per second had 1000 files, it would be once per millisecond for a file.

Re: Honker – Durable queues, streams, pub/sub, and cron scheduler in a SQLite file

#47

Post by nine_k »

felooboolooomba wrote: post #8
tptacek wrote: post #5

"Idle cost is that one lightweight SELECT per millisecond per database — no page-cache pressure, no writer-lock contention, no kernel file watcher in the mix." I think (respectfully) the LLM that probably wrote this overshot the mark here because busy-polling a select does not actually sound better to me than a "kernel file watcher".

"one lightweight SELECT per millisecond" This reminds me of the teenager who told her dad that she was just a tiny little bit pregnant.

One cannot be a little bit pregnant. But a DB can be only a little bit in the RAM, and specifically in the page cache. SQLite can act exactly like that, and it's damn fast as long as it does not need to durably write a transaction. Polling once a millisecond could spend a few microseconds.

I wonder if using a tiny Redis instance, or even something like LevelDB would be even more efficient.

Post reply on HN