Live data from Hacker News

Show HN: Honker – Postgres NOTIFY/LISTEN Semantics for SQLite

github.com

31–40 of 95 posts

Re: Show HN: Honker – Postgres NOTIFY/LISTEN Semantics for SQLite

#31

Earlier quoted context omitted.

That’s ignoring the other costs of syscalls like evicting your stuff from the CPU caches. But I agree with the conclusion, system calls are still pretty fast compared to a lot of other things.

Small correction on ambiguous wording - syscalls do not evict all your stuff from CPU caches. It just has to page in whatever is needed for kernel code/data accessed by the call, but that’s no different from if it was done in process as a normal function call.

Depending on implementation details of your CPU and OS, the syscall path may need to flush various auxillary caches (like one or more TLBs) to prevent speculation attacks, which may put additional "drag" on your program after syscall return.

Re: Show HN: Honker – Postgres NOTIFY/LISTEN Semantics for SQLite

#33

Shameless plug: In the upcoming release of PostgreSQL 19, LISTEN/NOTIFY has been optimized to scale much better with selective signaling, i.e. when lots of backends are listening on different channels, patch: https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit...

Great plug very relevant

Re: Show HN: Honker – Postgres NOTIFY/LISTEN Semantics for SQLite

#35
post #17

atomic commit with the business data is the selling point over separate IPC. external message passing always has the 'notification sent but transaction rolled back' problem and that gets messy. one thing i'm curious about: WAL checkpoint. when SQLite truncates WAL back to zero, does the stat() polling handle that correctly? feels like there's a window where events could get lost.

The WAL file sticks around but gets truncated so that counts as an update. Though I don’t have tests for this. Good input, thanks I’ll make sure

Re: Show HN: Honker – Postgres NOTIFY/LISTEN Semantics for SQLite

#36

Awesome. I’m currently using AWS SQS which invokes lambda functions for asynchronous tasks like email sends, but Honker seems like a great local replacement. Any conflicts or issues when running Litestream as well?

Nope! The extension just functions as a shortcut for raw SQL. Litestream edits the wal file but only like a normal checkpoint. So not too bad. Although I haven’t tested it directly. Probably need to

Re: Show HN: Honker – Postgres NOTIFY/LISTEN Semantics for SQLite

#37

Hey HN, I built this. Honker adds cross-process NOTIFY/LISTEN to SQLite. You get push-style event delivery with single-digit millisecond latency without a damon/broker, using your existing SQLite file. A lot of pretty high-traffic applications are just Framework+SQLite+Litestream on a VPS now, so I wanted to bring a sixer to the "just use SQLite" party. SQLite doesn't run a server like Postgres, so the trick is movin…

Is the main use case for this for languages that only have access to process based concurrency? Struggling to see why you would otherwise need this in java/go/clojure/C# your sqlite has a single writer, so you can notify all threads that care about inserts/updates/changes as your application manages the single writer (with a language level concurrent queue) so you know when it's writing and what it has just written.…

I actually hadn’t thought about it this way. The killer app I was imagining was 1ms reactivity without SQL polling and messaging atomic with business commits, plus “one db” and no daemon.

But this is actually a great main benefit as well.

Re: Show HN: Honker – Postgres NOTIFY/LISTEN Semantics for SQLite

#38
This is the kind of skill-level tool I wish existed earlier — I hit the exact pain point running a daily-chronicle site off SQLite + a static deploy a week ago. Ended up with a crude polling loop because the alternatives all wanted me to install Postgres for a single notification semantic.

Question: any thoughts on what breaks first when a single process has 10k+ concurrent listeners? I'm curious whether the SQLite side can sustain what Postgres does cheaply.

Re: Show HN: Honker – Postgres NOTIFY/LISTEN Semantics for SQLite

#39
post #14

Earlier quoted context omitted.

"Syscalls are slow" is only mostly true. They are slower than not having to cross the userspace OS barrier at all, but they're not "slow" like cross-ocean network calls can be. For example, non-VDSO syscalls in linux are about 250 nanoseconds (see for example https://arkanis.de/weblog/2017-01-05-measurements-of-system-... ), VDSO syscalls are roughly 10x faster. Slower than userspace function calls for sure, but more…

Filesystem stuff tends to be slower than average syscalls because of all the locks and complicated traversals needed. If this is using stat instead of fstat then it’s also going through the VFS layer - repeated calls likely go through the cache fast path for path resolution but accessing the stat structure. There’s also hidden costs in that number like atomic accesses that need to acquire cache line locks that are go…

Oh cool, so using io uring plus pragma data version would actually beat stat on Linux holistically speaking? The stat choice was all about cross platform consistency over inotify speed. But syscalls overwhelm can be real.

Re: Show HN: Honker – Postgres NOTIFY/LISTEN Semantics for SQLite

#40

This is the kind of skill-level tool I wish existed earlier — I hit the exact pain point running a daily-chronicle site off SQLite + a static deploy a week ago. Ended up with a crude polling loop because the alternatives all wanted me to install Postgres for a single notification semantic. Question: any thoughts on what breaks first when a single process has 10k+ concurrent listeners? I'm curious whether the SQLite s…

10k listeners is a lot. Thundering herd issue at stat(). SQLite may not be your best choice at this scale.
Post reply on HN