Live data from Hacker News

Show HN: Honker – Postgres NOTIFY/LISTEN Semantics for SQLite

github.com

51–60 of 95 posts

Re: Show HN: Honker – Postgres NOTIFY/LISTEN Semantics for SQLite

#51
post #23

Earlier quoted context omitted.

It's not.

Yep, definitely still in use. Do yall above have an opinion if the pragma is better than the syscall? What are the trade offs there? Another comment thread mentioned this as well and pointed to io uring. I was thinking that dism spam is worse than syscall spam.

Depends on what to mean by better.

I may be wrong, but I think you wrote somewhere that you're looking at the WAL size increasing to know if something was committed. Well, the WAL can be truncated, what then? Or even, however unlikely, it could be truncated, then a transaction comes and appends just enough to it to make it the same size.

If SQLite has an API it guarantees can notify you of changes, that seems better, in the sense that you're passing responsibility along to the experts. It should also work with rollback mode, another advantage. And I don't think wakes you up if a large transaction rolls back (a transaction can hit the WAL and never commit).

That said, I'm not sure what's lighter on average. For a WAL mode database, I will say that something that has knowledge of the WAL index could potentially be cheaper? That file is mmapped. The syscalls involved are file locks, if any.

Re: Show HN: Honker – Postgres NOTIFY/LISTEN Semantics for SQLite

#52

Earlier quoted context omitted.

I guess the idea is to have all writes go through a central server with local read replicas for improved read perf. The default litestream sync interval is 1s. I bet many use-cases would be satisfied with a few seconds delay for cross-region notifications.

It's good for pubsub but not for claim/ack workflow unless you do If-None-Match CAS semantics on a separate filesystem which, actually, yeah that's probably fine. Feels heavy on S3 ops. But! you do save on inter-AZ networking, the Warpstream hypothesis.

Claims kill this, IMO.

Unless you have a single "reader", you don't mind the delay, and don't worry about redoing a bunch of notifications after a crash (and so, can delay claims significantly), concurrency will kill this.

Re: Show HN: Honker – Postgres NOTIFY/LISTEN Semantics for SQLite

#53
post #48

Earlier quoted context omitted.

See comment below - Darwin silently drops same-process notifs. I could change the behavior depending on same vs cross process and platform but I wanted to”just one thing to worry about”. Potentially a good optimization later. Would help reduce syscalls.

I believe you are mistaken. If you are referring to the comment from ArielTM, that's an LLM bot regurgitating your readme.

Apologies for not being specific.

The specific thing I'm talking about is this: write events don't fire until the file handle is closed. [1] I didn't validate this myself btw, but my original design was certainly trying to use notify events rather than stat polling. My research (heavily AI assisted of course) led me away from that path as platforms differ in behavior and I wanted to avoid that.

[1] https://github.com/notify-rs/notify/issues/240

Re: Show HN: Honker – Postgres NOTIFY/LISTEN Semantics for SQLite

#54
post #51

Earlier quoted context omitted.

Yep, definitely still in use. Do yall above have an opinion if the pragma is better than the syscall? What are the trade offs there? Another comment thread mentioned this as well and pointed to io uring. I was thinking that dism spam is worse than syscall spam.

Depends on what to mean by better. I may be wrong, but I think you wrote somewhere that you're looking at the WAL size increasing to know if something was committed. Well, the WAL can be truncated, what then? Or even, however unlikely, it could be truncated, then a transaction comes and appends just enough to it to make it the same size. If SQLite has an API it guarantees can notify you of changes, that seems better,…

Interesting, thank you for the response and explanation. Honker workers/listerners are holding an open connection anyway. I do trust SQLite guarantees more than cross-platform sys behavior. I will explore the C API angle.

Re: Show HN: Honker – Postgres NOTIFY/LISTEN Semantics for SQLite

#55

This is the kind of skill-level tool I wish existed earlier — I hit the exact pain point running a daily-chronicle site off SQLite + a static deploy a week ago. Ended up with a crude polling loop because the alternatives all wanted me to install Postgres for a single notification semantic. Question: any thoughts on what breaks first when a single process has 10k+ concurrent listeners? I'm curious whether the SQLite s…

10k listeners is a lot. Thundering herd issue at stat(). SQLite may not be your best choice at this scale.

also this is designed for a single machine. 10k listeners on one machine seems like a lot!

Re: Show HN: Honker – Postgres NOTIFY/LISTEN Semantics for SQLite

#56
post #52

Earlier quoted context omitted.

It's good for pubsub but not for claim/ack workflow unless you do If-None-Match CAS semantics on a separate filesystem which, actually, yeah that's probably fine. Feels heavy on S3 ops. But! you do save on inter-AZ networking, the Warpstream hypothesis.

Claims kill this, IMO. Unless you have a single "reader", you don't mind the delay, and don't worry about redoing a bunch of notifications after a crash (and so, can delay claims significantly), concurrency will kill this.

exactly. then you're building distributed locking and it's probably time for a different tool

Re: Show HN: Honker – Postgres NOTIFY/LISTEN Semantics for SQLite

#57

Thanks for this! I have a proliferation of small apps backed by SQLite. And most of these need a queue and scheduler. I home rolled some stuff for it but was always pining for the elegance of the Postgres solutions. Will give this a spin very soon

"a small proliferation" is a nice way to describe the cluster that is my side project habit. if you bump into any issues pls pull a PR or drop an issue on the repo!

Re: Show HN: Honker – Postgres NOTIFY/LISTEN Semantics for SQLite

#58

Hey HN, I built this. Honker adds cross-process NOTIFY/LISTEN to SQLite. You get push-style event delivery with single-digit millisecond latency without a damon/broker, using your existing SQLite file. A lot of pretty high-traffic applications are just Framework+SQLite+Litestream on a VPS now, so I wanted to bring a sixer to the "just use SQLite" party. SQLite doesn't run a server like Postgres, so the trick is movin…

Is the main use case for this for languages that only have access to process based concurrency? Struggling to see why you would otherwise need this in java/go/clojure/C# your sqlite has a single writer, so you can notify all threads that care about inserts/updates/changes as your application manages the single writer (with a language level concurrent queue) so you know when it's writing and what it has just written.…

There's more process-based concurrency than you'd expect in shops that use those languages.

Cron jobs might need to coordinate with webservers. Even heavily threaded webservers might have some subprocesses/forking to manage connection pools and hot reloads and whatnot. Suid programs are process-separated from non-suid programs. Plenty of places are in the "permanent middle" of a migration from e.g. Java 7 to Java 11 and migrate by splitting traffic to multiple copies of the same app running on different versions of the runtime.

If you're heavily using SQLite for your DB already, you probably are reluctant to replace those situations with multiple servers coordinating around a central DB.

Nit:

> languages that only have process based concurrency python/JS/TS/ruby

Not true. There are tons and tons of threaded Python web frameworks/server harnesses, and there were even before GIL-removal efforts started. Just because gunicorn/multiprocessing are popular doesn't mean there aren't loads of huge deployments running threads (and not suffering for it much, because most web stacks are IO bound). Ruby's similar, though threads are less heavily-used than in Python. JS/TS as well: https://nodejs.org/api/worker_threads.html

Re: Show HN: Honker – Postgres NOTIFY/LISTEN Semantics for SQLite

#60
Very neat! I like this a lot, nice work.

After peeking the source, a few possible areas of improvement:

- You can use `fstat` and keep a file handle around, likely further improving performance (well, reducing the performance hit to other users of the filesystem by not resolving vfs nodes). If you do this, you'll have to check for file deletions.

- If you do stick with stat(2), it might be a good idea to track the inode number from the stat result in addition to the time,size tuple. That handles the "t,s = 1,2; honker gets SIGSTOPped/CRIU'd; database file replaced; honker started again", as well as renameat/symlink-swap fiddling. Changing inode probably should just trigger a crash.

- Also check the device number from the stat call. It sounds fringe, but the number of weird hellbugs I've dealt with in my career caused by code continually interacting with a file at the same time as something else mounted an equivalent path "over" the directory the file was originally in is nonzero.

- It's been a few years since I fought with this, but aren't there edge cases here if the system clock goes backwards? IIRC the inode timestamp isn't monotonic--right? There are various strategies for detecting clock adjustment, of various reliability, that you could use here, if so. Just checking if the mtime-vs-system-clock diff is negative is a start.

That covers the more common of the "vanishingly uncommon but I've still seen 'em" cases related to file modification detection. Whether you choose to cope with people messing with the file via utime(2) is up to you (past a point, it feels like coping with malicious misuse rather than edge cases). But since your code runs in a loop, you're well-positioned to do that (and detect drift/manipulations of the system clock): track a monotonic clock and use it to approximate the elapsed wall time between honker poller ticks (say it fast with an accent, and you get https://www.bbc.com/news/world-latin-america-11465127); if the timestamp reported by (f)stat(2) ever doesn't advance at the same rate, fall back to checksumming the file, or crashing or something. But this is well into the realm of abject paranoia by now.

It's been a decade or so since I worked in this area, so some of that knowledge is likely stale; you probably know a lot more than I do after developing this library even before considering how out-of-date my knowledge might be. When I worked on this stuff, I remember that statx(2) was going to solve all the problems any day now, and then didn't. More relevant, I also remember that the lsyncd (https://github.com/lsyncd/lsyncd) and watchman (https://github.com/facebook/watchman) codebases were really good sources of "what didn't I think of" information in this area.

But seriously, again, nice work! Those are nitpicks; this is awesome as-is!

Post reply on HN