[flagged]
Show HN: Honker – Postgres NOTIFY/LISTEN Semantics for SQLite
11–20 of 95 posts
Re: Show HN: Honker – Postgres NOTIFY/LISTEN Semantics for SQLite
#12Wouldn't processes on same machine be able to use different IPCs that don't even touch file ? It's neat but I have feeling in vast majority of cases just passing address to one of the IPC methods would be faster and then SQLite itself would only be needed for the durable parts.
Re: Show HN: Honker – Postgres NOTIFY/LISTEN Semantics for SQLite
#13Hey HN, I built this. Honker adds cross-process NOTIFY/LISTEN to SQLite. You get push-style event delivery with single-digit millisecond latency without a damon/broker, using your existing SQLite file. A lot of pretty high-traffic applications are just Framework+SQLite+Litestream on a VPS now, so I wanted to bring a sixer to the "just use SQLite" party. SQLite doesn't run a server like Postgres, so the trick is movin…
This is really interesting. I'm building something on Postgresql with LISTEN/NOTIFY and Postgraphile. I'd love to (in theory) be able to have a swappable backend and not be so tightly coupled to the database server.
Re: Show HN: Honker – Postgres NOTIFY/LISTEN Semantics for SQLite
#14Hey HN, I built this. Honker adds cross-process NOTIFY/LISTEN to SQLite. You get push-style event delivery with single-digit millisecond latency without a damon/broker, using your existing SQLite file. A lot of pretty high-traffic applications are just Framework+SQLite+Litestream on a VPS now, so I wanted to bring a sixer to the "just use SQLite" party. SQLite doesn't run a server like Postgres, so the trick is movin…
Nice, I had no idea that stat() every 1 ms is so affordable. Aparently it takes less than 1 μs per call on my hardware, so that's less than 0.1% cpu time for polling.
"Syscalls are slow" is only mostly true. They are slower than not having to cross the userspace OS barrier at all, but they're not "slow" like cross-ocean network calls can be. For example, non-VDSO syscalls in linux are about 250 nanoseconds (see for example https://arkanis.de/weblog/2017-01-05-measurements-of-system-...), VDSO syscalls are roughly 10x faster. Slower than userspace function calls for sure, but more than affordable outside the hottest of loops.
Re: Show HN: Honker – Postgres NOTIFY/LISTEN Semantics for SQLite
#15Wouldn't processes on same machine be able to use different IPCs that don't even touch file ? It's neat but I have feeling in vast majority of cases just passing address to one of the IPC methods would be faster and then SQLite itself would only be needed for the durable parts.
This extension piggyback SQLite native transactions. For example, queueing data will be rolled back if transaction is rolled back due to some constrains violations.
It is possible to achieve with external IPC, but require a lot of very careful programming.
Re: Show HN: Honker – Postgres NOTIFY/LISTEN Semantics for SQLite
#16Earlier quoted context omitted.
Nice, I had no idea that stat() every 1 ms is so affordable. Aparently it takes less than 1 μs per call on my hardware, so that's less than 0.1% cpu time for polling.
"Syscalls are slow" is only mostly true. They are slower than not having to cross the userspace OS barrier at all, but they're not "slow" like cross-ocean network calls can be. For example, non-VDSO syscalls in linux are about 250 nanoseconds (see for example https://arkanis.de/weblog/2017-01-05-measurements-of-system-... ), VDSO syscalls are roughly 10x faster. Slower than userspace function calls for sure, but more…
That’s ignoring the other costs of syscalls like evicting your stuff from the CPU caches.
But I agree with the conclusion, system calls are still pretty fast compared to a lot of other things.
Re: Show HN: Honker – Postgres NOTIFY/LISTEN Semantics for SQLite
#17atomic commit with the business data is the selling point over separate IPC. external message passing always has the 'notification sent but transaction rolled back' problem and that gets messy.
one thing i'm curious about: WAL checkpoint. when SQLite truncates WAL back to zero, does the stat() polling handle that correctly? feels like there's a window where events could get lost.
Re: Show HN: Honker – Postgres NOTIFY/LISTEN Semantics for SQLite
#18Thanks for this!
I have a proliferation of small apps backed by SQLite. And most of these need a queue and scheduler.
I home rolled some stuff for it but was always pining for the elegance of the Postgres solutions.
Will give this a spin very soon
Re: Show HN: Honker – Postgres NOTIFY/LISTEN Semantics for SQLite
#19Earlier quoted context omitted.
Nice, I had no idea that stat() every 1 ms is so affordable. Aparently it takes less than 1 μs per call on my hardware, so that's less than 0.1% cpu time for polling.
"Syscalls are slow" is only mostly true. They are slower than not having to cross the userspace OS barrier at all, but they're not "slow" like cross-ocean network calls can be. For example, non-VDSO syscalls in linux are about 250 nanoseconds (see for example https://arkanis.de/weblog/2017-01-05-measurements-of-system-... ), VDSO syscalls are roughly 10x faster. Slower than userspace function calls for sure, but more…
Filesystem stuff tends to be slower than average syscalls because of all the locks and complicated traversals needed. If this is using stat instead of fstat then it’s also going through the VFS layer - repeated calls likely go through the cache fast path for path resolution but accessing the stat structure. There’s also hidden costs in that number like atomic accesses that need to acquire cache line locks that are going to cause hidden contention for other processes on the CPU + the cache dirtying from running kernel code and then subsequently having to repopulate it when leaving all of which adds contended L3/RAM pressure.
In other words, there’s a lot of unmeasured performance degradation that’s a side effect of doing many syscalls above and beyond the CPU time to enter/leave the kernel which itself has shrunk to be negligible. But there’s a reason high performance code is switching to io_uring to avoid that.
Re: Show HN: Honker – Postgres NOTIFY/LISTEN Semantics for SQLite
#20Earlier quoted context omitted.
"Syscalls are slow" is only mostly true. They are slower than not having to cross the userspace OS barrier at all, but they're not "slow" like cross-ocean network calls can be. For example, non-VDSO syscalls in linux are about 250 nanoseconds (see for example https://arkanis.de/weblog/2017-01-05-measurements-of-system-... ), VDSO syscalls are roughly 10x faster. Slower than userspace function calls for sure, but more…
That’s ignoring the other costs of syscalls like evicting your stuff from the CPU caches. But I agree with the conclusion, system calls are still pretty fast compared to a lot of other things.
Small correction on ambiguous wording - syscalls do not evict all your stuff from CPU caches. It just has to page in whatever is needed for kernel code/data accessed by the call, but that’s no different from if it was done in process as a normal function call.