Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Honker – Durable queues, streams, pub/sub, and cron scheduler in a SQLite file

honker.dev

Post reply on HN
Print view View original

61–69 of 69 posts

Re: Honker – Durable queues, streams, pub/sub, and cron scheduler in a SQLite file

#61

Post by Retr0id »

nine_k wrote: post #47
felooboolooomba wrote: post #8

Earlier quoted context omitted.

"one lightweight SELECT per millisecond" This reminds me of the teenager who told her dad that she was just a tiny little bit pregnant.

One cannot be a little bit pregnant. But a DB can be only a little bit in the RAM, and specifically in the page cache. SQLite can act exactly like that, and it's damn fast as long as it does not need to durably write a transaction. Polling once a millisecond could spend a few microseconds. I wonder if using a tiny Redis instance, or even something like LevelDB would be even more efficient.

With the file-watch APIs is that you don't need to poll at all - free is better than cheap.

Re: Honker – Durable queues, streams, pub/sub, and cron scheduler in a SQLite file

#62

Post by SQLite »

tengbretson wrote: post #57

I'm a big fan of SQLite and all that, but if SQLite constrains you to a single writer process, why not do this in your application layer anyway?

SQLite allows multiple writers. The constraint is that only one of how writers can be actively writing at any moment in time. If there are multiple processes wanting to write, they take turns. SQLite prevents two or more writes from running concurrently, so there is nothing the application needs to do to implement this, other than responding to SQLITE_BUSY replies from failed (concurrent) write attempts and retrying after a short delay.

Why this constraint? Because SQLite is serverless. There is no central server available to coordinate concurrent writes.

At the lowest level of the stack, every database engine has this same constraint, as there is only one wire connecting the CPU to the SSD, and you cannot send multiple writes over the same wire at the same time. But in a client/server database, the server (in cooperation with the filesystem) is at hand to serialize the writes and prevent problems in ways that are not possible without a server. The server creates the illusion of concurrent writes by multiplexing the single write wire efficiently and making that multiplexing transparent to the application.

Re: Honker – Durable queues, streams, pub/sub, and cron scheduler in a SQLite file

#64

Post by ncruces »

russellthehippo wrote: post #43
tptacek wrote: post #5

"Idle cost is that one lightweight SELECT per millisecond per database — no page-cache pressure, no writer-lock contention, no kernel file watcher in the mix." I think (respectfully) the LLM that probably wrote this overshot the mark here because busy-polling a select does not actually sound better to me than a "kernel file watcher".

Respectfully (thanks haha) - yeah probably right. Original intent was to use inotify type thing but i avoided per-platform differences at the outset. this was definitely a for fun project that blew up unintentionally and am working to harden/improve. Love Fly.

One of the things people seem to forget is that SQLite itself polls every millisecond or so to grab a lock.

So yes, don't use this in a mobile device, or a server if you want to let the CPU enter a low power state.

Otherwise, a single thread doing this in an otherwise idle server, doesn't seem that terrible. And if it's not idle, inotify won't help you (need to query what changed afterwards).

Re: Honker – Durable queues, streams, pub/sub, and cron scheduler in a SQLite file

#66

Post by andai »

kimixa wrote: post #59
andai wrote: post #12

Earlier quoted context omitted.

What's the CPU usage? Like 2%? I had a manual fs polling thing a while back. It was ugly (low time budget, didn't wanna mess with the native watchers), just scanned the whole thing once per second. It averaged out to like 0.3% CPU. Not elegant, but acceptable for my purposes! (Small-ish directory, and "ping me within a second or two" was realtime enough for this use case.)

If this stops the core being able to drop to a lower power state it can be whole multiples of power use on some devices. Wake ups are death for mobile form factors, even if not really doing much work.

This is a pretty good argument against the way we do operating systems now, right?

Re: Honker – Durable queues, streams, pub/sub, and cron scheduler in a SQLite file

#67

Post by russellthehippo »

ncruces wrote: post #64
russellthehippo wrote: post #43

Earlier quoted context omitted.

Respectfully (thanks haha) - yeah probably right. Original intent was to use inotify type thing but i avoided per-platform differences at the outset. this was definitely a for fun project that blew up unintentionally and am working to harden/improve. Love Fly.

One of the things people seem to forget is that SQLite itself polls every millisecond or so to grab a lock. So yes, don't use this in a mobile device, or a server if you want to let the CPU enter a low power state. Otherwise, a single thread doing this in an otherwise idle server, doesn't seem that terrible. And if it's not idle, inotify won't help you (need to query what changed afterwards).

Appreciated your input on the original thread as well. Maybe I should note this recommendation in the docs or something.

Re: Honker – Durable queues, streams, pub/sub, and cron scheduler in a SQLite file

#68

Post by kimixa »

andai wrote: post #66
kimixa wrote: post #59

Earlier quoted context omitted.

If this stops the core being able to drop to a lower power state it can be whole multiples of power use on some devices. Wake ups are death for mobile form factors, even if not really doing much work.

This is a pretty good argument against the way we do operating systems now, right?

Why? Most modern OSs are "tickless" - where there's no regular scheduling tick and it can sleep pretty much indefinitely if there's no work.
Post reply on HN