Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Honker – Durable queues, streams, pub/sub, and cron scheduler in a SQLite file

honker.dev

Post reply on HN
Print view View original

11–20 of 69 posts

Re: Honker – Durable queues, streams, pub/sub, and cron scheduler in a SQLite file

#11

Post by ncruces »

tptacek wrote: post #5

"Idle cost is that one lightweight SELECT per millisecond per database — no page-cache pressure, no writer-lock contention, no kernel file watcher in the mix." I think (respectfully) the LLM that probably wrote this overshot the mark here because busy-polling a select does not actually sound better to me than a "kernel file watcher".

If you're not making any changes to the database, does the SELECT "kill" you?

And if you are making changes, don't you have to poll regardless after the file watcher wakes you?

For WAL mode, SQLite can probably satisfy this query just by inspecting some shared memory. But it is busy waiting, sure.

Re: Honker – Durable queues, streams, pub/sub, and cron scheduler in a SQLite file

#12

Post by andai »

d1l wrote: post #10
tptacek wrote: post #5

"Idle cost is that one lightweight SELECT per millisecond per database — no page-cache pressure, no writer-lock contention, no kernel file watcher in the mix." I think (respectfully) the LLM that probably wrote this overshot the mark here because busy-polling a select does not actually sound better to me than a "kernel file watcher".

Yeah, I had the same instinct - this feels very much like a "nice idea" but the execution falls short. I mean - busily banging on sqlite like this? Shit at that point just use Redis.

What's the CPU usage? Like 2%?

I had a manual fs polling thing a while back. It was ugly (low time budget, didn't wanna mess with the native watchers), just scanned the whole thing once per second. It averaged out to like 0.3% CPU.

Not elegant, but acceptable for my purposes! (Small-ish directory, and "ping me within a second or two" was realtime enough for this use case.)

Re: Honker – Durable queues, streams, pub/sub, and cron scheduler in a SQLite file

#13

Post by tptacek »

d1l wrote: post #10
tptacek wrote: post #5

"Idle cost is that one lightweight SELECT per millisecond per database — no page-cache pressure, no writer-lock contention, no kernel file watcher in the mix." I think (respectfully) the LLM that probably wrote this overshot the mark here because busy-polling a select does not actually sound better to me than a "kernel file watcher".

Yeah, I had the same instinct - this feels very much like a "nice idea" but the execution falls short. I mean - busily banging on sqlite like this? Shit at that point just use Redis.

I'm not even saying it's unworkable, just, my intuition is not that the "lightweight per-millisecond select" is an optimal design.

Re: Honker – Durable queues, streams, pub/sub, and cron scheduler in a SQLite file

#14

Post by giraffe_lady »

felooboolooomba wrote: post #8
tptacek wrote: post #5

"Idle cost is that one lightweight SELECT per millisecond per database — no page-cache pressure, no writer-lock contention, no kernel file watcher in the mix." I think (respectfully) the LLM that probably wrote this overshot the mark here because busy-polling a select does not actually sound better to me than a "kernel file watcher".

"one lightweight SELECT per millisecond" This reminds me of the teenager who told her dad that she was just a tiny little bit pregnant.

[flagged]

Re: Honker – Durable queues, streams, pub/sub, and cron scheduler in a SQLite file

#15

Post by giraffe_lady »

tptacek wrote: post #13
d1l wrote: post #10

Earlier quoted context omitted.

Yeah, I had the same instinct - this feels very much like a "nice idea" but the execution falls short. I mean - busily banging on sqlite like this? Shit at that point just use Redis.

I'm not even saying it's unworkable, just, my intuition is not that the "lightweight per-millisecond select" is an optimal design.

Really might be in sqlite. I've learned to never trust my intuition about performance with that thing. So many times I've gone to "optimize" something and discovered that the naive hack way I had been doing it was faster anyway. It's built for this sort of bullshit.

Re: Honker – Durable queues, streams, pub/sub, and cron scheduler in a SQLite file

#16

Post by tptacek »

giraffe_lady wrote: post #14
felooboolooomba wrote: post #8

Earlier quoted context omitted.

"one lightweight SELECT per millisecond" This reminds me of the teenager who told her dad that she was just a tiny little bit pregnant.

[flagged]

Yeah, again, to be clear: I get how SQLite works and I'm not dunking on the design, I'm just saying the comparison set up on this page snags. It's a classic LLM negated triptych, but "one of these things is not like the other": cache pressure: bad, writer contention: bad, kernel file watcher: ... good, actually? Intuitively seems better than this design?

Re: Honker – Durable queues, streams, pub/sub, and cron scheduler in a SQLite file

#17

Post by tptacek »

giraffe_lady wrote: post #15
tptacek wrote: post #13

Earlier quoted context omitted.

I'm not even saying it's unworkable, just, my intuition is not that the "lightweight per-millisecond select" is an optimal design.

Really might be in sqlite. I've learned to never trust my intuition about performance with that thing. So many times I've gone to "optimize" something and discovered that the naive hack way I had been doing it was faster anyway. It's built for this sort of bullshit.

Maybe, I'm really writing about the language on this page, not about the design (I responded about this upthread).

Re: Honker – Durable queues, streams, pub/sub, and cron scheduler in a SQLite file

#18

Post by koito17 »

d1l wrote: post #10
tptacek wrote: post #5

"Idle cost is that one lightweight SELECT per millisecond per database — no page-cache pressure, no writer-lock contention, no kernel file watcher in the mix." I think (respectfully) the LLM that probably wrote this overshot the mark here because busy-polling a select does not actually sound better to me than a "kernel file watcher".

Yeah, I had the same instinct - this feels very much like a "nice idea" but the execution falls short. I mean - busily banging on sqlite like this? Shit at that point just use Redis.

For what it's worth, Kine (software that k3s uses to replace etcd with SQL databases) implements etcd watches on SQLite through polling[1]. The reason being that SQLite does not offer NOTIFY/LISTEN like MySQL and Postgres do. Ironically, Honkey attempts implementing NOTIFY/LISTEN through polling.

k3s has been running on my home server for about three years now (using the default SQLite backend), and there doesn't seem to be excessive CPU usage despite dozens of watches existing in the simulated etcd. Of course, this doesn't say much about Honker, but it's nonetheless worth pointing out that sometimes the choice of database forces one towards a certain design.

[1] https://github.com/k3s-io/kine/blob/648a2daa/pkg/logstructur...

Re: Honker – Durable queues, streams, pub/sub, and cron scheduler in a SQLite file

#19

Post by giraffe_lady »

tptacek wrote: post #17
giraffe_lady wrote: post #15

Earlier quoted context omitted.

Really might be in sqlite. I've learned to never trust my intuition about performance with that thing. So many times I've gone to "optimize" something and discovered that the naive hack way I had been doing it was faster anyway. It's built for this sort of bullshit.

Maybe, I'm really writing about the language on this page, not about the design (I responded about this upthread).

Oh, yes, I see what you mean now.

Re: Honker – Durable queues, streams, pub/sub, and cron scheduler in a SQLite file

#20

Post by rv64imafdc »

giraffe_lady wrote: post #14
felooboolooomba wrote: post #8

Earlier quoted context omitted.

"one lightweight SELECT per millisecond" This reminds me of the teenager who told her dad that she was just a tiny little bit pregnant.

[flagged]

Hold on -- if it really is "one lightweight SELECT per millisecond", and you're saying a select is "a couple hundred microseconds", say generously 200us?, then you're spending 200us out of every 1000us just selecting. That's a lot of polling!
Post reply on HN