Very cool approach. I’m working on a related KV storage project (snkv - https://github.com/hash-anu/snkv)
How do you handle manifest consistency and updates—append-only or rewrite-heavy?
Show HN: Turbolite – a SQLite VFS serving sub-250ms cold JOIN queries from S3
31–40 of 55 posts
Re: Show HN: Turbolite – a SQLite VFS serving sub-250ms cold JOIN queries from S3
#32this is a great project, does it support wasm? i want to use it in browser with sqlite wasm.
Re: Show HN: Turbolite – a SQLite VFS serving sub-250ms cold JOIN queries from S3
#33Ovais - co-founder of Tigris here.
This is very cool. I have been thinking about embedded databases running on Tigris. Specially from an agent perspective, agents can suspend and continue their sessions. Would love to collaborate.
Re: Show HN: Turbolite – a SQLite VFS serving sub-250ms cold JOIN queries from S3
#34Re: Show HN: Turbolite – a SQLite VFS serving sub-250ms cold JOIN queries from S3
#35This is awesome! With all of the projects/teams working on improving sqlite, it feels like it's just a matter of time before it becomes a better default than postgres for serious projects. I do wonder - for projects that do ultimately enforce single writer sqlite setups - it still feels to me as if it would always be better to keep the sqlite db local (and then rsync/stream backups to whatever S3 storage one prefers)…
[flagged]
This is LLM-speak. Am I the only one who notices?
Re: Show HN: Turbolite – a SQLite VFS serving sub-250ms cold JOIN queries from S3
#36Nice set of experiments! I appreciate that you're running benchmarks on real object storage setups to validate rapid design variations. (Meta-note: I love how agents have recently made this kind of experimental-research work possible with much less human time investment.)
I've been doing some experiments of my own in a relatively similar space, also focusing on S3/Tigris-backed SQLite on ephemeral compute, also with B-tree aware prefetching (see https://github.com/wjordan/sqlite-prefetch).
I think the idea of storing grouped pages together to optimize read-locality is interesting. Note that it steers in the opposite direction of the temporal locality that a format like LTX/Litestream uses to provide transaction-aware features like point-in time restore. The tradeoff also involves significantly greater write amplification (re-upload the entire page group every time a single page dirties), heavily favoring cold-read-heavy workloads over mixed-write or warm-read workloads.
The query-plan frontrunning is a very novel experiment as well, discovering in advance that SQLite is about to run a full-table scan seems like a very useful optimization hint to work with. I'd love to see experiments validating how much of an improvement that offers compared to simple reactive prefetch (which takes at least a couple page faults to get up to speed).
Re: Show HN: Turbolite – a SQLite VFS serving sub-250ms cold JOIN queries from S3
#37Re: Show HN: Turbolite – a SQLite VFS serving sub-250ms cold JOIN queries from S3
#38Re: Show HN: Turbolite – a SQLite VFS serving sub-250ms cold JOIN queries from S3
#39Very cool approach. I’m working on a related KV storage project (snkv - https://github.com/hash-anu/snkv ) How do you handle manifest consistency and updates—append-only or rewrite-heavy?
Rewrite on every checkpoint. The manifest is small (a few hundred KB for a million pages), so a single PUT is fine. The PUT is the atomic commit point - old page groups become garbage, cleaned up later by gc(). The SQLite engine fetches updated pages from the WAL so turbolite doesn't have to manage manifest updates until checkpoint. If it's not in WAL mode, checkpoints happen on every write so consistency is still there.
Re: Show HN: Turbolite – a SQLite VFS serving sub-250ms cold JOIN queries from S3
#40Ovais - co-founder of Tigris here. This is very cool. I have been thinking about embedded databases running on Tigris. Specially from an agent perspective, agents can suspend and continue their sessions. Would love to collaborate.
Happy to hear from you, I'm a delighted customer. In fact I am building a database company on top of Tigris focused on exactly that use case. I'll reach out on X if that works.