Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Turbolite – a SQLite VFS serving sub-250ms cold JOIN queries from S3

github.com

Post reply on HN
Print view View original

51–55 of 55 posts

Re: Show HN: Turbolite – a SQLite VFS serving sub-250ms cold JOIN queries from S3

#51

Post by pdyc »

russellthehippo wrote: post #46
russellthehippo wrote: post #41

Earlier quoted context omitted.

Not today, but the architecture isn't fundamentally incompatible. The page grouping and seekable compression would translate well to browser fetch + range GETs. It would need a new storage backend targeting OPFS/fetch instead of S3/disk. I'm happy to discuss more if you'd like to open a Github issue - abstracting the storage API seems like a decent idea in itself.

Also I'm curious how you would handle credentials - would you be proxying through your backend? To me turbolite is definitely a backend tool.

correct it will be handled via backend which will just proxy requests to R2(its S3 compatible). i already have something working with vhttpfs but you have implemented some great ideas like optimizing request count instead of byte efficiency, i wanted something like that in vhttpfs but it will become another project for me. I think it can be great frontend tool as well since you have decoupled storage with db engine.

Re: Show HN: Turbolite – a SQLite VFS serving sub-250ms cold JOIN queries from S3

#52

Post by russellthehippo »

russellthehippo wrote: post #7
alex_hirner wrote: post #5

What are your thoughts on eviction, re how easy to add some basic policy?

Great question. I have some eviction functions in the Rust library; I don’t expose them through the extension/VFS yet. The open question is less “can I evict?” and more “when should eviction fire?” via user action, via policy, or both. The obvious policy-driven versions are things like: - when cache size crosses a limit - on checkpoint - every N writes (kind of like autocheckpoint) - after some idle / age threshold M…

I have implemented a full eviction suite for turbolite in https://github.com/russellromney/turbolite/pull/6. I includes connection-time and env-var-set eviction policies (size, count, etc.), user functions for manual eviction at the cache, tree, and even query-level (evict the pages that this query would have downloaded). I hope this definitively answers your question haha

Re: Show HN: Turbolite – a SQLite VFS serving sub-250ms cold JOIN queries from S3

#53

Post by russellthehippo »

hrmtst93837 wrote: post #49
russellthehippo wrote: post #38

Earlier quoted context omitted.

the comment history speaks for itself

[flagged]

your comment doesn't read like llm. the other one does (other fruit entirelY?). i commented in the wrong place. i apologize. >>> AbanoubRodolf 2 days ago | root | parent | prev | next [–] [flagged]

Re: Show HN: Turbolite – a SQLite VFS serving sub-250ms cold JOIN queries from S3

#54

Post by sjdv1982 »

russellthehippo wrote: post #7
alex_hirner wrote: post #5

What are your thoughts on eviction, re how easy to add some basic policy?

Great question. I have some eviction functions in the Rust library; I don’t expose them through the extension/VFS yet. The open question is less “can I evict?” and more “when should eviction fire?” via user action, via policy, or both. The obvious policy-driven versions are things like: - when cache size crosses a limit - on checkpoint - every N writes (kind of like autocheckpoint) - after some idle / age threshold M…

[dead]

Re: Show HN: Turbolite – a SQLite VFS serving sub-250ms cold JOIN queries from S3

#55

Post by wgjordan »

russellthehippo wrote: post #43
wgjordan wrote: post #36

Nice set of experiments! I appreciate that you're running benchmarks on real object storage setups to validate rapid design variations. (Meta-note: I love how agents have recently made this kind of experimental-research work possible with much less human time investment.) I've been doing some experiments of my own in a relatively similar space, also focusing on S3/Tigris-backed SQLite on ephemeral compute, also with…

Fantastic comment, thanks for jumping in. I have well-tuned replies here lol given we're working on exactly the same problems! First: your prefetch solution is precisely what I have in the roadmap for the next level of optimization: B-tree introspection at the page-child level not the table level. I haven't launched it yet as I only realized the potential in the past couple days and I wanted to focus on stability bef…

Thanks for the extra details, the frontrun benchmark numbers seem compelling for various cold-read use cases.

A system that combines both WAL frames with cold-read-optimized grouped pages is another interesting point in the design-space. Tuning the intervals separately could make it work well- frequent WAL-checkpoint uploads, and grouped pages only on higher-level compaction cycles for cold-read optimization on longer-lived objects.

Looking forward to seeing where you head with this!

Post reply on HN