Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Turbopuffer: Fast search on object storage

turbopuffer.com

Post reply on HN
Print view View original

31–40 of 68 posts

Re: Turbopuffer: Fast search on object storage

#31

Post by cdchn »

eknkc wrote: post #9

Is there a good general purpose solution where I can store a large read only database in s3 or something and do lookups directly on it? Duckdb can open parquet files over http and query them but I found it to trigger a lot of small requests reading bunch of places from the files. I mean a lot. I mostly need key / value lookups and could potentially store each key in a seperate object in s3 but for a couple hundred mi…

>Is there a good general purpose solution where I can store a large read only database in s3 or something and do lookups directly on it?

I think this is pretty much what AWS Athena is.

Re: Turbopuffer: Fast search on object storage

#33

Post by arcanemachiner »

tionis wrote: post #22
eknkc wrote: post #9

Is there a good general purpose solution where I can store a large read only database in s3 or something and do lookups directly on it? Duckdb can open parquet files over http and query them but I found it to trigger a lot of small requests reading bunch of places from the files. I mean a lot. I mostly need key / value lookups and could potentially store each key in a seperate object in s3 but for a couple hundred mi…

You could use a sqlite database and use range queries using something like this: https://github.com/psanford/sqlite3vfshttp https://github.com/phiresky/sql.js-httpvfs Simon Willison wrote about it: https://simonwillison.net/2022/Aug/10/sqlite-http/

That whole thing still blows my mind.

Re: Turbopuffer: Fast search on object storage

#34

Post by sitkack »

swyx wrote: post #25
cmcollier wrote: post #3

Unrelated to the core topic, I really enjoy the aesthetic of their website. Another similar one is from Fixie.ai (also, interestingly, one of their customers).

what does fixie do these days?

They pivoted, but will probably pivot back to their original quest.

Re: Turbopuffer: Fast search on object storage

#36

Post by k2so »

cmcollier wrote: post #3

Unrelated to the core topic, I really enjoy the aesthetic of their website. Another similar one is from Fixie.ai (also, interestingly, one of their customers).

This was my first thought too, after reading through their blog. This feels like a no-frills website made by an engineer, who makes things that just work.

The documentation is great, I really appreciate them putting the roadmap front and centre.

Re: Turbopuffer: Fast search on object storage

#37

Post by TechDebtDevin »

nh2 wrote: post #28

> $3600.00/TB/month It doesn't have to be that way. At Hetzner I pay $200/TB/month for RAM. That's 18x cheaper. Sometimes you can reach the goal faster with less complexity by removing the part with the 20x markup.

I will likely never leave Hetzner.

Re: Turbopuffer: Fast search on object storage

#38

Post by eknkc »

tionis wrote: post #22
eknkc wrote: post #9

Is there a good general purpose solution where I can store a large read only database in s3 or something and do lookups directly on it? Duckdb can open parquet files over http and query them but I found it to trigger a lot of small requests reading bunch of places from the files. I mean a lot. I mostly need key / value lookups and could potentially store each key in a seperate object in s3 but for a couple hundred mi…

You could use a sqlite database and use range queries using something like this: https://github.com/psanford/sqlite3vfshttp https://github.com/phiresky/sql.js-httpvfs Simon Willison wrote about it: https://simonwillison.net/2022/Aug/10/sqlite-http/

Yep this thing is the reason I thought about doing it in the first place. Tried duckdb which has built in support for range requests over http.

Whole idea makes sense but I feel like the file format should be specifically tuned for this use case. Otherwise you end up with a lot of range requests because it was designed for disk access. I wondered if anything was actually designed for that.

Re: Turbopuffer: Fast search on object storage

#39

Post by eknkc »

jiggawatts wrote: post #16
eknkc wrote: post #9

Is there a good general purpose solution where I can store a large read only database in s3 or something and do lookups directly on it? Duckdb can open parquet files over http and query them but I found it to trigger a lot of small requests reading bunch of places from the files. I mean a lot. I mostly need key / value lookups and could potentially store each key in a seperate object in s3 but for a couple hundred mi…

> trigger a lot of small requests reading bunch of places from the files. I mean a lot. That’s… the whole point. That’s how Parquet files are supposed to be used. They’re an improvement over CSV or JSON because clients can read small subsets of them efficiently! For comparison, I’ve tried a few other client products that don’t use Parquet files properly and just read the whole file every time, no matter how trivial t…

This makes sense but the problem I had with duckdb + parquet is it looks like there is no metadata caching so each and every query triggers a lot of requests.

Duckdb can query a remote duckdb database too, in that case it looks like there is caching. Which might be better.

I wonder if anyone actually worked on a specific file format for this use case (relatively high latency random access) to minimize reads to as little blocks as possible.

Re: Turbopuffer: Fast search on object storage

#40

Post by jiggawatts »

eknkc wrote: post #39
jiggawatts wrote: post #16

Earlier quoted context omitted.

> trigger a lot of small requests reading bunch of places from the files. I mean a lot. That’s… the whole point. That’s how Parquet files are supposed to be used. They’re an improvement over CSV or JSON because clients can read small subsets of them efficiently! For comparison, I’ve tried a few other client products that don’t use Parquet files properly and just read the whole file every time, no matter how trivial t…

This makes sense but the problem I had with duckdb + parquet is it looks like there is no metadata caching so each and every query triggers a lot of requests. Duckdb can query a remote duckdb database too, in that case it looks like there is caching. Which might be better. I wonder if anyone actually worked on a specific file format for this use case (relatively high latency random access) to minimize reads to as lit…

Sounds like a bug or missing feature in DuckDB more than an issue with the format
Post reply on HN