Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Turbopuffer: Fast search on object storage

turbopuffer.com

Post reply on HN
Print view View original

21–30 of 68 posts

Re: Turbopuffer: Fast search on object storage

#21

Post by masterj »

imiric wrote: post #17
eknkc wrote: post #9

Is there a good general purpose solution where I can store a large read only database in s3 or something and do lookups directly on it? Duckdb can open parquet files over http and query them but I found it to trigger a lot of small requests reading bunch of places from the files. I mean a lot. I mostly need key / value lookups and could potentially store each key in a seperate object in s3 but for a couple hundred mi…

ClickHouse can also read from S3. I'm not sure how it compares to DuckDB re efficiency, but it worked fine for my simple use case.

Neither of these support indexes afaik. They are designed to do fast scans / computation.

Re: Turbopuffer: Fast search on object storage

#22

Post by tionis »

eknkc wrote: post #9

Is there a good general purpose solution where I can store a large read only database in s3 or something and do lookups directly on it? Duckdb can open parquet files over http and query them but I found it to trigger a lot of small requests reading bunch of places from the files. I mean a lot. I mostly need key / value lookups and could potentially store each key in a seperate object in s3 but for a couple hundred mi…

You could use a sqlite database and use range queries using something like this: https://github.com/psanford/sqlite3vfshttp https://github.com/phiresky/sql.js-httpvfs

Simon Willison wrote about it: https://simonwillison.net/2022/Aug/10/sqlite-http/

Re: Turbopuffer: Fast search on object storage

#23

Post by hipadev23 »

That’s some woefully disappointing and incorrect metrics (read and write latency are both sub-second, storage medium would be “ Memory + Replicated SSDs”) you’ve got for Clickhouse there, but I understand what you’re going for and why you categorized it where you did.

Re: Turbopuffer: Fast search on object storage

#24

Post by endisneigh »

Slightly relevant - do people really want article recommendations? I don’t think I’ve ever read an article and wanted a recommendation. Even with this one - I sort of read it and that’s it; no feeling of wanting recommendations.

Am I alone in this?

In any case this seems like a pretty interesting approach. Reminds me of Warpstream which does something similar with S3 to replace Kafka.

Re: Turbopuffer: Fast search on object storage

#26

Post by xarope »

cmcollier wrote: post #3

Unrelated to the core topic, I really enjoy the aesthetic of their website. Another similar one is from Fixie.ai (also, interestingly, one of their customers).

Yes, I like the turboxyz123 animation and contrast to the minimalist website (reminds me of the zen garden with a single rock). I think people forget nowadays in their haste to add the latest and greatest react animation, that too much noise is a thing.

Re: Turbopuffer: Fast search on object storage

#27

Post by singhrac »

pushrax wrote: post #6
bigbones wrote: post #4

Sounds like a source-unavailable version of Quickwit? https://quickwit.io/

LSM tree storage engine vs time series storage engine, similar philosophy but different use cases

Maybe I misunderstood both products but I think neither Quickwit or Turbopuffer is either of those things intrinsically (though log structured messages are a good fit for Quickfit). I think Quickwit is essentially Lucene/Elasticsearch (i.e. sparse queries or BM25) and Turbopuffer does vector search (or dense queries) like say Faiss/Pinecone/Qdrant/Vectorize, both over object storage.

Re: Turbopuffer: Fast search on object storage

#29

Post by hodgesrm »

masterj wrote: post #21
imiric wrote: post #17

Earlier quoted context omitted.

ClickHouse can also read from S3. I'm not sure how it compares to DuckDB re efficiency, but it worked fine for my simple use case.

Neither of these support indexes afaik. They are designed to do fast scans / computation.

It depends on what you mean by "support." ClickHouse as I recall can read min/max indexes from Parquet row groups. One of my colleagues is working on a PR to add support for bloom filter indexes. So that will be covered as well.

Right now one of the main performance problems is that Clickhouse does not cache index metadata yet, so you still have to scan files rather than keeping the metadata in memory. ClickHouse does this for native MergeTree tables. There are a couple of steps to get there but I have no doubt that metadata caching will be properly handled soon.

Disclaimer: I work for Altinity, an enterprise provider for ClickHouse software.

Post reply on HN