Turbopuffer: Fast search on object storage • Hacker News Board

Re: Turbopuffer: Fast search on object storage

#51

Post by pushrax » Wed, Jul 10, 2024, 1:24 PM UTC

singhrac wrote: post #27

pushrax wrote: post #6

Earlier quoted context omitted.

LSM tree storage engine vs time series storage engine, similar philosophy but different use cases

Maybe I misunderstood both products but I think neither Quickwit or Turbopuffer is either of those things intrinsically (though log structured messages are a good fit for Quickfit). I think Quickwit is essentially Lucene/Elasticsearch (i.e. sparse queries or BM25) and Turbopuffer does vector search (or dense queries) like say Faiss/Pinecone/Qdrant/Vectorize, both over object storage.

It's true that turbopuffer does vector search, though it also does BM25.

The biggest difference at a low level is that turbopuffer records have unique primary keys, and can be updated, like in a normal database. Old records that were overwritten won't be returned in searches. The LSM tree storage engine is used to achieve this. The LSM tree also enables maintenance of global indexes that can be used for efficient retrieval without any time-based filter.

Quickwit records are immutable. You can't overwrite a record (well, you can, but overwritten records will also be returned in searches). The data files it produces are organized into a time series, and if you don't pass a time-based filter it has to look at every file.

Re: Turbopuffer: Fast search on object storage

#52

Post by Sirupsen » Wed, Jul 10, 2024, 1:36 PM UTC

zX41ZdbW wrote: post #35

A correction to the article. It mentions Warehouse BigQuery, Snowflake, Clickhouse ≥1s Minutes For ClickHouse, it should be: read latency Logging, real-time analytics, and RAG are also suitable for ClickHouse.

Yeah, thinking about this more I now understand Clickhouse to be more of an operational warehouse similar to Materialize, Pinot, Druid, etc. if I understand correctly? So bunching with BigQuery/Snowflake/Trino/Databricks... wasn't the right category (although operational warehouses certainly can have a ton of overlap)

I left that category out for simplicity (plenty of others that didn't make it into the taxonomy, e.g. queues, nosql, time-series, graph, embedded, ..)

Re: Turbopuffer: Fast search on object storage

#53

Post by canadiantim » Wed, Jul 10, 2024, 2:11 PM UTC

eknkc wrote: post #9

Is there a good general purpose solution where I can store a large read only database in s3 or something and do lookups directly on it? Duckdb can open parquet files over http and query them but I found it to trigger a lot of small requests reading bunch of places from the files. I mean a lot. I mostly need key / value lookups and could potentially store each key in a seperate object in s3 but for a couple hundred mi…

LanceDB

Re: Turbopuffer: Fast search on object storage

#54

Post by hodgesrm » Wed, Jul 10, 2024, 2:32 PM UTC

AYBABTME wrote: post #41

nh2 wrote: post #28

> $3600.00/TB/month It doesn't have to be that way. At Hetzner I pay $200/TB/month for RAM. That's 18x cheaper. Sometimes you can reach the goal faster with less complexity by removing the part with the 20x markup.

200$/TB/month for raw RAM, not RAM that's presented to you behind a usable API that's distributed and operated by someone else, freeing you of time. It's not particularly useful to compare the cost of raw unorganized information medium on a single node, to highly organized information platform. It's like saying "this CPU chip is expensive, just look at the price of this sand".

> It's not particularly useful to compare the cost of raw unorganized information medium on a single node, to highly organized information platform.

Except that it does prompt you to ask what you could do to use that cheap compute and RAM. In the case of Hetzner that might be large caches that allow you to apply those resources on remote data whilst minimizing transfer and API costs.

Re: Turbopuffer: Fast search on object storage

#55

Post by zkoch » Wed, Jul 10, 2024, 2:52 PM UTC

sitkack wrote: post #34

swyx wrote: post #25

Earlier quoted context omitted.

what does fixie do these days?

They pivoted, but will probably pivot back to their original quest.

Nah, we're pretty happy with the new trajectory. :)

Re: Turbopuffer: Fast search on object storage

#56

Post by singhrac » Wed, Jul 10, 2024, 3:13 PM UTC

pushrax wrote: post #51

singhrac wrote: post #27

Earlier quoted context omitted.

Maybe I misunderstood both products but I think neither Quickwit or Turbopuffer is either of those things intrinsically (though log structured messages are a good fit for Quickfit). I think Quickwit is essentially Lucene/Elasticsearch (i.e. sparse queries or BM25) and Turbopuffer does vector search (or dense queries) like say Faiss/Pinecone/Qdrant/Vectorize, both over object storage.

It's true that turbopuffer does vector search, though it also does BM25. The biggest difference at a low level is that turbopuffer records have unique primary keys, and can be updated, like in a normal database. Old records that were overwritten won't be returned in searches. The LSM tree storage engine is used to achieve this. The LSM tree also enables maintenance of global indexes that can be used for efficient ret…

Ah I didn’t catch that Quickwit had immutable records. That explains the focus on log usage. Thanks!

Re: Turbopuffer: Fast search on object storage

#57

Post by 5- » Wed, Jul 10, 2024, 3:35 PM UTC

cmcollier wrote: post #3

Unrelated to the core topic, I really enjoy the aesthetic of their website. Another similar one is from Fixie.ai (also, interestingly, one of their customers).

indeed! what a nice, minimal page... that comes with ~1.6mb of javascript.

Re: Turbopuffer: Fast search on object storage

#58

Post by omneity » Wed, Jul 10, 2024, 3:58 PM UTC

lyu07282 wrote: post #50

omneity wrote: post #19

Earlier quoted context omitted.

Tried and deployed in production with similar sized collections. You only need enough memory to load the index, definitely not the whole collection. A typical index would most likely fit within a few GBs. And even if you need dozens of GBs of RAM it won’t cost nearly as much as $20k/month as the article surmises.

How do you get to "a few GBs"? A hundred million embeddings, if you have 4 byte floats 1024 dimensions would be >400 GB alone.

I did say the index, not the embeddings themselves. The index is a more compact representation of your embeddings collection, and that's what you need in memory. One approach for indexing is to calculate centroids of your embeddings.

You have multiple parameters to tweak, that affect retrieval performance as well as the memory footprint of your indexes. Here's a rundown on that: https://tembo.io/blog/vector-indexes-in-pgvector

Re: Turbopuffer: Fast search on object storage

#59

Post by solatic » Wed, Jul 10, 2024, 4:19 PM UTC

Is it feasible to try to build this kind of approach (hot SSD cache nodes sitting in front of object storage) with prior open-source art (Lucene)? Or are the search indexes themselves also proprietary in this solution?

Having witnessed some very large Elasticsearch production deployments, being able to throw everything into S3 would be incredible. The applicability here isn't only for vector search.

Re: Turbopuffer: Fast search on object storage

#60

Post by francoismassot » Wed, Jul 10, 2024, 4:55 PM UTC

solatic wrote: post #59

Is it feasible to try to build this kind of approach (hot SSD cache nodes sitting in front of object storage) with prior open-source art (Lucene)? Or are the search indexes themselves also proprietary in this solution? Having witnessed some very large Elasticsearch production deployments, being able to throw everything into S3 would be incredible . The applicability here isn't only for vector search.

If you don't need vector search and have very large Elasticsearch deployment, you can have a look at Quickwit, it's a search engine on object storage, it's OSS and works for append-only datasets (like logs, traces, ...)

Repo: https://github.com/quickwit-oss/quickwit