Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Turbopuffer: Fast search on object storage

turbopuffer.com

Post reply on HN
Print view View original

61–68 of 68 posts

Re: Turbopuffer: Fast search on object storage

#61

Post by rohitnair »

solatic wrote: post #59

Is it feasible to try to build this kind of approach (hot SSD cache nodes sitting in front of object storage) with prior open-source art (Lucene)? Or are the search indexes themselves also proprietary in this solution? Having witnessed some very large Elasticsearch production deployments, being able to throw everything into S3 would be incredible . The applicability here isn't only for vector search.

Elasticsearch and OpenSearch already support S3 backed indices. See features like https://opensearch.org/docs/latest/tuning-your-cluster/avail... The files in S3 are plain old Lucene segment files (just wrapped in OpenSearch snapshots which provide a way to track metadata around those files).

Re: Turbopuffer: Fast search on object storage

#62

Post by francoismassot »

rohitnair wrote: post #61
solatic wrote: post #59

Is it feasible to try to build this kind of approach (hot SSD cache nodes sitting in front of object storage) with prior open-source art (Lucene)? Or are the search indexes themselves also proprietary in this solution? Having witnessed some very large Elasticsearch production deployments, being able to throw everything into S3 would be incredible . The applicability here isn't only for vector search.

Elasticsearch and OpenSearch already support S3 backed indices. See features like https://opensearch.org/docs/latest/tuning-your-cluster/avail... The files in S3 are plain old Lucene segment files (just wrapped in OpenSearch snapshots which provide a way to track metadata around those files).

But you don’t have fast search on those files stored on object storage.

Re: Turbopuffer: Fast search on object storage

#63

Post by rohitnair »

francoismassot wrote: post #62
rohitnair wrote: post #61

Earlier quoted context omitted.

Elasticsearch and OpenSearch already support S3 backed indices. See features like https://opensearch.org/docs/latest/tuning-your-cluster/avail... The files in S3 are plain old Lucene segment files (just wrapped in OpenSearch snapshots which provide a way to track metadata around those files).

But you don’t have fast search on those files stored on object storage.

Yes, there is a cold start penalty but once the data is cached, it is equivalent to disk backed indices. There is also active work being done to improve the performance, example https://github.com/opensearch-project/OpenSearch/issues/1380...

Re: Turbopuffer: Fast search on object storage

#64

Post by nh2 »

formerly_proven wrote: post #42
nh2 wrote: post #28

> $3600.00/TB/month It doesn't have to be that way. At Hetzner I pay $200/TB/month for RAM. That's 18x cheaper. Sometimes you can reach the goal faster with less complexity by removing the part with the 20x markup.

You seem to be quoting the highest figure from the article out of context as-if that is their pricing, but the opposite is the case. > $3600.00/TB/month (incumbents) > $70.00/TB/month (turbopuffer) That's still 3x cheaper than your number and it's a SaaS API, not just a piece of rented hardware.

> as-if that is their pricing

No, that's not what I'm saying. Their "Storage Costs" table shows costs to rent storage from some provider (AWS?). It's clear that those are costs that the user has to pay for infrastructure needed for certain types of software (e.g. Turbopuffer is designed to be running on "S3 + SSD Cache", while other software may be designed to run on "RAM + 3x SSD").

I'm comparing RAM costs from that table with RAM costs in the real world.

The idea backed by that table is "RAM is so expensive, so we need to build software to run it on cheaper storage instead".

My statement is "RAM is that expensive only on that provider, there are others where it is not; on those, you may just run it in RAM and save on software complexity".

You will still need some software for your SaaS API to serve queries from RAM, but it won't need the complexity of trying to make it fast when serving from a higher-latency storage backend (S3).

Re: Turbopuffer: Fast search on object storage

#65

Post by orthecreedence »

masterj wrote: post #21
imiric wrote: post #17

Earlier quoted context omitted.

ClickHouse can also read from S3. I'm not sure how it compares to DuckDB re efficiency, but it worked fine for my simple use case.

Neither of these support indexes afaik. They are designed to do fast scans / computation.

Depends what you mean by "indexes." DuckDB can read path parameters (ex s3://my-bucket/category=beverages/month=2022-01-01/*/*.parquet) where `category` and `month` can be filtered at the query level, skipping any non-matching files. I think that qualifies as an index. Obviously, you'd have to create these up-front, or risk moving lots of data between paths.

Re: Turbopuffer: Fast search on object storage

#66

Post by bmar »

5- wrote: post #57
cmcollier wrote: post #3

Unrelated to the core topic, I really enjoy the aesthetic of their website. Another similar one is from Fixie.ai (also, interestingly, one of their customers).

indeed! what a nice, minimal page... that comes with ~1.6mb of javascript.

What's wrong with that?

Re: Turbopuffer: Fast search on object storage

#67

Post by snthpy »

softwaredoug wrote: post #2

Having worked with Simon he knows his sh*t. We talked a lot about what the ideal search stack would look when we worked together at Shopify on search (him more infra, me more ML+relevance). I discussed how I just want a thing in the cloud to provide my retrieval arms, let me express ranking in a fluent "py-data" first way, and get out of my way My ideal is that turbopuffer ultimately is like a Polars dataframe where…

Could you give an example of what you mean by _fluent "py-data" first way_ ?

You mean like a fluent API like `data.transform().filter()...` , that sort of thing?

Re: Turbopuffer: Fast search on object storage

#68

Post by fulmicoton »

bigbones wrote: post #4

Sounds like a source-unavailable version of Quickwit? https://quickwit.io/

Quickwit is targetting logs:

    - it does not do vector search. It can rank docs using BM25, but usually people just want to sort by timestamp.
    - its does not use an SSD cache. Quickwit reads directly into the object storage.
    - it is append-only (you can't modify documents)
    - it scales really well and typically shines on the 1TB .. 100PB range
    - it has a Elastic search compatible API.
Post reply on HN