Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: SeekStorm – open-source sub-millisecond search in Rust

github.com

Post reply on HN
Print view View original

51–60 of 64 posts

Re: Show HN: SeekStorm – open-source sub-millisecond search in Rust

#51

Post by wolfgarbe »

emmanueloga_ wrote: post #50

The documentation seems a bit sparse. Also, I couldn't find binaries so I'm guessing building from source is required at the moment? I'm curious about the binary size of it all. Could this be compiled with WASM and run on static pages?

The Seekstorm library is 9 MB, and the Seekstorm server executable is 8 MB, depending on the features selected in cargo.

You add the library via 'cargo add seekstorm' to your project which you anyway have to compile.

As for the server, we may add binaries for the main OS in the future.

WASM and Python bindings are on our roadmap.

Re: Show HN: SeekStorm – open-source sub-millisecond search in Rust

#52

Post by philippemnoel »

anonzzzies wrote: post #46
yellow_lead wrote: post #37

Earlier quoted context omitted.

When I tried to use FTS in Postgres, I got terrible performance, but maybe I was doing something wrong. I'm using Meili now.

Same here, this would easily beat it as far as I have seen, but maybe I did something wrong.

ParadeDB (paradedb.com) is similar to this HN, but baked into Postgres to solve this very problem you are describing

Re: Show HN: SeekStorm – open-source sub-millisecond search in Rust

#53

Post by 0bit »

throwaway888abc wrote: post #17

Impressive, bookmarked, upvoted. Appreciate the demo: https://deephn.org/?q=apple+silicon

Counter-example: https://deephn.org/?q=embeddings Contrast with https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

It is fast but, nowhere close to accurate or useful for this specific example. Could not find a way to force the plural form. Neither quotes nor plus worked.

Re: Show HN: SeekStorm – open-source sub-millisecond search in Rust

#54

Post by ghita_ »

Very impressive results. I'm curious how you benchmarked against bm25 in terms of accuracy? I couldn't find metrics around that, just one search example. I think there are use cases where latency is king, but when it comes to vector search / hybrid search accuracy is probably more important.

Re: Show HN: SeekStorm – open-source sub-millisecond search in Rust

#55

Post by wolfgarbe »

ghita_ wrote: post #54

Very impressive results. I'm curious how you benchmarked against bm25 in terms of accuracy? I couldn't find metrics around that, just one search example. I think there are use cases where latency is king, but when it comes to vector search / hybrid search accuracy is probably more important.

For the latency benchmarks we used vanilla BM25 (SimilarityType::Bm25f for a single field) for comparability, so there are no differences in terms of accuracy.

For SimilarityType::Bm25fProximity which takes into account the proximity between query term matches within the document, we have so far only anecdotal evidence that it returns significantly more relevant results for many queries.

Systematic relevancy benchmarks like BeIR, MS MARCO are planned.

Re: Show HN: SeekStorm – open-source sub-millisecond search in Rust

#56

Post by J_Shelby_J »

llIIllIIllIIl wrote: post #35

How is it different from Meilisearch[1]? I’m running search for my small multi tenant SaaS and self hosted Meilisearch gives me grief like any relatively new tech, so I’m shopping for new solutions. 1: https://www.meilisearch.com/

Well, off the bat, this seems to be able to embedded directly into your rust project without the need for a standalone server.

Re: Show HN: SeekStorm – open-source sub-millisecond search in Rust

#57

Post by jszymborski »

wolfgarbe wrote: post #41
treefarmer wrote: post #24

Is there distributed server support? I see it on the list of new features with (currently PoC) next to it, but is the code for the PoC available anywhere? Also, would there be any potential issues if the index was mounted on shared storage between multiple instances?

The code for the distributed search cluster is not yet stable enough to be published, but it will be released as open-source as well. As for shared storage, do you mean something like NAS or, rather Amazon S3? Cloud-native support of object storage and separating storage and compute is on our roadmap. Challenges will be maintaining latency and the need for more sophisticated caching.

S3 support would be absolutely killer.

Re: Show HN: SeekStorm – open-source sub-millisecond search in Rust

#58

Post by d3Xt3r »

throwaway888abc wrote: post #17

Impressive, bookmarked, upvoted. Appreciate the demo: https://deephn.org/?q=apple+silicon

Seems to buggy. According to the SeekStorm github, it's supposed to support boolean operators right? But they don't seem to work.

Eg: https://deephn.org/?q=Linux+OR+KDE

Re: Show HN: SeekStorm – open-source sub-millisecond search in Rust

#60

Post by nextaccountic »

wolfgarbe wrote: post #45
infamouscow wrote: post #27

I'm not sure it's a good idea to use mmap for this. https://db.cs.cmu.edu/mmap-cidr2022/

In SeekStorm you can choose per index whether to use Mmap or let SeekStorm fully control Ram access. There is a slight performance advantage to the latter, at the cost of higher index load time of the former. https://docs.rs/seekstorm/latest/seekstorm/index/enum.Access...

Does seekstorm use io_uring? Could io_uring lower load time here?

Or at least lazy loading of index in RAM (emulating what mmap would do anyway)

Post reply on HN