Show HN: SeekStorm – open-source sub-millisecond search in Rust • Hacker News Board

Re: Show HN: SeekStorm – open-source sub-millisecond search in Rust

#61

Post by wolfgarbe » Mon, Dec 09, 2024, 6:47 PM UTC

emmanueloga_ wrote: post #50

The documentation seems a bit sparse. Also, I couldn't find binaries so I'm guessing building from source is required at the moment? I'm curious about the binary size of it all. Could this be compiled with WASM and run on static pages?

>> The documentation seems a bit sparse.

We just released a new OpenAPI based documentation for the SeekStorm server REST API: https://seekstorm.apidocumentation.com

For the library we have the standard rust doc: https://docs.rs/seekstorm/latest/seekstorm/

Re: Show HN: SeekStorm – open-source sub-millisecond search in Rust

#62

Post by wolfgarbe » Mon, Dec 09, 2024, 6:56 PM UTC

bosky101 wrote: post #59

Why isn't there a http interface? That would increase adoption by far

SeekStorm comes with an http interface.

The SeekStorm server features an REST API via http: https://seekstorm.apidocumentation.com

It also comes with an embedded Web UI: https://github.com/SeekStorm/SeekStorm?tab=readme-ov-file#bu...

Or did you mean a Web based interface to create and manage indices, define index schemas, add documents etc?

Re: Show HN: SeekStorm – open-source sub-millisecond search in Rust

#63

Post by ghita_ » Wed, Dec 11, 2024, 12:46 AM UTC

wolfgarbe wrote: post #55

ghita_ wrote: post #54

Very impressive results. I'm curious how you benchmarked against bm25 in terms of accuracy? I couldn't find metrics around that, just one search example. I think there are use cases where latency is king, but when it comes to vector search / hybrid search accuracy is probably more important.

For the latency benchmarks we used vanilla BM25 (SimilarityType::Bm25f for a single field) for comparability, so there are no differences in terms of accuracy. For SimilarityType::Bm25fProximity which takes into account the proximity between query term matches within the document, we have so far only anecdotal evidence that it returns significantly more relevant results for many queries. Systematic relevancy benchmar…

got it - i think the anecdotal evidence is what intrigued me a little bit looking forward to seeing the systematic relevancy benchmarks

Re: Show HN: SeekStorm – open-source sub-millisecond search in Rust

#64

Post by wolfgarbe » Sat, Dec 14, 2024, 2:12 PM UTC

nextaccountic wrote: post #60

wolfgarbe wrote: post #45

Earlier quoted context omitted.

In SeekStorm you can choose per index whether to use Mmap or let SeekStorm fully control Ram access. There is a slight performance advantage to the latter, at the cost of higher index load time of the former. https://docs.rs/seekstorm/latest/seekstorm/index/enum.Access...

Does seekstorm use io_uring? Could io_uring lower load time here? Or at least lazy loading of index in RAM (emulating what mmap would do anyway)

SeekStorm does currently not use io_uring, but it is on our roadmap. Challenges are the cross-platform compatibility. Linux (io_uring) and Windows (IoRing) use different implementations, and other OS don't support it. There is no abstraction layer over those implementations in Rust, so we are on our own.

It would increase concurrent read and write speed (index loading, searching) by removing the need to lock seek and read/write.

But I would expect that the mmap implementations do already use io_uring / IoRing.

Yes, lazy loading would be possible, but pure RAM access does not offer enough benefits to justify the effort to replicate much of the memory mapping.