Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: SeekStorm – open-source sub-millisecond search in Rust

github.com

Post reply on HN
Print view View original

41–50 of 64 posts

Re: Show HN: SeekStorm – open-source sub-millisecond search in Rust

#41

Post by wolfgarbe »

treefarmer wrote: post #24

Is there distributed server support? I see it on the list of new features with (currently PoC) next to it, but is the code for the PoC available anywhere? Also, would there be any potential issues if the index was mounted on shared storage between multiple instances?

The code for the distributed search cluster is not yet stable enough to be published, but it will be released as open-source as well.

As for shared storage, do you mean something like NAS or, rather Amazon S3? Cloud-native support of object storage and separating storage and compute is on our roadmap. Challenges will be maintaining latency and the need for more sophisticated caching.

Re: Show HN: SeekStorm – open-source sub-millisecond search in Rust

#42

Post by wolfgarbe »

ronjakoi wrote: post #40
wolfgarbe wrote: post #10

Earlier quoted context omitted.

We started with making the core search technology faster. Then we added a Unicode character folding/normalization tokenizer (diacritics, accents, umlauts, bold, italic, full-width chars...). Last week we added a tokenizer that supports Chinese word segmentation. Currently, we are working on a multi-language tokenizer, that segments Chinese, Japanese an Korean without switching the tokenizer.

I hope the folding and normalization is configurable by language. I really hate it when some search decides that a and ä are the same letter. In Finnish they really aren't; "saari" is an island, "sääri" is the lower leg or shin.

Currently, you can choose between tokenizers with or without folding. But configurability per language or full customizability of the folding logic by the user is a good idea.

Re: Show HN: SeekStorm – open-source sub-millisecond search in Rust

#43

Post by jamil7 »

Thaxll wrote: post #25

It feels like everyone re-implement the same application, searching text in language x.y.z has been done a million times, search speed in not a problem so what differenciate this solution with the dozen+ mature ones. The speed looks great but isn't everything else already fast enough?

Software is currently extremely inefficient, driven by years of increasingly powerful cheap hardware. Once that starts to slow it makes sense that we start squeezing efficiency out of software again. We’ve also seen in the last 20 years the rise of languages that make writing performant, higher-level software a lot easier.

We’re also at a point where cloud compute is consuming a significant amount of energy globally.

Re: Show HN: SeekStorm – open-source sub-millisecond search in Rust

#44

Post by wolfgarbe »

littlestymaar wrote: post #39

I don't know how fair the benchmark is, but beating Tantivy by that margin is impressive to say the least. Any plan to make it run on WASM? I wanted to add this feature to Tantivy a few years ago but they weren't interested, and I had to fall back to a JavaScript search engine that was much slower.

The benchmark should be fairly fair, as it was developed by Tantivy themselves (and Jason Wolfe). So, the choice of corpus and queries was theirs. But, of course, your mileage may vary. It is always best to benchmark it on your machine with your data and your queries.

Yes, WASM and Python bindings are on our roadmap.

Re: Show HN: SeekStorm – open-source sub-millisecond search in Rust

#45

Post by wolfgarbe »

infamouscow wrote: post #27

I'm not sure it's a good idea to use mmap for this. https://db.cs.cmu.edu/mmap-cidr2022/

In SeekStorm you can choose per index whether to use Mmap or let SeekStorm fully control Ram access. There is a slight performance advantage to the latter, at the cost of higher index load time of the former. https://docs.rs/seekstorm/latest/seekstorm/index/enum.Access...

Re: Show HN: SeekStorm – open-source sub-millisecond search in Rust

#46

Post by anonzzzies »

yellow_lead wrote: post #37
wolfgarbe wrote: post #23

Earlier quoted context omitted.

PostgreSQL is an SQL database that also offers full-text search (FTS), with extensions like pg_search it also supports BM25 scoring which is essential for lexical search. SeekStorm is centered around full-text search only, it doesn't offer SQL. Performance-wise it would be indeed interesting to run a benchmark. The third-party open-source benchmark we are currently using (search_benchmark_game) does not yet support P…

When I tried to use FTS in Postgres, I got terrible performance, but maybe I was doing something wrong. I'm using Meili now.

Same here, this would easily beat it as far as I have seen, but maybe I did something wrong.

Re: Show HN: SeekStorm – open-source sub-millisecond search in Rust

#47

Post by kreyenborgi »

jazzyjackson wrote: post #13
remram wrote: post #9

What is the story for multi-language corpus? Do I have to do my own stop word pruning, tokenizing, lemming, etc? This is usually the case with full-text search solutions and it is a pain.

Re: stemming and lemming, I just want to plug the most impressive NLP stack I ever used, "chat script", really it's for building dialog trees where it walks down a branch of conversation using effectively switch statements but with really rich conceptual pattern matching and capturing - so somewhere in the middle of the stack it has excellent abstracting from word input to general concept (in WordNet), performing all…

Their company home page, http://brilligunderstanding.com/ wow..

Re: Show HN: SeekStorm – open-source sub-millisecond search in Rust

#48

Post by fulmicoton »

littlestymaar wrote: post #39

I don't know how fair the benchmark is, but beating Tantivy by that margin is impressive to say the least. Any plan to make it run on WASM? I wanted to add this feature to Tantivy a few years ago but they weren't interested, and I had to fall back to a JavaScript search engine that was much slower.

Developer of tantivy chiming in! (I hope that's ok) Database performance is a space where there are a lot of lies and bullshit, so you are 100% right to be suspicious.

I don't know SeekStorm's team and I did not dig much into the details, but my impression so far is that their benchmark's results are fair. At least I see no reason not to trust them.

Re: Show HN: SeekStorm – open-source sub-millisecond search in Rust

#49

Post by PSeitz »

fulmicoton wrote: post #48
littlestymaar wrote: post #39

I don't know how fair the benchmark is, but beating Tantivy by that margin is impressive to say the least. Any plan to make it run on WASM? I wanted to add this feature to Tantivy a few years ago but they weren't interested, and I had to fall back to a JavaScript search engine that was much slower.

Developer of tantivy chiming in! (I hope that's ok) Database performance is a space where there are a lot of lies and bullshit, so you are 100% right to be suspicious. I don't know SeekStorm's team and I did not dig much into the details, but my impression so far is that their benchmark's results are fair. At least I see no reason not to trust them.

Also we are working on some performance improvements based on the benchmark comparison, as they highlighted some areas we can improve in tantivy.
Post reply on HN