Show HN: SeekStorm – open-source sub-millisecond search in Rust
1–10 of 64 posts
Re: Show HN: SeekStorm – open-source sub-millisecond search in Rust
#2Sub-millisecond latency sounds impressive, but isn't network latency going to overshadow these gains in most real-world scenarios?
Re: Show HN: SeekStorm – open-source sub-millisecond search in Rust
#3Sub-millisecond latency sounds impressive, but isn't network latency going to overshadow these gains in most real-world scenarios?
It depends on the application.
When using SeekStorm as a server, keeping the latency per query low increases the throughput and the number of parallel queries a server can handle on top of a given hardware. An efficient search server can reduce the required investments in server hardware.
In other cases, only the local search performance matters, e.g., for data mining or RAG.
Also, it's not only about averages but also about tail latencies. While network latencies dominate the average search time, that is not the case for tail latencies, which in turn heavily influence user satisfaction and revenue in online shopping.
Re: Show HN: SeekStorm – open-source sub-millisecond search in Rust
#4Sub-millisecond latency sounds impressive, but isn't network latency going to overshadow these gains in most real-world scenarios?
When search is cheap and quick, it's possible to improve search by postprocessing search results and running more queries when necessary.
I use Tantivy, and add refinements like: if the top result is objectively a low-quality one, it's usually a query with a typo finding a document with the same typo, so I run the query again with fuzzy spelling. If all the top results have the same tag (that isn't in the query), then I mix in results from another search with the most common tag excluded. If the query is a word that has multiple meanings, I can ensure that each meaning is represented in the top results.
Re: Show HN: SeekStorm – open-source sub-millisecond search in Rust
#5Demo = impressed.
How's SeekStorm's prowess in mid-cap enterprise? How hairy is the ingest pipeline for sources like: decade old sharepoint sites, PDFs with partial text layers, excel, email.msg files, etc...
Re: Show HN: SeekStorm – open-source sub-millisecond search in Rust
#6Demo = impressed. How's SeekStorm's prowess in mid-cap enterprise? How hairy is the ingest pipeline for sources like: decade old sharepoint sites, PDFs with partial text layers, excel, email.msg files, etc...
Yes, integration in complex legacy systems is always challenging.
As a small startup, we are concentrating on core search technology to make search faster and to make the most of available server infrastructure.
As SeekStorm is open-source, system integrators can take it from there.
Re: Show HN: SeekStorm – open-source sub-millisecond search in Rust
#7Demo = impressed. How's SeekStorm's prowess in mid-cap enterprise? How hairy is the ingest pipeline for sources like: decade old sharepoint sites, PDFs with partial text layers, excel, email.msg files, etc...
Same as any other full-text search solution - it's your job to integrate it.
Re: Show HN: SeekStorm – open-source sub-millisecond search in Rust
#8[flagged]
Re: Show HN: SeekStorm – open-source sub-millisecond search in Rust
#9What is the story for multi-language corpus? Do I have to do my own stop word pruning, tokenizing, lemming, etc? This is usually the case with full-text search solutions and it is a pain.
Re: Show HN: SeekStorm – open-source sub-millisecond search in Rust
#10What is the story for multi-language corpus? Do I have to do my own stop word pruning, tokenizing, lemming, etc? This is usually the case with full-text search solutions and it is a pain.
We started with making the core search technology faster. Then we added a Unicode character folding/normalization tokenizer (diacritics, accents, umlauts, bold, italic, full-width chars...). Last week we added a tokenizer that supports Chinese word segmentation. Currently, we are working on a multi-language tokenizer, that segments Chinese, Japanese an Korean without switching the tokenizer.