Viewing profile — wolfgarbe

Peter Norvig shows that an edit distance = 2 will cover 98.9% spelling errors. https://impythonist.wordpress.com/2014/03/18/peter-norvigs-2... That's the reason why the default max…

Author of SymSpell here. Congrats on the launch of Lexiathan. Unfortunately, the comparison of Lexiathan vs. Symspell on your website regarding accuracy is misleading. 1. SymSpell …

The stopword list in SeekStorm is purely optional, per default it is empty. The query "to be or not to be" that you mentioned, consisting solely of stopwords, returns complete resu…

Can the index size exceed the RAM size (e.g., via memory mapping), or are index size and document number limited by RAM size? It would be good to mention those limitations in the R…

Sure, but it says "High-performance" Full Text Search Engine. Shouldn't that claim be backed up by numbers, comparing it to the state of the art?

Great work! Would be interesting to see how it compares to Lucene performance-wise, e.g. with a benchmark like https://github.com/quickwit-oss/search-benchmark-game

The most widely used DHT is Kademlia from Petar Maymounkov and David Mazières. It is used in Ethereum, IPFS, I2P, Gnutella DHT, and many other applications. https://en.wikipedia.or…

SeekStorm does currently not use io_uring, but it is on our roadmap. Challenges are the cross-platform compatibility. Linux (io_uring) and Windows (IoRing) use different implementa…

SeekStorm comes with an http interface. The SeekStorm server features an REST API via http: https://seekstorm.apidocumentation.com It also comes with an embedded Web UI: https://gi…

>> The documentation seems a bit sparse. We just released a new OpenAPI based documentation for the SeekStorm server REST API: https://seekstorm.apidocumentation.com For the librar…

For the latency benchmarks we used vanilla BM25 (SimilarityType::Bm25f for a single field) for comparability, so there are no differences in terms of accuracy. For SimilarityType::…

The Seekstorm library is 9 MB, and the Seekstorm server executable is 8 MB, depending on the features selected in cargo. You add the library via 'cargo add seekstorm' to your proje…

In SeekStorm you can choose per index whether to use Mmap or let SeekStorm fully control Ram access. There is a slight performance advantage to the latter, at the cost of higher in…

The benchmark should be fairly fair, as it was developed by Tantivy themselves (and Jason Wolfe). So, the choice of corpus and queries was theirs. But, of course, your mileage may …

Currently, you can choose between tokenizers with or without folding. But configurability per language or full customizability of the folding logic by the user is a good idea.

The code for the distributed search cluster is not yet stable enough to be published, but it will be released as open-source as well. As for shared storage, do you mean something l…

Its not just about speed. Speed reflects efficiency. Efficiency is needed to serve more queries in parallel, to search within exponentially growing data, with less expensive hardwa…

PostgreSQL is an SQL database that also offers full-text search (FTS), with extensions like pg_search it also supports BM25 scoring which is essential for lexical search. SeekStorm…

Yes. We waited long for AOT compilation to become mature, to remove the need for the user to install the .Net framework. But two years ago when we decided to switch, we still could…

The 2-4 speed ratio was not meant to denounce C#, which is a great language I loved to program in for over two decades, coming from Delphi. Unfortunately, C# has not a complete SIM…

We started with making the core search technology faster. Then we added a Unicode character folding/normalization tokenizer (diacritics, accents, umlauts, bold, italic, full-width …

Yes, integration in complex legacy systems is always challenging. As a small startup, we are concentrating on core search technology to make search faster and to make the most of a…

It depends on the application. When using SeekStorm as a server, keeping the latency per query low increases the throughput and the number of parallel queries a server can handle o…

Series of blog posts on search and information retrieval https://medium.com/@dtunkelang What AI Engineers Should Know about Search from Doug Turnbull https://softwaredoug.com/blog/…