Show HN: BM25opt – 30-40 x faster BM25 search algorithms (FOSS) • Hacker News Board

Show HN: BM25opt – 30-40 x faster BM25 search algorithms (FOSS)

#1

Post by jankovicsandras » Thu, Oct 31, 2024, 8:35 AM UTC

BM25opt is a score-compatible optimized rewrite of the popular https://github.com/dorianbrown/rank_bm25 , which is used by e.g. LangChain and Llamaindex. It's much faster and tries to fix some issues as well.

https://en.wikipedia.org/wiki/Okapi_BM25

Show HN: BM25opt – 30-40 x faster BM25 search algorithms (FOSS)
github.com

Re: Show HN: BM25opt – 30-40 x faster BM25 search algorithms (FOSS)

#2

Post by compressedgas » Thu, Oct 31, 2024, 8:54 AM UTC

I expected them to be API compatible. Not that it matters to me but I had looked to see.

Re: Show HN: BM25opt – 30-40 x faster BM25 search algorithms (FOSS)

#3

Post by jankovicsandras » Thu, Oct 31, 2024, 10:00 AM UTC

compressedgas wrote: post #2

I expected them to be API compatible. Not that it matters to me but I had looked to see.

This is a good point and was a difficult design decision. The reasons for changing the API are:

- easier to use with untokenized corpus and questions

- to fix issues with the tokenizing ( e.g. https://github.com/dorianbrown/rank_bm25/issues/38 ); also rank_bm25 provides no default tokenizer, a naive split-on-whitespace is a wrong choice

- considerably simplify the code (way less SLOC)

- point out the similarities of the algorithms for educational purpuses / further development

In practice, the differences are minimal ( see Example 3: comparison with rank_bm25 ).