Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I wrote a full text search engine in Go

github.com

Post reply on HN
Print view View original

11–20 of 56 posts

Re: Show HN: I wrote a full text search engine in Go

#11

Post by novocayn »

n_u wrote: post #10

Cool project! I see you are using a positional index rather than doing bi-word matching to support positional queries. Positional indexes can be a lot larger than non-positional. What is the ratio of the size of all documents to the size of the positional inverted index?

Observation is spot on. Biword matching would definitely ease this. Stealing bi-word matching for a future iteration, tysm :D

Re: Show HN: I wrote a full text search engine in Go

#13

Post by novocayn »

eudoxus wrote: post #12

Would love to hear how this compares to another popular go based full text search engine (with a not too dissimilar name) https://github.com/blevesearch/bleve ?

Bleve is an absolute beast! built with <3 at Couchbase Fun fact: the folks who maintain it sit right across from me at work

Re: Show HN: I wrote a full text search engine in Go

#15

Post by n_u »

novocayn wrote: post #11
n_u wrote: post #10

Cool project! I see you are using a positional index rather than doing bi-word matching to support positional queries. Positional indexes can be a lot larger than non-positional. What is the ratio of the size of all documents to the size of the positional inverted index?

Observation is spot on. Biword matching would definitely ease this. Stealing bi-word matching for a future iteration, tysm :D

Well bi-word matching requires that you still have all of the documents stored to verify the full phrase occurs in the document rather than just the bi-words. So it isn't always better.

For example the phrase query "United States of America" doesn't occur in the document "The United States is named after states of the North American continent. The capital of America is Washington DC". But "United States", "states of" and "of America" all appear in it.

There's a tradeoff because we still have to fetch the full document text (or some positional structure) for the filtered-down candidate documents containing all of the bi-word pairs. So it requires a second stage of disk I/O. But as I understand most practitioners assume you can get away with less IOPS vs positional index since that info only has to fetched for a much smaller filtered-down candidate set rather than for the whole posting list.

But that's why I was curious about the storage ratio of your positional index.

Re: Show HN: I wrote a full text search engine in Go

#16

Post by llllm »

novocayn wrote: post #8
wolfgarbe wrote: post #4

Great work! Would be interesting to see how it compares to Lucene performance-wise, e.g. with a benchmark like https://github.com/quickwit-oss/search-benchmark-game

Thanks! Honestly, given it's hacked together in a weekend not sure it’d measure up to Lucene/Bleve in any serious way. I intended this to be an easy on-ramp for folks who want to get a feel for how FTS engines work under the hood :)

Not _that_ long ago Bleve was also hacked together over a few weekends.

I appreciate the technical depth of the readme, but I’m not sure it fits your easy on-ramp framing.

Keep going and keep sharing.

Post reply on HN