Building a Simple Search Engine That Works
karboosx.net
Building a Simple Search Engine That Works
1–10 of 82 posts
Re: Building a Simple Search Engine That Works
#2I completely agree with the insight that full text search has been complexified. People seem to want to jump straight to clustering or other enterprise level things.
I also appreciate the moxie of getting in there and building it yourself.
Myself, I reach for Lucene. Then you don’t need to build all this yourself if you don’t want. It lives in a dir on disk. True, it’s a separate database, but one optimized for this problem.
Re: Building a Simple Search Engine That Works
#3Great read. It makes you wonder how heavily optimised the tokenizers used by popular search enginea truly are.
Re: Building a Simple Search Engine That Works
#4I completely agree with the insight that full text search has been complexified. People seem to want to jump straight to clustering or other enterprise level things. I also appreciate the moxie of getting in there and building it yourself. Myself, I reach for Lucene. Then you don’t need to build all this yourself if you don’t want. It lives in a dir on disk. True, it’s a separate database, but one optimized for this…
This was the solution I was thinking about, but I thought, well that's the way someone would have done it 20 years ago
Re: Building a Simple Search Engine That Works
#5About a decade ago, I was working with a guy who was getting a PhD in search engine design, which I knew/know nothing about.
It was actually a lot of fun to chat with him, because he was so enthusiastic about how searching works and how it can integrate with databases, and he was eager to explain this all to anyone who would listen. I learned a fair amount from him, though admittedly I still don't know much about the intricacies of how search engines work.
Some day, I am going to really go through the guts of Apache Solr and Lucene to understand the internals (like I did for Kafka a few years ago), and maybe I'll finally be competent with it.
Re: Building a Simple Search Engine That Works
#6Reminds me of reading Programming Collective Intelligence by Toby Segaran, which inspired me with a range of things, like building search, recommenders, classifiers etc.
Re: Building a Simple Search Engine That Works
#7Reminds me of reading Programming Collective Intelligence by Toby Segaran, which inspired me with a range of things, like building search, recommenders, classifiers etc.
That was a great book, I wonder what the 2025 equivalent of it is...
Re: Building a Simple Search Engine That Works
#8Good. Now please someone replace Google's search engine.
I am always annoyed using it, how bad it is these days. Then I try the alternatives such as Duck Duck Go and they manage to be even worse.
Qwant is semi-ok but it also omits tons of things that Google Search finds (and also is slower, for some weird reason).
Google's UI nerf is also annoying - so much useless stuff. In the past I could disable that via ublock origin but Google killed that for chrome.
We need to do something against this Evil that Google brought into this world.
Re: Building a Simple Search Engine That Works
#9Incredible article. Does what it claims in the title, is written well and follows a linear chain of reasoning with a minumum of surprises.
Re: Building a Simple Search Engine That Works
#10I completely agree with the insight that full text search has been complexified. People seem to want to jump straight to clustering or other enterprise level things. I also appreciate the moxie of getting in there and building it yourself. Myself, I reach for Lucene. Then you don’t need to build all this yourself if you don’t want. It lives in a dir on disk. True, it’s a separate database, but one optimized for this…
This was the solution I was thinking about, but I thought, well that's the way someone would have done it 20 years ago
Alright but why do we not have more search engines that are actually good?
I'd love to cut myself off from Google, including Google Search, but any alternatives manage to be even worse. Consistently so. It's as if Google won the war by being just permanently slightly better - while everyone is actually really crap. That wasn't the case, say, 10 years ago or so.