Building a semantic search engine in Rust
sachaarbonel.medium.com
Building a semantic search engine in Rust
1–10 of 29 posts
Re: Building a semantic search engine in Rust
#2I remember when "semantic search" was the Next Big Thing (back when all we had were simple keyword searches).
I don't know enough about the internals of Google's search engine to know if it could be called a "semantic search engine", but not, it gets close enough to fool me.
But I feel like I'm still stuck on keyword searches for a lot of other things, like email (outlook and mutt), grepping IRC logs, searching for products in small online stores, and sometimes even things like searching for text in a long webpage.
I'm sure people have thought about these things: what technical challenges exist in improving search in these areas? is it just a matter of integrating engines like the one that's linked here? Or maybe keyword searches are often Good Enough, so no one is really clamoring for something better
Re: Building a semantic search engine in Rust
#3Thank you for posting this! I'll update http://searchthebook.com/ to use this method instead of interfacing with python like it is now. Your method might enable me to run it on a small server with reasonable response times.
Re: Building a semantic search engine in Rust
#4I remember when "semantic search" was the Next Big Thing (back when all we had were simple keyword searches). I don't know enough about the internals of Google's search engine to know if it could be called a "semantic search engine", but not, it gets close enough to fool me. But I feel like I'm still stuck on keyword searches for a lot of other things, like email (outlook and mutt), grepping IRC logs, searching for p…
A lot of times search is intentionally broken on websites to get you looking at more things.
Re: Building a semantic search engine in Rust
#5But isn’t that PyTorch with a few extra steps?
Re: Building a semantic search engine in Rust
#6………in Rust!
Re: Building a semantic search engine in Rust
#7I remember when "semantic search" was the Next Big Thing (back when all we had were simple keyword searches). I don't know enough about the internals of Google's search engine to know if it could be called a "semantic search engine", but not, it gets close enough to fool me. But I feel like I'm still stuck on keyword searches for a lot of other things, like email (outlook and mutt), grepping IRC logs, searching for p…
A lot of times search is intentionally broken on websites to get you looking at more things.
That doesn't explain why Atlassian can't build a working search function for Confluence.
I get the feeling the biggest problem with site-local search engines is a tacit requirement that the search index always must be up to date. That severely hamstrings any search engine, since there's a wealth of supplemental information to be gathered by considering the corpus as a whole that is simply not available if you support real-time updates.
Re: Building a semantic search engine in Rust
#8I remember when "semantic search" was the Next Big Thing (back when all we had were simple keyword searches). I don't know enough about the internals of Google's search engine to know if it could be called a "semantic search engine", but not, it gets close enough to fool me. But I feel like I'm still stuck on keyword searches for a lot of other things, like email (outlook and mutt), grepping IRC logs, searching for p…
I think you mean when semantic web was the next Big Thing. Better search would have been a nice side effect of a semantic web.
I was a big enthusiast of all the potential a semantic web could have brought. In my opinion due to the rise of social networks, content got heavily centralized, and it wasn't in the centralized platform's interest to annotate it, or allow others to consume it. For a short period of my time on the internet, when most in my close circle of friends, had WP blogs and blogrolls in the sidebar, you'd get at least some FOAF annotations on those links. I could complain for a whole afternoon about the clunkiness of supporting software and technologies (RDF, triple stores, ontologies, graph databases, etc), as it wasn't easy as a coder to hack on these technologies developed by consortiums.
As far as semantic search goes. I think that due to the heavy SaaS-ification of software, now there isn't even an incentive to create better search tooling. I know the landscape of search systems is huge, and while there is no way for me to assess all existing software, I've just stuck with the the tried and tested Apache Solr (or ElasticSearch) on projects I worked on. And those are not easily tweaked into semantic search engines.
My experience with Google Search for the past few years is that results based on their knowledge graph have been gamed heavily. There's a lot of junk in those results that is very much adjacent to the keywords I'm searching. You'll see the common suggestion on HN as well, when you search for a product review to also include news.ycombinator.com/reddit.com in your query based on the type of product you're looking for.
Re: Building a semantic search engine in Rust
#9I remember when "semantic search" was the Next Big Thing (back when all we had were simple keyword searches). I don't know enough about the internals of Google's search engine to know if it could be called a "semantic search engine", but not, it gets close enough to fool me. But I feel like I'm still stuck on keyword searches for a lot of other things, like email (outlook and mutt), grepping IRC logs, searching for p…
Semantic similarity more concretely means to use neural nets to embed the text, then use cosine similarity or dot product to compute the score between two entities.
embed1 = neural_net(txt1)
embed2 = neural_net(txt2)
sim_score = np.dot(embed1, embed2)
If you're making a search engine you precompute the embeds for all the items in your database. When a user performs a search you just need to embed the query and do the dot products, which are pretty fast for small indexes.
Assuming you want to index millions or billions of entities doing dot products is inefficient because it scales linearly in the size of the index. There is a trick (similar to binary search) that will find the top-k most similar results in O(log(N)) time, called approximate nearest neighbour (ANN). There are a few good libraries for that.
Re: Building a semantic search engine in Rust
#10But isn’t that PyTorch with a few extra steps?
well technically yes, but since it's in Rust once you have your embeddings, with some tweakings you could just store them and query them offline via ffi or wasm in an app for example. Also I am not a huge fan of maintaining python projects they often don't run after a while if I didn't properly maintain a virtual env etc never had this issue with Rust projects.