Building a Search Engine from Scratch
1–10 of 151 posts
Re: Building a Search Engine from Scratch
#2Interesting approach, certainly very different than what I'm used to as someone working on search.
Will probably keep an eye on this blog.
Re: Building a Search Engine from Scratch
#3Definitely interesting.
But...my searching has gotten to the point where well over half the time I am no longer looking at Google's conventional search results.
Re: Building a Search Engine from Scratch
#4Also having built a search engine from scratch, we use a similar method: https://insideropinion.com/
In our case the "queries" are also the index creation components. Every time someone discusses something, we are indexing it, so you can search media, documents, people from context. We hint at how this works here: https://austingwalters.com/fast-full-text-search-in-postgres...
The downside of our approach is it needs lots of conversation data. From their TLDR version:
"""
- Our model of a web page is based on queries only. These queries could either be observed in the query logs or could be synthetic, i.e. we generate them. In other words, during the recall phase, we do not try to match query words directly with the content of the page. This is a crucial differentiating factor – it is the reason we are able to build a search engine with dramatically less resources in comparison to our competitors.
- Given a query, we first look for similar queries using a multitude of keyword and word vector based matching techniques.
- We pick the most similar queries and fetch the pages associated with them.
- At this point, we start considering the content of the page. We utilize it for feature extraction during ranking, filtering and dynamic snippet generation.
"""
It appears 0x65 has similarly figured this out, the name of the game is forming proper search queries. In their case, their results would be good as soon as they start indexing and create synthetic queries. IMO might be better for documents and what not.
Either way, interesting to compare notes! Kudos to the work.
Re: Building a Search Engine from Scratch
#5Strangely, cliqz.com was blocked by my pihole.
Re: Building a Search Engine from Scratch
#6[dead]
Re: Building a Search Engine from Scratch
#7Can someone comment on how to use knowledge graphs for search? I have seen some applications in NLP but I am curious how it can tie in with traditional search.
Re: Building a Search Engine from Scratch
#8I don’t see how this will ultimately be successful.
They’re basically reverse engineering Google by looking at user logs.
Google will always have a leg up here because they have all the Google data.
And even if it does work for a while, there still needs to be the original signal to copy. Someone will have to crawl the web and index content.
I’m super eager to find new approaches to search, but another Google clone is not that.
Re: Building a Search Engine from Scratch
#9Strangely, cliqz.com was blocked by my pihole.
Hi,
Thank you for bringing this up. Although, this is not relevant in the context of the blog post, we are on that list by mistake. We do NOT collect any personal data in our browser: (more details e.g., here: https://0x65.dev/blog/2019-12-02/is-data-collection-evil.htm... and https://0x65.dev/blog/2019-12-03/human-web-collecting-data-i...) and we go a long way to make sure not even implicit indentifiers go through. We believe we ended up on that list for a bad Firefox experiment and we will reach out to the maintainers, make our case.
Disclaimer: I work for Cliqz.
Re: Building a Search Engine from Scratch
#10google's suggested autocorrect is one it's most impressive features; idk I'd say the relevance of the search results almost comes in a near second to that.
so make a competitive "suggested autocorrect" solution and then I think you'd have a stew going.