That stack element is amazing
I wish more people showed their whole exploded stack like that and in an elegant way
Really well done writeup!
Show HN: Building a web search engine from scratch with 3B neural embeddings
21–30 of 124 posts
Re: Show HN: Building a web search engine from scratch with 3B neural embeddings
#22This then begs the question for me, without an LLM what is the approach to build a search engine? Google search used to be razor sharp, then it degraded in the late 2000s and early 2010s and now its meh. They filter out so much content for a billion different reasons and the results are just not what they used to be. I've found better results from some LLMs like Grok (surprisingly) but I can't seem to understand why…
My understanding was that every few months Google was forced to adjust their algorithms because the search results would get flooded by people using black hat SEO techniques. At least that's the excuse I heard for why it got so much worse over time. Not sure if that's related to it ignoring quotes and operators though. I'd imagine that to be a cost saving measure (and very rarely used, considering it keeps accusing m…
This has always been the explanation, but I've always wondered if it wasn't so much battling SEO as balancing the appearance of battling SEO while not killing some factor related to their revenue.
Re: Show HN: Building a web search engine from scratch with 3B neural embeddings
#23This then begs the question for me, without an LLM what is the approach to build a search engine? Google search used to be razor sharp, then it degraded in the late 2000s and early 2010s and now its meh. They filter out so much content for a billion different reasons and the results are just not what they used to be. I've found better results from some LLMs like Grok (surprisingly) but I can't seem to understand why…
The internet itself has changed over time, and a lot of content has just disappeared. It shouldn't appear in search because it's just not there anymore, it'd be a 404.
Re: Show HN: Building a web search engine from scratch with 3B neural embeddings
#24Just wow. My greatest respect! Also an incredible write up. I like the take-away that an essential ingredient to a search engine is curated and well filtered data (garbage in garbage out) I feel like this has been a big learning of the LLM training too, rather work with less much higher quality data. I'm curious how a search engine would perform where all content has been judged by an LLM.
Re: Show HN: Building a web search engine from scratch with 3B neural embeddings
#25Earlier quoted context omitted.
I wish all the best wishes to fediverse too. I'd like to take this one step too that communities have gone a similar transition too from forums to mostly now discord and I wish them to move to something like matrix which is federated (yes I know it has issues, but trust me sacrifices must be made) What are your thoughts on things like bluesky/nostr and (matrix) too. Bluesky does seem centralized in its current stage…
I think BlueSky still needs to prove itself. It is what Twitter/X was a decade ago, before the enshittification, and I enjoy the content a lot, with my reservations. The weakness of Mastodon (and the Fediverse IMO), is that you can join one of many instances, and it becomes easier to form an echo chamber. Your feed will the the Fediverse hose (lots of irrelevant content), your local instance (an echo chamber), or you…
I mean both bluesky and fediverse are just decentralized technologies, so lets say that you are worried about bluesky "enshittening"
I doubt it to happen because of its decentralized-enough nature.
I also agree with the subscriptions curation part the last time I checked, but I didn't use mastodon as often as I used lemmy and it was a less of an issue on lemmy.
Still, I feel like bluesky as an technology is goated and doesn't feel like it can be enshittened.
Nostr on the other hand does seem to me as an echo chamber of crypto bros but honestly, that's the most decentralization as you can ever get. Shame that we are going to get mostly nothing meaningful out of it imo. Which in that case bluesky seems to me as good enough but things like search etc. / the current bluesky is definitely centralized but honestly the same problems kept coming up on fediverse too, lemmy.world was getting too bloated with too many members and even mastodon had only one really famous home server afaik iirc mastodon.social right?
Also I may be wrong, I usually am but iirc mastodon only allows you to comment/ interact with posts on your own server like, I wanted to comment on mastodon.social from some other server but I don't remember being able to do so, maybe skill issue from my side.
Re: Show HN: Building a web search engine from scratch with 3B neural embeddings
#26Such a big inspiration! One of the few times where I genuinely read and liked the work - didn't even notice how the time flew by.
Feels like it's more and more about consuming data & outputting the desired result.
Re: Show HN: Building a web search engine from scratch with 3B neural embeddings
#27This then begs the question for me, without an LLM what is the approach to build a search engine? Google search used to be razor sharp, then it degraded in the late 2000s and early 2010s and now its meh. They filter out so much content for a billion different reasons and the results are just not what they used to be. I've found better results from some LLMs like Grok (surprisingly) but I can't seem to understand why…
My understanding was that every few months Google was forced to adjust their algorithms because the search results would get flooded by people using black hat SEO techniques. At least that's the excuse I heard for why it got so much worse over time. Not sure if that's related to it ignoring quotes and operators though. I'd imagine that to be a cost saving measure (and very rarely used, considering it keeps accusing m…
Google certainly had to update their algorithms to cope with SEO, but that's not why their results have become so poor in the last five years or so. They made a conscious decision to prioritize profit over search quality. This came out in internal emails that were published as part of discovery for one of the antitrust suits.
To reiterate: Google search results are shit because shit ad-laden results make them more money in the short term.
That's it. And it's sad that so many people continue to give them the benefit of the doubt when there is no doubt.
Re: Show HN: Building a web search engine from scratch with 3B neural embeddings
#28Incredibly cool. What a write-up. What an engineer.
Re: Show HN: Building a web search engine from scratch with 3B neural embeddings
#29Wow, looks like a tremendous commitment and depth of knowledge went into this one-man project. I couldn't even read the whole write up, I had to skim part of it. I'm super impressed.
Re: Show HN: Building a web search engine from scratch with 3B neural embeddings
#30I couldn't get the search working (there was some cors error) . But what a feat and writeup. Wonder Stuck!