Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Building a web search engine from scratch with 3B neural embeddings

blog.wilsonl.in

Post reply on HN
Print view View original

31–40 of 124 posts

Re: Show HN: Building a web search engine from scratch with 3B neural embeddings

#31

Post by cosmic_cheese »

mike_hearn wrote: post #23
giancarlostoro wrote: post #3

This then begs the question for me, without an LLM what is the approach to build a search engine? Google search used to be razor sharp, then it degraded in the late 2000s and early 2010s and now its meh. They filter out so much content for a billion different reasons and the results are just not what they used to be. I've found better results from some LLMs like Grok (surprisingly) but I can't seem to understand why…

The internet itself has changed over time, and a lot of content has just disappeared. It shouldn't appear in search because it's just not there anymore, it'd be a 404.

A search engine that kept dead entries but maybe put them in an “missing” tab or something would’ve been monstrously useful for me in so many situations. There’s been numerous times I’ve remembered looking at something N years ago only for all but the faintest traces of it to have disappeared from the internet. With a “missing” tab I’d at least have former URLs, page titles, etc to work with (archive.org, etc).

Re: Show HN: Building a web search engine from scratch with 3B neural embeddings

#32

Post by mwcz »

ASalazarMX wrote: post #18
Imustaskforhelp wrote: post #17

Earlier quoted context omitted.

I wish all the best wishes to fediverse too. I'd like to take this one step too that communities have gone a similar transition too from forums to mostly now discord and I wish them to move to something like matrix which is federated (yes I know it has issues, but trust me sacrifices must be made) What are your thoughts on things like bluesky/nostr and (matrix) too. Bluesky does seem centralized in its current stage…

I think BlueSky still needs to prove itself. It is what Twitter/X was a decade ago, before the enshittification, and I enjoy the content a lot, with my reservations. The weakness of Mastodon (and the Fediverse IMO), is that you can join one of many instances, and it becomes easier to form an echo chamber. Your feed will the the Fediverse hose (lots of irrelevant content), your local instance (an echo chamber), or you…

There was a Neal Stephenson novel where curated feeds had become a big business because it was the only tolerable way to browse the Internet. Lately I've been thinking that's more likely to happen.

Re: Show HN: Building a web search engine from scratch with 3B neural embeddings

#35

Post by Flux159 »

bstsb wrote: post #34
Flux159 wrote: post #33

Getting a CORS error from the API - is the demo at https://search.wilsonl.in/ working for anyone else?

cors error is due to the actual request failing (502 Bad Gateway). hug of death?

Yeah just saw the 502 - probably hug of death.

Re: Show HN: Building a web search engine from scratch with 3B neural embeddings

#36

Post by mmargenot »

I love this and think that your write-up is fantastic, thank you for sharing your work in such detail.

What are you thinking in terms of improving [and using] the knowledge graph beyond the knowledge panel on the side? If I'm reading this correctly, it seems like you only have knowledge panel results for those top results that exist in Wikipedia, is that correct?

Re: Show HN: Building a web search engine from scratch with 3B neural embeddings

#38

Post by throwawaylaptop »

lysecret wrote: post #24

Just wow. My greatest respect! Also an incredible write up. I like the take-away that an essential ingredient to a search engine is curated and well filtered data (garbage in garbage out) I feel like this has been a big learning of the LLM training too, rather work with less much higher quality data. I'm curious how a search engine would perform where all content has been judged by an LLM.

I'm currently trying to get a friends small business website to rank. I have a decent understanding of SEO, doing more technically correct things and did a decent amount of hand written content specific to local areas and services provided.

Two months in, bing still hasn't crawled the fav icon. Google finally did after a month. I'm still getting outranked by tangentially related services, garbage national lead collection sites, yelp top 10 blog spam, and even exact service providers from 300 miles away that definitely don't serve the area.

Something is definitely wrong with pagerank and crawling in general.

Re: Show HN: Building a web search engine from scratch with 3B neural embeddings

#39

Post by mv4 »

throwawaylaptop wrote: post #38
lysecret wrote: post #24

Just wow. My greatest respect! Also an incredible write up. I like the take-away that an essential ingredient to a search engine is curated and well filtered data (garbage in garbage out) I feel like this has been a big learning of the LLM training too, rather work with less much higher quality data. I'm curious how a search engine would perform where all content has been judged by an LLM.

I'm currently trying to get a friends small business website to rank. I have a decent understanding of SEO, doing more technically correct things and did a decent amount of hand written content specific to local areas and services provided. Two months in, bing still hasn't crawled the fav icon. Google finally did after a month. I'm still getting outranked by tangentially related services, garbage national lead collec…

Sadly, that ship has sailed. The web is dead. SEO should be called SEM (Search Engine Manipulation).

Re: Show HN: Building a web search engine from scratch with 3B neural embeddings

#40

Post by poly2it »

Very cool project!

Just out of interest, I sent a query I've had difficulties getting good results for with major engines: "what are some good options for high-resolution ultrawide monitors?".

The response in this engine for this query at this point seems to have the same fallacy as I've seen in other engines. Meta-pages "specialising" in broad rankings are preferred above specialist data about the specific sought-after item. It seems that the desire for a ranking weighs the most.

If I were to manually try to answer this query, I would start by looking at hardware forums and geeky blogs, pick N candidates, then try to find the specifications and quirks for all products.

Of course, it is difficult to generically answer if a given website has performed this analysis. It can be favourable to rank sites citing specific data higher in these circumstances.

As a user, I would prefer to be presented with the initial sources used for assembling this analysis. Of course, this doesn't happen because engines don't perform this kind of bottom-to-top evaluation.

Post reply on HN