Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Building a web search engine from scratch with 3B neural embeddings

blog.wilsonl.in

Post reply on HN
Print view View original

101–110 of 124 posts

Re: Show HN: Building a web search engine from scratch with 3B neural embeddings

#101

Post by dieortin »

kiririn wrote: post #92
noosphr wrote: post #47

Earlier quoted context omitted.

It's been clear to anyone familiar with encoder only LLMs that Google is effectively dead. The only reason why it still lives is that it takes a while to crawl the whole web and keep the index up to date. If someone like common crawl, or even a paid service, solves the crawling of the web in real time then the moat Google had for the last 25 years is dead and search is commoditized.

You can see their panic - in my country they are running TV ads for Google search, showing it answering LLM-prompt-like queries. They are desperately trying to win back that mind share, and if they lose traditional keyword search too they’re cooked

Which country is that?

Re: Show HN: Building a web search engine from scratch with 3B neural embeddings

#102

Post by bjornsing »

jll29 wrote: post #87

Not sure where you are based, but if you were in the EU and had no commercial intentions, you might want to consider adding the crawls from OpenWebSearch.eu, an EU-funded research project to provide an open crawl of a substantial part of the Web (they also collaborate with Common Crawl), its plain text and an index: https://openwebsearch.eu/ It would be fantastic if someone could provide a not-for-profit decent quali…

Why the hell do all those ”renowned institutions” need this (admittedly brilliant) guy to turn their crawl into a usable search engine? What’s wrong with this continent…?

Re: Show HN: Building a web search engine from scratch with 3B neural embeddings

#103

Post by jacobr1 »

ricardobeat wrote: post #46
poly2it wrote: post #40

Very cool project! Just out of interest, I sent a query I've had difficulties getting good results for with major engines: "what are some good options for high-resolution ultrawide monitors?". The response in this engine for this query at this point seems to have the same fallacy as I've seen in other engines. Meta-pages "specialising" in broad rankings are preferred above specialist data about the specific sought-af…

You could argue that it is not really a search query. There is not a particular page that answers the question “correctly”, it requires collating multiple sources and reasoning . That is not a search problem.

And yet ... that is exactly the kind of problem average people want to have solved for them by search engines, and google kept trying to solve. It is probably one reason why ai-chat with websearch is going to beat just search.

Re: Show HN: Building a web search engine from scratch with 3B neural embeddings

#104

Post by dymk »

dweinus wrote: post #90
cedws wrote: post #71

Earlier quoted context omitted.

Yes, it would be disastrous for OpenAI if it got out they are training on B2B data despite saying they don’t.

We're both talking about the company whose entire business model is built on top of large scale copyright infringement, right?

Not the same when the people you infringe on can sue you into the dirt

Re: Show HN: Building a web search engine from scratch with 3B neural embeddings

#105

Post by j33zusjuice »

mattigames wrote: post #74
cedws wrote: post #71

Earlier quoted context omitted.

Yes, it would be disastrous for OpenAI if it got out they are training on B2B data despite saying they don’t.

Yeah, so many companies have been completely ruined after similar PR disasters /s

Their terms of service say they won’t use the data for training, so it wouldn’t just be a PR disaster; it’d be a breach of contract. They’d be sued into oblivion.

Re: Show HN: Building a web search engine from scratch with 3B neural embeddings

#106

Post by nickpsecurity »

ccgreg wrote: post #66
noosphr wrote: post #47

Earlier quoted context omitted.

It's been clear to anyone familiar with encoder only LLMs that Google is effectively dead. The only reason why it still lives is that it takes a while to crawl the whole web and keep the index up to date. If someone like common crawl, or even a paid service, solves the crawling of the web in real time then the moat Google had for the last 25 years is dead and search is commoditized.

The team that runs the Common Crawl Foundation is well aware of how to crawl and index the web in real time. It's expensive, and it's not our mission. There are multiple companies that are using our crawl data and our web graph metadata to build up-to-date indexes of the web.

Your terms and conditions include a lot of restrictions with some ambiguous in how they can be interpreted.

Would Common Crawl do a "for all purposes and no restrictions" license if it is for AI training, comouter analyses, etc? Especially given the bad actors are ignoring copyrights and terms while such restrictions only affect moral, law-abiding people?

Also, even simpler, would Common Crawl release under a permissive license a list of URL's that others could scrape themselves? Maybe with metadata per URL from your crawls, such as which use Cloudflare or other limiters. Being able to rescrape the CC index independently would be very helpful under some legal theories about AI training. Independent, search operators benefit, too.

Re: Show HN: Building a web search engine from scratch with 3B neural embeddings

#107

Post by ccgreg »

nickpsecurity wrote: post #106
ccgreg wrote: post #66

Earlier quoted context omitted.

The team that runs the Common Crawl Foundation is well aware of how to crawl and index the web in real time. It's expensive, and it's not our mission. There are multiple companies that are using our crawl data and our web graph metadata to build up-to-date indexes of the web.

Your terms and conditions include a lot of restrictions with some ambiguous in how they can be interpreted. Would Common Crawl do a "for all purposes and no restrictions" license if it is for AI training, comouter analyses, etc? Especially given the bad actors are ignoring copyrights and terms while such restrictions only affect moral, law-abiding people? Also, even simpler, would Common Crawl release under a permiss…

Common Crawl doesn't own the content in its crawl, so no, our terms of use do not grant anyone permission to ignore the actual content owner's license.

We carefully preserve robots.txt permissions in robots.txt, in http headers, and in html meta tags.

We do publish 2 different url indexes, if you wanted to recrawl for some reason.

Re: Show HN: Building a web search engine from scratch with 3B neural embeddings

#110

Post by lelanthran »

supriyo-biswas wrote: post #96
lelanthran wrote: post #70

Earlier quoted context omitted.

> One effective old technique for ranking is to capture the search-to-click relationship by real users. It's basically the training data by human mapping the search terms they entered to the links they clicked. With just a few of clicks, the ranking relevance goes way up. That's not very effective. Ever heard of clickbait? Like I've said uncountable times before, the only effective technique to clean out the search r…

This is often touted as the solution to remove SEO garbage, except that you'd also get rid of the news websites along with it which are fairly reliant on advertising.

> This is often touted as the solution to remove SEO garbage, except that you'd also get rid of the news websites along with it which are fairly reliant on advertising.

That's fine; those who want to search for news articles can use any number of existing search engines that don't penalise ads.

Post reply on HN