Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Two upstart search engines are teaming up to take on Google

wired.com

Post reply on HN
Print view View original

51–60 of 299 posts

Re: Two upstart search engines are teaming up to take on Google

#51

Post by jasode »

arnaudsm wrote: post #4

Every new search engine I've seen was a Bing wrapper with sometimes light reranking. I understand that competing with Google was borderline impossible a decade ago. But in 2024, we have cheap compute, great OSS distributed DBs, powerful new vector search tech. Amateur search engines like Marginalia even run on consumer hardware. CommonCrawl text-only is ~100TB, and can fit on my home server. Why is no company buildin…

>competing with Google was borderline impossible a decade ago. But in 2024, we have cheap compute, great OSS distributed DBs, powerful new vector search tech. [...] CommonCrawl text-only is ~100TB,

Those example 3 bullet points of today's improved 2024 computing power you list isn't even enough to process Google's scale 14 years ago in 2010 when the search index was 100+ petabytes: https://googleblog.blogspot.com/2010/06/our-new-search-index...

A modest homegrown tech stack of 2024 can maybe compete with a smaller Google circa ~1998 but that thought experiment is handicapping Google's current state-of-the-art. Instead, we have to compare OSS-today vs Google-today. There's still a big delta gap between the 2024 OSS tech stack and Google's internal 2024 tech stack.

E.g. for all the billions Microsoft spent on Bing, there are still some queries that I noticed Google was better at. Google found more pages of obscure people I was researching (obituaries, etc). But Bing had the edge when I was looking up various court cases with docket #s. The internet is now so big that even billion dollar search engines can't get to all of it. Each has blindspots. I have to use both search engines every single day.

Re: Two upstart search engines are teaming up to take on Google

#52

Post by [deleted] »

marginalia_nu wrote: post #46
eitland wrote: post #41

Earlier quoted context omitted.

The Swede behind search.marginalia.nu has had a working search engine running at a single desktop class computer in a living room, all programmed and maintained on his spare time, that was so good that in its niches (history, programming, open source comes to mind) it would often outshine Google. Back before I found Kagi I used to use it everytime Google failed me. So, yes, given he is the only one I know who manages…

I think the problem with search is that while it's relatively doable to build something that is competitive in one or a few niches, Google's real sticking power is how broad their offering is. Google search has seamless integration with maps, with commercial directories, with translation, with their browser, with youtube, etc. Even though there's more than a few queries they leave something to desire, the breadth of…

[deleted]

Re: Two upstart search engines are teaming up to take on Google

#53

Post by bananapub »

lexlambda wrote: post #42

> “We could de-rank results from unethical or unsustainable companies and rank good companies higher,” Kroll says of the eco-minded Ecosia. Understandable knowing Ecosias goals, but I find it rather concerning their vision of a better search involves deciding what is good and bad. Ranking by quality (against spam & SEO sites) is fine, but it should be applied equally to all Websites, and not target specific companies…

> I find it rather concerning their vision of a better search involves deciding what is good and bad

the entire purpose of a search engine is to do this, you've been grossly confused about the entire space if you think this isn't exactly what everyone is trying to do.

Re: Two upstart search engines are teaming up to take on Google

#54

Post by ColinHayhurst »

arnaudsm wrote: post #4

Every new search engine I've seen was a Bing wrapper with sometimes light reranking. I understand that competing with Google was borderline impossible a decade ago. But in 2024, we have cheap compute, great OSS distributed DBs, powerful new vector search tech. Amateur search engines like Marginalia even run on consumer hardware. CommonCrawl text-only is ~100TB, and can fit on my home server. Why is no company buildin…

We are @mojeek

Re: Two upstart search engines are teaming up to take on Google

#55

Post by ColinHayhurst »

arnaudsm wrote: post #4

Every new search engine I've seen was a Bing wrapper with sometimes light reranking. I understand that competing with Google was borderline impossible a decade ago. But in 2024, we have cheap compute, great OSS distributed DBs, powerful new vector search tech. Amateur search engines like Marginalia even run on consumer hardware. CommonCrawl text-only is ~100TB, and can fit on my home server. Why is no company buildin…

Mostly because Google bought, developed, acquired or effectively control all the major distribution points with default placement deals: eg Apple, Samsung, Chrome, Android, Firefox. In time remedies are coming though, in the antitrust case lost versus DoJ.

Another major factor is that building a search index and algorithms that searches across billions of pages with good enough latency is very hard. Easy enough for 10s of millions scale search but a different challenge for billions.

Some claim(ed) click-query data is needed at scale, and are hoping for that remedy. Our take is what is the point of replicating Google. Anyway, will this data be free or low cost? You know the answer.

Cloud infrastructure is very expensive. We save massively on costs by building our own servers, but that means capital outlay.

Re: Two upstart search engines are teaming up to take on Google

#56

Post by stephantul »

marginalia_nu wrote: post #38
stephantul wrote: post #6

Earlier quoted context omitted.

Because it is super expensive and difficult to keep an index up to date. People expect to be able to get current events, and expect search results to be updated in minutes/seconds.

No search engine is refreshing every website every minute. Most websites don't update frequently, and if you poll them more than once every month, your crawler will get blocked incredibly fast. The problem of being able to provide fresh results is best solved by having different tiers of indices, one for frequently updating content, and one for slowly updating content with a weekly or monthly cadence. You can get a l…

I meant this in response to the parent that Common Crawl only updates every month, which seemed to imply that this was sufficient.

This is too slow for a lot of the purposes people tend to use search engines for. I agree that you don't need to crawl everything every minute. My previous employer also crawled a large portion of the internet every month, but most of it didn't update between crawls.

Re: Two upstart search engines are teaming up to take on Google

#57

Post by ColinHayhurst »

arnaudsm wrote: post #45
unionpivo wrote: post #35

Earlier quoted context omitted.

Because nowdays more than ever content you need is in silos. Your facebooks/twiters/instagram/stack overflow/reddit ... And they all have limited expensive api's, and have bulk scrapping detection. Sure you can clobber together something that will work for a while, but you can't runn a buissness on that. Aditionaly most paywalled sites (like news) explicitly whitlist google and bing, and if someone cretes new site, t…

This is the best (and saddest) answer. LLMs break the social contract of the internet, we're in a feudalisation process. The decentralized nature of the internet was amazing for businesses, and monopolization could ruin the space and slow innovation down significantly.

> LLMs break the social contract of the internet

The legal concept of fair usage has and is being challenged, and will best tested in court. Is the Golden Age of Fair Use Over? Maybe [0].

[0] https://blog.mojeek.com/2024/05/is-the-golden-age-of-fair-us...

Re: Two upstart search engines are teaming up to take on Google

#59

Post by crestfallen »

ohstopitu wrote: post #32

> Ask the search engine Ecosia about “Paris to Prague” and flight booking websites dominate the results. Ecosia’s CEO Christian Kroll would prefer to present more train options, which he considers better for the environment. But because its results are licensed from Google and Microsoft’s Bing, Ecosia has little control over what’s shown. Kroll is ready for that to change. While I think Google sucks right now and we…

Respectfully, the search engine is "allowed" to do what it wants to for its business purpose. In Ecosia's case, that is to prefer environmentally sound modes of travel, sites, or businesses. And that's fine! What it means is that it might not be the search engine for you or for me. And that's fine too!
Post reply on HN