Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Waiting for dawn in search: Search index, Google rulings and impact on Kagi

blog.kagi.com

Post reply on HN
Print view View original

101–110 of 266 posts

Re: Waiting for dawn in search: Search index, Google rulings and impact on Kagi

#101

Post by jeromechoo »

ghm2199 wrote: post #24

> Building a comparable one from scratch is like building a parallel national railroad.. Not too be pedantic here but I do have a noob question or two here: 1. One is building the index, which is a lot harder without a google offering its own API to boot. If other tech companies really wanted to break this monopoly, why can't they just do it — like they did with LLM training for base models with the infamous "pile" d…

Building an index is easy. Building a fresh index is extremely hard.

Ranking an index is hard. It's not just BM25 or cosine similarity. How do you prioritize certain domains over others? How do you rank homepages that typically have no real content in them for navigational queries?

Changing the behavior of 90% of the non-Chinese internet is unraveling 25 years and billions of dollars spent on ensuring Google is the default and sometimes only option.

Historically, it takes a significant technological counter position or anti-trust breakup for a behemoth like Google to lose its footing. Unfortunately for us, Google is currently competing well in the only true technological threat to their existence to appear in decades.

Re: Waiting for dawn in search: Search index, Google rulings and impact on Kagi

#102

Post by charcircuit »

walls wrote: post #62
ghm2199 wrote: post #24

> Building a comparable one from scratch is like building a parallel national railroad.. Not too be pedantic here but I do have a noob question or two here: 1. One is building the index, which is a lot harder without a google offering its own API to boot. If other tech companies really wanted to break this monopoly, why can't they just do it — like they did with LLM training for base models with the infamous "pile" d…

A huge amount of the web is only crawlable with a googlebot user-agent and specific source IPs.

If a crawler offered enough money they could be allowed too. It's not like Google has exclusive crawling rights.

Re: Waiting for dawn in search: Search index, Google rulings and impact on Kagi

#103

Post by renegat0x0 »

KellyCriterion wrote: post #69
ghm2199 wrote: post #24

> Building a comparable one from scratch is like building a parallel national railroad.. Not too be pedantic here but I do have a noob question or two here: 1. One is building the index, which is a lot harder without a google offering its own API to boot. If other tech companies really wanted to break this monopoly, why can't they just do it — like they did with LLM training for base models with the infamous "pile" d…

Scraping is hard. Very good scraping is even harder. And today, being a scraping business is veeery difficult; there are some "open"/public indices, but none of these other indices ever took off

Scraping is hard, and is not hard that much at the same time. There are many projects about scraping, so with a few lines you can do implement scraper using curl cffi, or playwright.

People complain that user-agent need to be filled. Boo-hoo, are we on hacker news, or what? Can't we just provide cookies, and user-agent? Not a big deal, right?

I myself have implemented a simple solution that is able to go through many hoops, and provide JSON response. Simple and easy [0].

On the other hand it was always an arms race. It will be. Eventually every content will be protected via walled gardens, there is no going around it.

Search engines affect me less, and less every day. I have my own small "index" / "bookmarks" with many domains, github projects, youtube channels [1].

Since the database is so big, the most used by me places is extracted into simple and fast web page using SQLite table [2]. Scraping done right is not a problem.

[0] https://github.com/rumca-js/crawler-buddy

[1] https://github.com/rumca-js/Internet-Places-Database

[2] https://rumca-js.github.io/search

Re: Waiting for dawn in search: Search index, Google rulings and impact on Kagi

#104

Post by 0x1ch »

direwolf20 wrote: post #88
0x1ch wrote: post #22

Earlier quoted context omitted.

Google is only blocked in places where it would already be hard for a company with morals to work in, if not outright blocked as well. This probably represents traffic globally, excluding those places. Instead of downvoting blindly, please state which countries are currently blocking Google that would willingly allow Kagi, a AI/Privacy focused search engine company to exist in their domain? The results may surprise y…

Google is not blocked in the USA.

Interesting. I'm in the US and use Kagi everyday.

Re: Waiting for dawn in search: Search index, Google rulings and impact on Kagi

#105

Post by charcircuit »

paxys wrote: post #63
ghm2199 wrote: post #24

> Building a comparable one from scratch is like building a parallel national railroad.. Not too be pedantic here but I do have a noob question or two here: 1. One is building the index, which is a lot harder without a google offering its own API to boot. If other tech companies really wanted to break this monopoly, why can't they just do it — like they did with LLM training for base models with the infamous "pile" d…

Apple had a chance to break Google's search monopoly, but they chose to take billions from them instead. Microsoft had a chance (well another chance, after they gave up IE's lead) to break up Google's browser monopoly, but they decided to use Chromium for free instead. Ultimately all these decisions come down to what's more profitable, not what's in the best interests of the public. We have learned this lesson x10000…

>but they chose to take billions from them instead.

They chose to use Google with a revenue sharing agreement. Google is very well monetized. It would be very difficult for Apple to monetize their own search as good as Google can.

>they decided to use Chromium

Windows ships with Microsoft Edge as the browser which Microsoft has full control over.

Re: Waiting for dawn in search: Search index, Google rulings and impact on Kagi

#106

Post by keeda »

stephen_cagle wrote: post #43

One interesting point was the original PageRank algorithm greatly benefited from the fact that we kinda only had "text matching" search before Google (my memory was AltaVista at the time). Because text matching was so difficult to search with, whenever you went to a site, it would often have a "web of trust" at the bottom where an actual human being had curated a list of other sites that you might like if you liked t…

> I am making the point that Google effectively benefited from the large amount of human labor...

This is exactly right, but the thing most people miss is that Google has been using human intelligence at massive scale even to this day to improve their search results.

Basically, as people search and navigate the results, Google harvests their clicks, hovers, dwell-time and other browsing behavior to extract critical signals that help it "learn" which pages the users actually found useful for the given query. (Overly simplified: click on a link but click back within a minute to go to the next link -> downrank, but spend more time on that link -> uprank.)

This helps it rank results better and improve search overall, which keeps people coming back and excluding competitors. It's like the web of trust again, except it's clicks of trust, and it's only visible to Google and is a never-ending self-reinforcing flywheel!

And if you look at the infrastructure Google has built to harvest this data, it is so much bigger than the massive index! They harvest data through Chrome, ad tracking, Android, Google Analytics, cookies (for which they built Gmail!), YouTube, Maps and so much more.

So to compete with Google Search, you don't need just a massive index, you also need the extensive web infra footprint to harvest user interactions at massive scale, which means the most popular and widely deployed browser, mobile OS, ad tracking, analytics script, email provider, maps, etc, etc.

This also explains why Google spent so many billions in "traffic acquisition costs" (i.e. payments for being the Search default) every year, because that was a direct driver to both, 1) ad revenue, and 2) maintaining its search quality.

This wasn't really a secret, but it (rightfully) turned out to be a major point in the recent Antitrust trial, which is why the proposed remedies (a TFA mentions) include the sharing of search index and "interaction data."

Re: Waiting for dawn in search: Search index, Google rulings and impact on Kagi

#107

Post by malfist »

eli wrote: post #23
xnx wrote: post #3

> Because direct licensing isn’t available to us on compatible terms, we - like many others - use third-party API providers for SERP-style results Crazy for a company to admit: "Google won't let us whitelabel their core product so we steal it and resell it."

Seems like an open question as to whether that violates any laws. Another way to look at it is that if you publish a service on the web, you have limited rights to restrict what people do with it. Isn't that the logic Google search relies on in the first place? I didn't give permission for Google to crawl and index and deep link to my site (let alone summarize and train LLMs on it). They just did it anyway, because i…

Google's stance is "I can copy you and you can't stop me" as well as "You can't copy me, I'll sue you"

Re: Waiting for dawn in search: Search index, Google rulings and impact on Kagi

#108

Post by charcircuit »

b3kart wrote: post #78
xnx wrote: post #60

Earlier quoted context omitted.

> "Could a meaningfully better search engine realistically displace Google today?” ChatGPT clearly demonstrated that displacing Google is possible. All previous monopoly arguments seemed even more flimsy after that.

I think you’re proving the monopoly argument yourself: if they only way to compete with Google is an innovation that generations of scientists have been working towards, it does paint a grim picture of competition in this space. Besides, are we ignoring Gemini?

Google already used AI and language models before ChatGPT came out. If you wanted a state of the art search / recommendation engine you needed that innovations from scientists already.

Re: Waiting for dawn in search: Search index, Google rulings and impact on Kagi

#109

Post by senko »

A full up-to-date index of the searchable web should be a public commons good.

This would not only allow better competition in search, but fix the "AI scrapers" problem: No need to scrape if the data has already been scraped.

Crawling is technically a solved problem, as witnessed by everyone and their dog seemingly crawling everything. If pooled together, it would be cheaper and less resource intensive.

The secret sauce is in what happens afterwards, anyway.

Here's the idea in more detail: https://senkorasic.com/articles/ai-scraper-tragedy-commons

I'm under no illusion something like that will happen .. but it could.

Re: Waiting for dawn in search: Search index, Google rulings and impact on Kagi

#110

Post by throw-the-towel »

creato wrote: post #95
oh_fiddlesticks wrote: post #72

Earlier quoted context omitted.

> 1. One is building the index, which is a lot harder without a google offering its own API to boot. If other tech companies really wanted to break this monopoly, why can't they just do it? FTA: > Context matters: Google built its index by crawling the open web before robots.txt was a widespread norm, often over publishers’ objections. Today, publishers “consent” to Google’s crawling because the alternative - being i…

robots.txt was being enforced in court before google even existed, let alone before google got so huge: > The robots.txt played a role in the 1999 legal case of eBay v. Bidder's Edge,[12] where eBay attempted to block a bot that did not comply with robots.txt, and in May 2000 a court ordered the company operating the bot to stop crawling eBay's servers using any automatic means, by legal injunction on the basis of tr…

Nitpick: Google incorporated in 1998, so, before the Bidder's Edge case.
Post reply on HN