Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Waiting for dawn in search: Search index, Google rulings and impact on Kagi

blog.kagi.com

Post reply on HN
Print view View original

131–140 of 266 posts

Re: Waiting for dawn in search: Search index, Google rulings and impact on Kagi

#131

Post by ghm2199 »

renegat0x0 wrote: post #103
KellyCriterion wrote: post #69

Earlier quoted context omitted.

Scraping is hard. Very good scraping is even harder. And today, being a scraping business is veeery difficult; there are some "open"/public indices, but none of these other indices ever took off

Scraping is hard, and is not hard that much at the same time. There are many projects about scraping, so with a few lines you can do implement scraper using curl cffi, or playwright. People complain that user-agent need to be filled. Boo-hoo, are we on hacker news, or what? Can't we just provide cookies, and user-agent? Not a big deal, right? I myself have implemented a simple solution that is able to go through many…

When I saw the Internet-Places-Database I thought it was an index on some sort of PoI and I got curious. But the personal internet spiel is pretty cool. One good addition to this could be the Foursquare PoI dataset for places search: https://opensource.foursquare.com/os-places/

Re: Waiting for dawn in search: Search index, Google rulings and impact on Kagi

#132

Post by AlienRobot »

jeromechoo wrote: post #101
ghm2199 wrote: post #24

> Building a comparable one from scratch is like building a parallel national railroad.. Not too be pedantic here but I do have a noob question or two here: 1. One is building the index, which is a lot harder without a google offering its own API to boot. If other tech companies really wanted to break this monopoly, why can't they just do it — like they did with LLM training for base models with the infamous "pile" d…

Building an index is easy. Building a fresh index is extremely hard. Ranking an index is hard. It's not just BM25 or cosine similarity. How do you prioritize certain domains over others? How do you rank homepages that typically have no real content in them for navigational queries? Changing the behavior of 90% of the non-Chinese internet is unraveling 25 years and billions of dollars spent on ensuring Google is the d…

Good news! Google doesn't know how to rank pages either!

Re: Waiting for dawn in search: Search index, Google rulings and impact on Kagi

#133

Post by AlienRobot »

z64 wrote: post #71
direwolf20 wrote: post #4

I hope they cache search results to further reduce the number of calls to Google. And Marginalia Search was not mentioned? Marginalia Search says they are licensing their index to Kagi. Perhaps it's counted under "Our own small-web index" which is highly misleading if true.

There is a practical limit that we can't cache results for too long; Search engine users are particularly sensitive to stale data, especially around current events. Without a holistic and realiable way to know when the cache ought to be invalidated, our caching is mostly focused on mitigating "abuse", e.g., someone / bunch of people spamming the same search in a short timespan; no sense in repeating all those upstrea…

To me, a lot of problems with "building a search engine" don't seem to be problems with "building a search engine," they seem to be problems with "building a Google."

Nobody said a search engine needs to have fresh data, for example. Nor has anybody said a search engine needs to index the entire web. Yet these are two things every search engine tries to do, and then they usually fail to compare with Google.

To put it in another way, the reason why TikTok succeeded against Youtube is exactly because TikTok wasn't trying to be a Youtube.

Re: Waiting for dawn in search: Search index, Google rulings and impact on Kagi

#134

Post by Nextgrid »

dylan604 wrote: post #112
baggachipz wrote: post #83

Earlier quoted context omitted.

A classic case of climbing the wall, and pulling the ladder up afterward. Others try to build their own ladder, and Google uses their deep pockets and political influence to knock the ladder over before it reaches the top.

Why does Google even need to know about your ladder? Build the bot, scale it up, save all the data, then release. You can now remove the ladder and obey robots.txt just like G. Just like G, once you have the data, you have the data. Why would you tell G that you are doing something? Why tell a competitor your plans at all? Just launch your product when the product is ready. I know that's anathema to SV startup logic,…

Running the bot nowadays is hard, because a lot of sites will now block you - not just by asking nicely via robots.txt, but by checking your actual source IP. Once they see it's not Google, they send you a 403.

Re: Waiting for dawn in search: Search index, Google rulings and impact on Kagi

#135

Post by citizenpaul »

ghm2199 wrote: post #24

> Building a comparable one from scratch is like building a parallel national railroad.. Not too be pedantic here but I do have a noob question or two here: 1. One is building the index, which is a lot harder without a google offering its own API to boot. If other tech companies really wanted to break this monopoly, why can't they just do it — like they did with LLM training for base models with the infamous "pile" d…

>why can't they just do it

Money. Google controls 99% of the adverting market. That's why its called a monopoly. No one else can compete because they can never make enough money to make it worth the costs of doing it themselves.

Re: Waiting for dawn in search: Search index, Google rulings and impact on Kagi

#136

Post by Nextgrid »

ghm2199 wrote: post #73
KellyCriterion wrote: post #69

Earlier quoted context omitted.

Scraping is hard. Very good scraping is even harder. And today, being a scraping business is veeery difficult; there are some "open"/public indices, but none of these other indices ever took off

Well sure yes, I don't contend with the fact that its hard, but if the top tech companies joined their heads I am sure if for example, Meta, Apple, MS have enough talent between to make an open source index if only to reap gains from the de-monopolization of it all.

All these companies have the exact same business model as Google (advertising) and have the same mismatched incentives: good search results are not something they want.

Google Search sucks not because Google is incapable of filtering out spam and SEO slop (though they very much love that people believe they can't), but that spam/slop makes the ads on the SERP page more enticing, and some of the spam itself includes Google Ads/analytics and benefits them there too.

There is no incentive for these companies to build a good search engine by themselves to begin with, let alone provide data to allow others to build one.

Re: Waiting for dawn in search: Search index, Google rulings and impact on Kagi

#137

Post by Nextgrid »

charcircuit wrote: post #102
walls wrote: post #62

Earlier quoted context omitted.

A huge amount of the web is only crawlable with a googlebot user-agent and specific source IPs.

If a crawler offered enough money they could be allowed too. It's not like Google has exclusive crawling rights.

There is a logistics problem here - even if you had enough money to pay, how would you get in touch with every single site to even let them know you're happy to pay? It's not like site operators routinely scan their error logs to see your failed crawling attempts and your offer in the user-agent.

Even if they see it, it's a classic chicken & egg problem: it's not worth the time of the site operator to engage with your offer until your search engine popular enough to matter, but your search engine will never become popular enough to matter if it doesn't have a critical mass of sites to begin with.

Re: Waiting for dawn in search: Search index, Google rulings and impact on Kagi

#138

Post by monooso »

dylan604 wrote: post #112
baggachipz wrote: post #83

Earlier quoted context omitted.

A classic case of climbing the wall, and pulling the ladder up afterward. Others try to build their own ladder, and Google uses their deep pockets and political influence to knock the ladder over before it reaches the top.

Why does Google even need to know about your ladder? Build the bot, scale it up, save all the data, then release. You can now remove the ladder and obey robots.txt just like G. Just like G, once you have the data, you have the data. Why would you tell G that you are doing something? Why tell a competitor your plans at all? Just launch your product when the product is ready. I know that's anathema to SV startup logic,…

Cost, presumably. From the article:

> Microsoft spent roughly $100 billion over 20 years on Bing and still holds single-digit share. If Microsoft cannot close the gap, no startup can do it alone.

Re: Waiting for dawn in search: Search index, Google rulings and impact on Kagi

#139

Post by Nextgrid »

AlienRobot wrote: post #133
z64 wrote: post #71

Earlier quoted context omitted.

There is a practical limit that we can't cache results for too long; Search engine users are particularly sensitive to stale data, especially around current events. Without a holistic and realiable way to know when the cache ought to be invalidated, our caching is mostly focused on mitigating "abuse", e.g., someone / bunch of people spamming the same search in a short timespan; no sense in repeating all those upstrea…

To me, a lot of problems with "building a search engine" don't seem to be problems with "building a search engine," they seem to be problems with "building a Google." Nobody said a search engine needs to have fresh data, for example. Nor has anybody said a search engine needs to index the entire web. Yet these are two things every search engine tries to do, and then they usually fail to compare with Google. To put it…

I don't think TikTok "succeeded" compared to Youtube? TikTok succeeded in popularizing short-form video, but I'd argue that's a different product. YouTube is still king for longform video.

While there might be arguments for building a different product (and LLM-based search like Perplexity is trying it), there appears to be enough demand for a "good Google" that Kagi is trying to address.

Re: Waiting for dawn in search: Search index, Google rulings and impact on Kagi

#140

Post by Nextgrid »

keeda wrote: post #117

Google's advantage is not just in its index and algorithms, it is that it has built a self-reinforcing flywheel that data mines human attention at massive scale to improve their search results. This comment ( https://news.ycombinator.com/item?id=46709957 ) points out that Google got its start via PageRank, which essentially ranked sites based on links created by humans . As such, its primary heuristic was what humans…

> "learn" which pages the users actually found useful for the given query

But due to their business model I'm not sure they are ranking "usefulness" as much as you think.

Useful results ultimately don't benefit Google because Google makes no money on them. Google makes money on ads - either ads on the search results page, ads on the destination pages or (indirectly) from steering users to pages which have Google Analytics.

It's likely the actual algorithm balances usefulness to the user with usefulness to Google. You don't want to serve up exclusively spam/slop as users might bounce, but you also don't want to serve up the best result because the user will prefer it over the ad on the SRP page. So it has to be a mix of both - you'll eventually get a good result, after many attempts (during which you've been exposed to ads).

Google does enjoy the myth that they are unable to combat spam/slop while in reality they do profit off it.

Post reply on HN