Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Waiting for dawn in search: Search index, Google rulings and impact on Kagi

blog.kagi.com

Post reply on HN
Print view View original

181–190 of 266 posts

Re: Waiting for dawn in search: Search index, Google rulings and impact on Kagi

#181

Post by Ronsenshi »

zhfanlqeo wrote: post #179

I used kagi for a while but got lazy with updating the subscription when moving and needing to change credit cards so I went back to DDG/Google and having to go back to having to skip the first result or first few results shows you just how obnoxious this practice is. When I have a few moments I'll resubscribe to kagi...

I've been trying to use DDG for the past 2-3 years, but way too often I have to add !g at the end to go to google where I can get better results. So I've been considering giving Kagi a try. Can you tell if in your experience Kagi has better results than DDG?

Re: Waiting for dawn in search: Search index, Google rulings and impact on Kagi

#182

Post by gkbrk »

stacktraceyo wrote: post #148

Is there a crowd indexed style search index? Like instead of relying on the crawling completely you rely on a maybe like an extension in your browser that indexes as people are using their browser. Or maybe indexing your site to this index instead of waiting to be crawled.

I think Brave Search does something similar with their Web Discovery Project, but I don't think it indexes full web pages from users.

https://support.brave.app/hc/en-us/articles/4409406835469-Wh...

Re: Waiting for dawn in search: Search index, Google rulings and impact on Kagi

#183

Post by senko »

moebrowne wrote: post #121
senko wrote: post #109

A full up-to-date index of the searchable web should be a public commons good. This would not only allow better competition in search, but fix the "AI scrapers" problem: No need to scrape if the data has already been scraped. Crawling is technically a solved problem, as witnessed by everyone and their dog seemingly crawling everything. If pooled together, it would be cheaper and less resource intensive. The secret sa…

Isn't this what CommonCrawl are doing? https://commoncrawl.org/

Yes. But they don't crawl everything (probably due to lack of funding), and, as the article and other commenters here note, people are incentivised to allow Google and only Google to crawl. In practice, the CommonCrawl dataset is too small for a realistic search engine competitor.

I'd love to see Google, Bing and others being incentivized (wink, wink) to contribute (technically, financially, etc) to CommonCrawl or Internet Archive since they already do this.

Re: Waiting for dawn in search: Search index, Google rulings and impact on Kagi

#184

Post by bilekas »

I've tried Kagi and while it is better than google these days, to be fair that's not hard with the enshitification slop that's out there.

But Kagi funds Yandex which fund the RU government, and I think it should be known to anyone looking to use it.

https://ounapuu.ee/posts/2025/07/17/kagi/

https://kagifeedback.org/d/5445-reconsider-yandex-integratio...

Re: Waiting for dawn in search: Search index, Google rulings and impact on Kagi

#185

Post by fuzzy2 »

m-schuetz wrote: post #164
pfist wrote: post #87

I am rooting for Kagi here, and I applaud their transparency on such matters. It is quite enlightening for someone like me who understands technology but knows little about the inner workings of search. It remains to be seen how or if the remedies will be enforced, and, of course, how Google will choose to comply with them. I am not optimistic, but at least there is some hope. As an aside: The 1998 white paper by Bri…

I'm rooting for Kagi solely because the block feature. It's amazing to be able to block undeservedly SEO'd garbage sites from future search results.

fyi: DuckDuckGo has blocking now, too. I use it extensively to do away with all the clone sites of Stack Exchange, GitHub etc

All without using an account, saved locally in the browser.

Re: Waiting for dawn in search: Search index, Google rulings and impact on Kagi

#186

Post by senko »

azornathogron wrote: post #123
senko wrote: post #109

A full up-to-date index of the searchable web should be a public commons good. This would not only allow better competition in search, but fix the "AI scrapers" problem: No need to scrape if the data has already been scraped. Crawling is technically a solved problem, as witnessed by everyone and their dog seemingly crawling everything. If pooled together, it would be cheaper and less resource intensive. The secret sa…

Is crawling really solved? Any naive crawler is going to run into the problem that servers can give different responses to different clients which means you can show the crawler something different to what you show real users. That turns crawling into an antagonistic problem where the crawler developers need to continually be on the lookout for new ways of servers doing malicious things that poison/mislead the index.…

I don't mean to say it's trivial. I'm sure there are many hard problems such as the one you mention - though that particular one is more "cleaning the index" part which might work on top of the open common corpus.

But my impression is that it's more a question of scale and engineering time than having to invent something new.

(disclaimer: I also never worked on a internet-scale search system, maybe I'm very off the bat here as well).

Re: Waiting for dawn in search: Search index, Google rulings and impact on Kagi

#187

Post by m-schuetz »

fuzzy2 wrote: post #185
m-schuetz wrote: post #164

Earlier quoted context omitted.

I'm rooting for Kagi solely because the block feature. It's amazing to be able to block undeservedly SEO'd garbage sites from future search results.

fyi: DuckDuckGo has blocking now, too. I use it extensively to do away with all the clone sites of Stack Exchange, GitHub etc All without using an account, saved locally in the browser.

Oh nice, that's good to know. Yes, those clones sites are also instantly on my block list, as well as Userbenchmark, sites with AI-generated "info" pages (if I want AI answers, I'll just ask ChatGPT), sites that won't work without third-part cookies, low-quality game guide sites that were evidently made for users to visit, but not actually to help them, etc.

Re: Waiting for dawn in search: Search index, Google rulings and impact on Kagi

#188

Post by pas »

AlienRobot wrote: post #132
jeromechoo wrote: post #101

Earlier quoted context omitted.

Building an index is easy. Building a fresh index is extremely hard. Ranking an index is hard. It's not just BM25 or cosine similarity. How do you prioritize certain domains over others? How do you rank homepages that typically have no real content in them for navigational queries? Changing the behavior of 90% of the non-Chinese internet is unraveling 25 years and billions of dollars spent on ensuring Google is the d…

Good news! Google doesn't know how to rank pages either!

yet ... it works "ok" most of the time.

not to mention that people mostly need wikipedia, the news, navigating the infuriating world of websites of big service providers (gov sites, or try to find anything on Microsoft's dark corner of the web), porn and brainrot

but it's awfully hard to make traction on a business that provides this.

Re: Waiting for dawn in search: Search index, Google rulings and impact on Kagi

#190

Post by ddtaylor »

hattmall wrote: post #158
walls wrote: post #62

Earlier quoted context omitted.

A huge amount of the web is only crawlable with a googlebot user-agent and specific source IPs.

Are these websites not serving public content? If there's some legal concerns just create a separate scraping LLC that fakes user agent and uses residential IPs or VPN or something. I can't imagine that the companies would follow through with some sort of lawsuit against a scraper that's trying to index their site to get them more visitors, if they allow GoogleBot.

Isnt that what SerpAPI was doing?
Post reply on HN