121–130 of 266 posts
A full up-to-date index of the searchable web should be a public commons good. This would not only allow better competition in search, but fix the "AI scrapers" problem: No need to scrape if the data has already been scraped. Crawling is technically a solved problem, as witnessed by everyone and their dog seemingly crawling everything. If pooled together, it would be cheaper and less resource intensive. The secret sa…
Earlier quoted context omitted.
> 1. One is building the index, which is a lot harder without a google offering its own API to boot. If other tech companies really wanted to break this monopoly, why can't they just do it? FTA: > Context matters: Google built its index by crawling the open web before robots.txt was a widespread norm, often over publishers’ objections. Today, publishers “consent” to Google’s crawling because the alternative - being i…
robots.txt was being enforced in court before google even existed, let alone before google got so huge: > The robots.txt played a role in the 1999 legal case of eBay v. Bidder's Edge,[12] where eBay attempted to block a bot that did not comply with robots.txt, and in May 2000 a court ordered the company operating the bot to stop crawling eBay's servers using any automatic means, by legal injunction on the basis of tr…
A full up-to-date index of the searchable web should be a public commons good. This would not only allow better competition in search, but fix the "AI scrapers" problem: No need to scrape if the data has already been scraped. Crawling is technically a solved problem, as witnessed by everyone and their dog seemingly crawling everything. If pooled together, it would be cheaper and less resource intensive. The secret sa…
Any naive crawler is going to run into the problem that servers can give different responses to different clients which means you can show the crawler something different to what you show real users. That turns crawling into an antagonistic problem where the crawler developers need to continually be on the lookout for new ways of servers doing malicious things that poison/mislead the index.
Otherwise you'll return junk spam results from spammers that lied to the crawler.
I've never done it so maybe it's easier than I imagine but I wouldn't be quick to assume that crawling is solved.
Earlier quoted context omitted.
> Yeah but no one uses it. I am not even sure people like using it because it was productized it pretty poorly. They invested 100 Billion dollars, which is a lot of wasted money TBH. I mean... Duckduckgo uses bing api iirc and I use duckduckgo and many people use duckduckgo. I also used bing once because bing used to cache websites which weren't available in wayback archive, I don't know how but It was pretty cool so…
I would imagine the users of DDG to be closer to a rounding error than an actual percentage of users. I'd imagine theGoog would love and hate to have 100%. They'd love it because all the data, and hate it as it would prove the monopoly. At the end of the day, the % that is not going to them probably doesn't cause theGoog to lose much sleep
It's available in all major browsers (Here in zen browser, it doesn't even have a default browser but rather on the start page it asks between the three options, google duckduckgo and bing but yes if you press next it starts from google but zen can even start from ddg, its not such a big deal)
Duckduckgo is super amazing. I mean they are so amazing and their duck.ai or ai actually provides concise data instead of Google's AI
DDG is leaps ahead of Google in terms of everything. I found Kagi to be pleasant too but with PPP it might make sense in Europe and America but privacy isn't/ shouldn't be the only who only pays. So DDG is great for me personally and I can't recommend it enough for most cases.
Brave/Startpage is a second but DDG is so good :)
It just works (for most cases, the only use case I use google is for uploading images to then get more images like this or use an image as a search query and I just do !gi and open images.google.com but I only use this function very rarely, bangs are amazing feature by ddg)
Earlier quoted context omitted.
I would imagine the users of DDG to be closer to a rounding error than an actual percentage of users. I'd imagine theGoog would love and hate to have 100%. They'd love it because all the data, and hate it as it would prove the monopoly. At the end of the day, the % that is not going to them probably doesn't cause theGoog to lose much sleep
It's just so wild how great Duckduckgo is & how under-rated it is. It's available in all major browsers (Here in zen browser, it doesn't even have a default browser but rather on the start page it asks between the three options, google duckduckgo and bing but yes if you press next it starts from google but zen can even start from ddg, its not such a big deal) Duckduckgo is super amazing. I mean they are so amazing an…
Kagi's "waiting for dawn" is just waiting for Google to legitimize their reseller business Meanwhile, users pay a premium to pretend they're not using Google Fascinating delusion
> Building a comparable one from scratch is like building a parallel national railroad.. Not too be pedantic here but I do have a noob question or two here: 1. One is building the index, which is a lot harder without a google offering its own API to boot. If other tech companies really wanted to break this monopoly, why can't they just do it — like they did with LLM training for base models with the infamous "pile" d…
> If other tech companies really wanted to break this monopoly, why can't they just do it Google is a verb, nobody can compete with that level of mindshare.
Earlier quoted context omitted.
Scraping is hard. Very good scraping is even harder. And today, being a scraping business is veeery difficult; there are some "open"/public indices, but none of these other indices ever took off
Scraping is hard, and is not hard that much at the same time. There are many projects about scraping, so with a few lines you can do implement scraper using curl cffi, or playwright. People complain that user-agent need to be filled. Boo-hoo, are we on hacker news, or what? Can't we just provide cookies, and user-agent? Not a big deal, right? I myself have implemented a simple solution that is able to go through many…
Building a metasearch engine is not hard to do (especially with AI now). It's so liberating when you control the ranking algorithm, and can supplement what the big engines provide as results with your own index of sites and pages that are important to you. I admit, my results & speed aren't as good as Kagi, but still good enough that my personal search engine has been my sole search engine for a year now.
If a site doesn't want me to crawl them, that's fine. I probably don't need them. In practice it hasn't gotten in the way as much as I might have thought it would. But I do still rely on Brave / Mojeek / Marginalia to do much of the heavy lifting for me.
I especially appreciate Marginalia for publicly documenting as much about building a search engine as they have: https://www.marginalia.nu/log/
https://storage.courtlistener.com/recap/gov.uscourts.dcd.223...
Will Kagi file an amicus brief in support of the plaintiffs
Perhaps Google will fund amici in support of their position as they did in the Epic appeal
https://www.law.com/nationallawjournal/2025/01/10/fight-over...
I think one side problem is that part of the web is not even searchable with a search engine. Here are some examples: - Discord - WeChat (is it the web?) - Rednote - TikTok (partially) - X (partially) - JSTOR (it finds daily, but you find more stuff on the website directly) - any stuff with a login, obviously.
Damn, I can't stand open-source projects that host their "forums" on Discord. It's a nigthmare to use, it's heavy, slow, and it's completely unsearchable from the web.
I wonder what went wrong with our society.