google.com/goto: Google's anti-scraping update • Hacker News Board

Re: google.com/goto: Google's anti-scraping update

#181

Post by mitxela » Sat, Sep 12, 2026, 7:01 AM UTC

nine_k wrote: post #110

lubujackson wrote: post #82

As much as I am sad that Google died like 15 years ago, I am past the mourning phase. That was when they announced they were shifting from returning websites to "returning answers" and it has been a long slide into shittification I do enjoy using their free AI. For actual web search I actually like using Yandex. It reminds me of old Google, returning reasonable results and much less "shaping results to please our cor…

The real "old-school Google, but modern" is Kagi, with the caveat of being paid. (Worth the $5 for me.) But LLMs can be commanded. This is may bookmark alias for invoking the spirit of old Gog=ogle from within the new, AI-based Google: https://google.com/search?q=You%20are%20Google%20Search%20fr... (Cleaned / decoded: 'You are Google Search from 2004. Given a search request, provide 10 links to relevant pages, each w…

Kagi works by scraping Google and other engines, so this should still worry you.

I still support the use of Kagi though, as a market signal to Google that we'd even pay them for their product if it wasn't dogshit.

Re: google.com/goto: Google's anti-scraping update

#182

Post by avereveard » Sat, Sep 12, 2026, 7:01 AM UTC

hogs_get_fat wrote: post #154

sofixa wrote: post #94

Earlier quoted context omitted.

> Yandex. It reminds me of old Google, returning reasonable results and much less "shaping results to please our corpo-political masters". As long as you don't search anything related to Russia itself, or to Russian interests elsewhere (like their invasion of Ukraine). Then it's very heavily biased, priority is given to state ran propaganda mills, independent media is hidden from results, etc. And before anyone start…

So, I decided to check, and the first links that show up when I search for "war ukraine" in yandex.com are Google news are from rbc.ua and bbc.com. When I search for it in yandex.com, the first link is CNN, the second google news, the third is indeed TASS, but the fourth is novayagazeta. I then picked "bucha" as the query most likely the be affected by propaganda and censorship. In yandex.ru, there are propaganda out…

Have your tests been ran on vpn with vpn cloaks?

Re: google.com/goto: Google's anti-scraping update

#183

Post by emaro » Sat, Sep 12, 2026, 7:01 AM UTC

noisy_boy wrote: post #145

nine_k wrote: post #110

Earlier quoted context omitted.

The real "old-school Google, but modern" is Kagi, with the caveat of being paid. (Worth the $5 for me.) But LLMs can be commanded. This is may bookmark alias for invoking the spirit of old Gog=ogle from within the new, AI-based Google: https://google.com/search?q=You%20are%20Google%20Search%20fr... (Cleaned / decoded: 'You are Google Search from 2004. Given a search request, provide 10 links to relevant pages, each w…

> Worth the $5 for me Steep at 300 searches per month or about 10 per day. If you search often, it is too expensive. If you rarely search, not worth 5 bucks. They are plainly trying to push people to their $10 unlimited plan. I would have appreciated if they allowed, say, 600 searches per month for $5 or so.

I do agree it's kinda expensive. I'm lucky enough that my employer pays the $25 plan for met because it includes access to many LLMs.

Re: google.com/goto: Google's anti-scraping update

#184

Post by Terr_ » Sat, Sep 12, 2026, 7:01 AM UTC

leobg wrote: post #104

InsideOutSanta wrote: post #48

Earlier quoted context omitted.

It's primarily relevant because it makes scraping search results much more expensive, solidifying Google's effective monopoly on Internet search. Google has previously tried to prevent scraping of search results using legal means, but courts correctly think that scraping of Google's search results should be legal, just as Google's scraping of the whole Internet is legal. This is Google's reaction to that.

This is the best explanation. They’ve been doing the same in Google News. Each entry comes not with a URL to the source, but with a hash. To resolve it, you must send requests to Google’s servers. Anyone who wants to create a list of URLs of sources automatically can therefore be blocked by Google now on two levels rather than one - the search for a list of results, and identifying the source URL for each result. In…

> It would be interesting to see if courts object to that.

It is also a measure of how enshittified and exploitative thing have become, that we look towards litigious copyright holders for assistance...

Re: google.com/goto: Google's anti-scraping update

#185

Post by vachina » Sat, Sep 12, 2026, 7:03 AM UTC

lubujackson wrote: post #82

As much as I am sad that Google died like 15 years ago, I am past the mourning phase. That was when they announced they were shifting from returning websites to "returning answers" and it has been a long slide into shittification I do enjoy using their free AI. For actual web search I actually like using Yandex. It reminds me of old Google, returning reasonable results and much less "shaping results to please our cor…

How is this comment relevant to the post (and be so upvoted?)

Google is making it harder for people to game the index, working in your favor.

Re: google.com/goto: Google's anti-scraping update

#186

Post by mitxela » Sat, Sep 12, 2026, 7:04 AM UTC

Jskewel wrote: post #152

JsonDemWitOster wrote: post #114

While a lot of people are concerned with local model performance, I wonder how feasible is it now to run a local indexed web search? Surely running an old school Google is possible with the beefy AI rigs today. I know the problem will be crawling which would be bottlenecked by the ISP but I use Google to search SO, Wikipedia, programming language docs, Github issues, and AWS docs. I think a feasible workflow would be…

Impossible. The majority of websites firewall automated crawler traffic (because of the rise of the bots), only making exceptions for the largest search engines. There is no possibility of starting a new crawler.

The majority of websites try to do that but they do not catch as much traffic as they think they do. A starting point for a scraper is to run it on your home connection in an undetectable web driver framework such as zendriver.

Re: google.com/goto: Google's anti-scraping update

#187

Post by knorker » Sat, Sep 12, 2026, 7:04 AM UTC

aucisson_masque wrote: post #123

So why are we angry about that ? I mean the end result for the users are exactly the same, it matters only for bots. Google have such a (justified) bad reputation that whatever they do, people assume it’s entishification. I don’t believed it is on that matter.

Well, for one it breaks right click and copy link location.

Re: google.com/goto: Google's anti-scraping update

#188

Post by noisy_boy » Sat, Sep 12, 2026, 7:05 AM UTC

emaro wrote: post #183

noisy_boy wrote: post #145

Earlier quoted context omitted.

> Worth the $5 for me Steep at 300 searches per month or about 10 per day. If you search often, it is too expensive. If you rarely search, not worth 5 bucks. They are plainly trying to push people to their $10 unlimited plan. I would have appreciated if they allowed, say, 600 searches per month for $5 or so.

I do agree it's kinda expensive. I'm lucky enough that my employer pays the $25 plan for met because it includes access to many LLMs.

> because it includes access to many LLMs.

That is interesting, which LLMs? Asking as someone paying monthly Claude subscription.

Re: google.com/goto: Google's anti-scraping update

#189

Post by mitxela » Sat, Sep 12, 2026, 7:05 AM UTC

1e1a wrote: post #118

joe_the_user wrote: post #78

Earlier quoted context omitted.

The link you followed when you clicked hasn't a direct link for years, decade afaik (they mangle so they can see what's followed). The page used to show the direct on the search text but now it shows some stand in for it - sometimes. You can see the direct link on the bottom of the screen when you hover - sometimes (and sometimes you see a mangled link). Sometimes the google link contains the original link in the cen…

I've just checked this again using a google account where search result pages are still following the old behavior: It looks like the 'href' attribute is the direct link, and the 'ping' attribute is the /url redirect link you are referring to. So it looks like it is actually sending me to the direct link, it just also requests /url at the same time in order to log the click. This means the user was not waiting for th…

Which is exactly the game theory that was predicted when some browsers started ignoring to "protect privacy". If your browser supports ping you get ping, otherwise the website gets the data anyway but with a worse user experience.

Re: google.com/goto: Google's anti-scraping update

#190

Post by stncls » Sat, Sep 12, 2026, 7:06 AM UTC

When Google stopped paid API search a few months ago, I looked for an alternative for my agents that I felt would be sustainable (one-time setup, then out of my mind). I quickly excluded SERP as I feard Google would pull exactly this type of shenanigans to cut them off.

I somehow found Mojeek and settled on it. I had never heard of them. Unlike Kagi, their business model is ads (so they hold no particular moral high ground). But they have a cheap, working paid API.

What I was astonished by is the quality of the results. For my uses, it's undistinguishable from Google. The conventional wisdom is that web search is a Google-sized problem. How did those obscure Brits pull it off?