Live data from Hacker News

google.com/goto: Google's anti-scraping update

autom.dev

41–50 of 545 posts

Re: google.com/goto: Google's anti-scraping update

#41

ISPs can presumably correlate the Google query string with the request following the response to the goto and so make a search index? I guess they would charge too much. Do any large ISPs use visit data to feed into a search index?

ISPs do not see query strings since Google uses HTTPS. ISPs can only see the domain and IP address you are connecting to.

Re: google.com/goto: Google's anti-scraping update

#42
post #33

I've been using DuckDuckGo for years now, ever since it became noticeable that two different people searching for the same search term would get two different results back from Google. Meaning they were no longer completely reliable: they might show one person a result that they hide from the other person by burying it on page 3 where few people ever look. DDG's search results have been poorer recently than they used…

Speaking of page 3, Google's been asking me to sign in if I try to go beyond page 2 of search results. What a wonderful modern WWW we live in.

Re: google.com/goto: Google's anti-scraping update

#43
post #24

Can someone explain why this matters? Not being flippant I just don’t understand why this would be important.

The security implications of this are very severe when you consider the amount of people who google government websites, banking, crypto and others. And google will happily serve you a phishing website either in ads or results.

[deleted]

Re: google.com/goto: Google's anti-scraping update

#45
post #33

I've been using DuckDuckGo for years now, ever since it became noticeable that two different people searching for the same search term would get two different results back from Google. Meaning they were no longer completely reliable: they might show one person a result that they hide from the other person by burying it on page 3 where few people ever look. DDG's search results have been poorer recently than they used…

Personalized search results are not ipso facto unreliable.

It means they're capable of burying news stories that would contradict your worldview and pushing news stories that support your pre-existing biases, leading to more engagement from you (a win from their point of view) but also burying you in an echo chamber. And unless you were in the habit of doing the occasional search in Incognito Mode, you wouldn't know. (And even then, they probably would be able to put together enough clues to figure out your identity even without your Google login cookie).

Yes, the fact that they could do it does not prove that they were doing it, not right away. I ditched them as soon as I found out that they could do it, because I was absolutely certain that eventually, they would end up doing it. And I wanted neutral search results, not biased ones, even ones biased towards my own point of view.

Re: google.com/goto: Google's anti-scraping update

#47

I... Don't see it? It's the result page right? I just search some random string on Google and the results are all direct URLs. Do they get resolved via javascript after page load and replaced automatically? Or am I looking at something else?

It looks like it isn't yet rolled out to some users, try with a different browser session.

Re: google.com/goto: Google's anti-scraping update

#48

Can someone explain why this matters? Not being flippant I just don’t understand why this would be important.

It's primarily relevant because it makes scraping search results much more expensive, solidifying Google's effective monopoly on Internet search.

Google has previously tried to prevent scraping of search results using legal means, but courts correctly think that scraping of Google's search results should be legal, just as Google's scraping of the whole Internet is legal. This is Google's reaction to that.

Re: google.com/goto: Google's anti-scraping update

#49

I use ChatGPT for almost all my searches now. I’m not joking.

I wish one of those free AIs made a search product already. I don't want my search results to be interspersed with text I guess that the web chat can have a search skill to remove the prose and give only links, plus maybe an excerpt of each result

Z.ai has a search MCP server, it should be trivial to use that to build a basic search UI on top.

Re: google.com/goto: Google's anti-scraping update

#50

I've been using the 'ClearURLs' FF addon for quite a while now, and it gives you the URLs back in the results. https://docs.clearurls.xyz/

That wouldn’t work at all with this url format - the url can only be resolved by requesting the goto link. I recommend reading more than the headline

[deleted]
Post reply on HN