Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

google.com/goto: Google's anti-scraping update

autom.dev

Post reply on HN
Print view View original

71–80 of 545 posts

Re: google.com/goto: Google's anti-scraping update

#71

Post by mentalpiracy »

darknavi wrote: post #40
1e1a wrote: post #2

Direct URLs in Google search results have been replaced with redirect URLs in the form of www.google.com/goto?url= . The base64 data appears to consist of a very basic protobuf structure, containing a long string of bytes in field 2 which presumably identify the URL. Sometimes, these redirect URLs take a perceivable amount of time to load, which is very irritating.

> Sometimes, these redirect URLs take a perceivable amount of time to load, which is very irritating. Great. On top of my on-going battle with Windows + Firefox + DNS/TLS resolution sometimes stalling for seconds at a time, another few second server-side stall is introduced. I swear that every day modern computing scenarios get slower and slower instead of snappier and snappier.

Wow - this exact same bug has been happening to me too. I gave up on troubleshooting it after the first few attempts came up with nothing, assumed it was just unique to me.

Re: google.com/goto: Google's anti-scraping update

#72

Post by adtac »

firefoxd wrote: post #55

At this point, we can all just drop SEO [0]. We are writing content for a robot that hides the source of information. [0]: https://news.ycombinator.com/item?id=49665572

but you can click the link and reveal the information. what is being hidden? from who?

Re: google.com/goto: Google's anti-scraping update

#74

Post by josephcsible »

liquid_thyme wrote: post #5

I've been using the 'ClearURLs' FF addon for quite a while now, and it gives you the URLs back in the results. https://docs.clearurls.xyz/

The reason this change is so bad is that it encrypts the actual URL, specifically to break addons like that.

Re: google.com/goto: Google's anti-scraping update

#76

Post by nomilk »

> Combined with earlier moves like removing &num=100

Removing this made google search horrible to use. I often use command+f to quickly identify relevant search results, but doing it on 10 results at a time is so laborious that I just don't bother using Google search, resulting in less searches and use of other tools instead.

Re: google.com/goto: Google's anti-scraping update

#77

Post by demibabs »

MiroslavPokorny wrote: post #64
demibabs wrote: post #10

Can someone explain why this matters? Not being flippant I just don’t understand why this would be important.

It should be obvious, they are doing this for a reason to benefit themselves. As others have elaborated, the reasons are so they can track who you are and sell your profile advertising.

> the reasons are so they can track who you are and sell your profile advertising.

What? Like they weren’t doing this before? Obviously Google’s telemetry is tracking every link you click regardless; there’s no extra tracking benefit to this.

The reason they’re doing this seems to be to stop competitors from scraping their search results.

Re: google.com/goto: Google's anti-scraping update

#78

Post by joe_the_user »

1e1a wrote: post #2

Direct URLs in Google search results have been replaced with redirect URLs in the form of www.google.com/goto?url= . The base64 data appears to consist of a very basic protobuf structure, containing a long string of bytes in field 2 which presumably identify the URL. Sometimes, these redirect URLs take a perceivable amount of time to load, which is very irritating.

The link you followed when you clicked hasn't a direct link for years, decade afaik (they mangle so they can see what's followed). The page used to show the direct on the search text but now it shows some stand in for it - sometimes. You can see the direct link on the bottom of the screen when you hover - sometimes (and sometimes you see a mangled link). Sometimes the google link contains the original link in the center also[1].

The situation seems to vary from result to result even on the same page of the same search - at least on the test search I just did. You can figure out what happening to an extent but this very inconsistency seems to speak to a dystopian quality to today's information gatekeepers.

[1] Example. https://www.google.com/url?sa=t&source=web&rct=j&opi=8997844...

Re: google.com/goto: Google's anti-scraping update

#79

Post by genezeta »

Lvl999Noob wrote: post #46

I... Don't see it? It's the result page right? I just search some random string on Google and the results are all direct URLs. Do they get resolved via javascript after page load and replaced automatically? Or am I looking at something else?

What I see now, and it's been like this for a while, is this:

You get the results and they do have direct URLs. But then, if you do some things with the link, e.g. right click to open it in a new tab, it swaps the URL to the indirect one. The idea is that initially you see a normal link, with a normal URL which will be displayed correctly when you hover the mouse over it, but right before you click it, it's swapped for the indirect one.

So, they have been doing stuff like this for a while and it has been somewhat fluid, because the swapping can occur on different events and I have also seen it load with all the links pre-swapped to the indirect ones, sometimes.

So, yes, what you see may be different and you may get the indirect URLs swapped at different stages.

Post reply on HN