Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

google.com/goto: Google's anti-scraping update

autom.dev

Post reply on HN
Print view View original

111–120 of 545 posts

Re: google.com/goto: Google's anti-scraping update

#111

Post by sureMan6 »

sofixa wrote: post #94
lubujackson wrote: post #82

As much as I am sad that Google died like 15 years ago, I am past the mourning phase. That was when they announced they were shifting from returning websites to "returning answers" and it has been a long slide into shittification I do enjoy using their free AI. For actual web search I actually like using Yandex. It reminds me of old Google, returning reasonable results and much less "shaping results to please our cor…

> Yandex. It reminds me of old Google, returning reasonable results and much less "shaping results to please our corpo-political masters". As long as you don't search anything related to Russia itself, or to Russian interests elsewhere (like their invasion of Ukraine). Then it's very heavily biased, priority is given to state ran propaganda mills, independent media is hidden from results, etc. And before anyone start…

> And before anyone starts thinking about whatabouting, Yandex are based in a country where journalists are openly and publicly assasinated to intimadate

Thankfully Gary Webb is here with us today to laugh at this

Re: google.com/goto: Google's anti-scraping update

#112

Post by bitpush »

leobg wrote: post #95
bitpush wrote: post #89

Earlier quoted context omitted.

> There are still illegal streaming sports and movie sites everywhere (who knew) and all other seedy corners of the internet that have been neatly erased by Google This is such a strange position to take. Apple doesnt allow all sorts of apps on the appstore, but that is never said as "Apple is erasing illegal streaming". Google is simply not showing the results. They are not taking down, or banning, or doing anything…

The pain being made is: If you use a search engine as your eyes to see what exists “on the internet”, then, absolutely, whatever Google hides from its results or fails to index is “erased” from “your” experience of the internet.

I see where you are coming from but the argument would be stronger is Google Chrome refused to open illegal streaming sites etc. AFAIK, there's no such restriction.

(I dont agree to this but ...) using your argument of " If you use a X as your eyes to see what exists “on the internet” .. " - we should all be mad at Apple. I use iPhone as the primary device to access apps, and they not just hides but actively ban and cut whole swathes of developers.

If I ask Siri to give me a link to illegal streaming site and if it refuses, is that cause for concern? I'd say no. Infact, I dont expect it to give me that and I get it. Same for Google Search in my humble opinion.

Re: google.com/goto: Google's anti-scraping update

#113

Post by brainwad »

adtac wrote: post #72
firefoxd wrote: post #55

At this point, we can all just drop SEO [0]. We are writing content for a robot that hides the source of information. [0]: https://news.ycombinator.com/item?id=49665572

but you can click the link and reveal the information. what is being hidden? from who?

Like the OP says, it's being hidden form scrapers. Like you I don't see a reason for normal users to be concerned.

Re: google.com/goto: Google's anti-scraping update

#114

Post by JsonDemWitOster »

While a lot of people are concerned with local model performance, I wonder how feasible is it now to run a local indexed web search? Surely running an old school Google is possible with the beefy AI rigs today. I know the problem will be crawling which would be bottlenecked by the ISP but I use Google to search SO, Wikipedia, programming language docs, Github issues, and AWS docs. I think a feasible workflow would be to build a set of sites of most interest to you and then prioritize those in crawling.

While typing this out I remembered https://en.wikipedia.org/wiki/Google_Search_Appliance which I never personally used but shows feasibility for the idea. I'm pretty sure one of the newly-announced Macbooks is more than up to the task of matching GSA's offering.

Re: google.com/goto: Google's anti-scraping update

#116

Post by stickfigure »

Lvl999Noob wrote: post #46

I... Don't see it? It's the result page right? I just search some random string on Google and the results are all direct URLs. Do they get resolved via javascript after page load and replaced automatically? Or am I looking at something else?

I was also super confused because it doesn't do this if you are logged in (to google).

I ran the same search in an incognito window and it showed the /goto links.

Re: google.com/goto: Google's anti-scraping update

#117

Post by brainwad »

JohnFen wrote: post #22
scottyah wrote: post #11

Earlier quoted context omitted.

Why? I don't really see why this is bad, other than if your company blocks url shorteners/redirects. What are you losing?

Because I wouldn't want Google to know what links I click on. When I used to use Google, and they added such redirects that included the destination URL as a parameter, I'd edit the link to make it just that URL before resolving it. This new scheme would make that impossible.

They own the JS on the page, they don't need the redirects to know where you clicked...

Re: google.com/goto: Google's anti-scraping update

#118

Post by 1e1a »

joe_the_user wrote: post #78
1e1a wrote: post #2

Direct URLs in Google search results have been replaced with redirect URLs in the form of www.google.com/goto?url= . The base64 data appears to consist of a very basic protobuf structure, containing a long string of bytes in field 2 which presumably identify the URL. Sometimes, these redirect URLs take a perceivable amount of time to load, which is very irritating.

The link you followed when you clicked hasn't a direct link for years, decade afaik (they mangle so they can see what's followed). The page used to show the direct on the search text but now it shows some stand in for it - sometimes. You can see the direct link on the bottom of the screen when you hover - sometimes (and sometimes you see a mangled link). Sometimes the google link contains the original link in the cen…

I've just checked this again using a google account where search result pages are still following the old behavior: It looks like the 'href' attribute is the direct link, and the 'ping' attribute is the /url redirect link you are referring to. So it looks like it is actually sending me to the direct link, it just also requests /url at the same time in order to log the click. This means the user was not waiting for the logging/redirect request to come back.
Post reply on HN