Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

google.com/goto: Google's anti-scraping update

autom.dev

Post reply on HN
Print view View original

261–270 of 545 posts

Re: google.com/goto: Google's anti-scraping update

#261

Post by seba_dos1 »

fc417fc802 wrote: post #230
rkagerer wrote: post #158

I did an interview with Google around 20 years ago, where they posed a challenge involving tracking which specific search results people click. It's obvious in hindsight the solution required rewriting all the urls to redirect through their servers. Note this was in the days before they already did so as a matter of course. I failed to gain traction on the problem, because to me the very idea of doing such a thing wa…

The right click copy thing annoys me to no end.

I switched to DuckDuckGo around 2018 or so, which was when I realized that Google's results quality has deteriorated so much that I won't lose anything of value by doing so. This thread is how I learn about the atrocities that Google Search is committing these days.

Which is an elaborate way to say: you don't have to live like this.

Re: google.com/goto: Google's anti-scraping update

#262

Post by FerretFred »

aucisson_masque wrote: post #123

So why are we angry about that ? I mean the end result for the users are exactly the same, it matters only for bots. Google have such a (justified) bad reputation that whatever they do, people assume it’s entishification. I don’t believed it is on that matter.

Personally, I put a lot of effort into writing my blog posts, some of which have been really well viewed by humans. While mine is effectively a "hobbyist" site, it's nice when readers look at other blog posts as a result of reading the original one. Google's approach ensures that readers get a very narrow view of my content, so everybody expect Google loses out.

Re: google.com/goto: Google's anti-scraping update

#263

Post by chadgpt3 »

ta8903 wrote: post #254
yokoprime wrote: post #222

Earlier quoted context omitted.

What does israel have to do with this?

According to OP >It would be delusional to expect any sort of press and related (like search engine or aggregation) freedom, or to try to compare this to anything in any other developed country. Israel is responsible for killing more journalists than any other country. Google has profited by providing services to Israel in the ongoing genocide. https://cpj.org/special-reports/record-129-press-members-kil... https://a…

It only counts when the country is our declared enemy. When our ally does a holocaust, it isn't a holocaust, it's just anti-terrorism operations.

Re: google.com/goto: Google's anti-scraping update

#264

Post by nairboon »

JsonDemWitOster wrote: post #114

While a lot of people are concerned with local model performance, I wonder how feasible is it now to run a local indexed web search? Surely running an old school Google is possible with the beefy AI rigs today. I know the problem will be crawling which would be bottlenecked by the ISP but I use Google to search SO, Wikipedia, programming language docs, Github issues, and AWS docs. I think a feasible workflow would be…

It's still kind of an open problem. There are partial solutions, but not yet really an integrated one. There is Hister: https://hister.org/ that builds some sort of drive-by index of what you're browsing anyway. And there are "true p2p" solutions like YaCy: https://yacy.net/ but it takes ages to crawl the open web (and lots of storage). There's still a lot of optimization to do in this space.

Re: google.com/goto: Google's anti-scraping update

#265

Post by nairboon »

Cakez0r wrote: post #178
Jskewel wrote: post #152

Earlier quoted context omitted.

Impossible. The majority of websites firewall automated crawler traffic (because of the rise of the bots), only making exceptions for the largest search engines. There is no possibility of starting a new crawler.

It would be interesting to see a decentralised, residential collective that builds and publishes an index. There are surely enough interested people on HN alone that would be willing to run software at home to scrape a small slice of the internet.

Something like YaCy?

Re: google.com/goto: Google's anti-scraping update

#266

Post by kolinko »

tyrabound wrote: post #234
rkagerer wrote: post #161

Earlier quoted context omitted.

surprising to see how much they have stripped from our view Don't forget cached pages.

You may not even appreciate the full extent of it. Long ago now, when Google took off and started dominating because it was “the best results” people didn’t realize that even though they were the best results, they were not actually good results. To be the best, you just have to be notably better than your nearest competitor, you don’t actually have to be good or even decent, just noticeably better. Today Google real…

A nice story, but I don’t think there really was a time where closed source models’ development stalled really.

Re: google.com/goto: Google's anti-scraping update

#268

Post by Lvl999Noob »

stickfigure wrote: post #116
Lvl999Noob wrote: post #46

I... Don't see it? It's the result page right? I just search some random string on Google and the results are all direct URLs. Do they get resolved via javascript after page load and replaced automatically? Or am I looking at something else?

I was also super confused because it doesn't do this if you are logged in (to google). I ran the same search in an incognito window and it showed the /goto links.

Thanks! This was it. Saw the /goto links in private window.

Re: google.com/goto: Google's anti-scraping update

#270

Post by kolinko »

marttt wrote: post #246
lubujackson wrote: post #82

As much as I am sad that Google died like 15 years ago, I am past the mourning phase. That was when they announced they were shifting from returning websites to "returning answers" and it has been a long slide into shittification I do enjoy using their free AI. For actual web search I actually like using Yandex. It reminds me of old Google, returning reasonable results and much less "shaping results to please our cor…

This is interesting. Do you consider Yandex absolutely safe to use (wrt their possible connections with the Russian government, and current war in Ukraine)? Asking this as a geeky end-user from Estonia -- I wonder if I would be considered "suspicios" by local internet service providers, our govt/police etc if I did a lot of searching via Yandex instead of e.g. Google these days. (For us in Estonia, I suppose "everyth…

I’d expect everything that you type into Yandex that can be of use to the Russia will be used by Russia - they will nit care about hurting you.

Just like Russia doesn’t care about online criminals being located within their borders - as long as they target outside.

While this may be true to some extent of all the countries, Russia is the one to be clearly against Estonia, Baltics and eastern europeans.

Post reply on HN