Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Only Google is really allowed to crawl the web

knuckleheads.club

Post reply on HN
Print view View original

11–20 of 365 posts

Re: Only Google is really allowed to crawl the web

#11

Post by jrockway »

I think there are plenty of other people crawling the web. There's Common Crawl, there's the Wayback machine... it's not just Google. Then there is a very long tail of crawlers that show up in the logs for my small-potatoes personal website. Whatever they're doing, they seem to be existing in peace, at the very least.

To some extent, I agree with this site that people are nicer to Google than other crawlers. That's because the crawl consumes their resources but provides benefits -- you show up on Google, the only search engine people actually use. But at the same time, they are happy to drag Google in front of Congress for some general abuse, so... maybe there is actually a little bit of balance there.

Re: Only Google is really allowed to crawl the web

#12

Post by tyingq »

The bigger problem, to me, is not around crawling. It's the asymmetrical power Google has after crawling.

Google is obviously on a mission to keep people on Google owned properties. So, they take what they crawl and find a way to present that to the end user without anyone needing to visit the place that data came from.

Airlines are a good example. If you search for flight status for a particular flight, Google presents that flight status in a box. As an end user, that's great. However, that sort of search used to (most times) lead to a visit to the airline web site.

The airline web site could then present things Google can't do. Like "hey, we see you haven't checked in yet" or "TSA wait times are longer than usual" or "We have a more-legroom seat upgrade if you want it".

Google took those eyeballs away. Okay, fine, that's their choice. But they don't give anything back, which removes incentives from the actual source to do things better.

You see this recently with Wikipedia. Google's widgets have been reducing traffic to Wikipedia pretty dramatically. Enough so that Wikipedia is now pushing back with a product that the Googles of the world will have to pay for.

In short, I don't think the crawler is the problem. And I don't think Google will realize what the problem is until they start accidentally killing off large swaths of the actual sources of this content by taking the audience away.

Re: Only Google is really allowed to crawl the web

#13

Post by calimac »

henriquez wrote: post #2

I'd like to see some data on their claim that website operators are giving googlebot special privileges. As far as I can tell it would be a huge pain in the ass to block crawler bots from my servers, not that I've tried. I have some weird pages that tend to get crawlers caught in infinite loops, and I try to give them hints with robots.txt but most of the bots don't even respect robots.txt. If I actually wanted to re…

The studies and data to support their claim is in the first paragraph of the article you "read" before posting the question.

Re: Only Google is really allowed to crawl the web

#14

Post by Zelphyr »

wunderflix wrote: post #10

Even that won't change much. There is no way Google can be out-googled by other search engines because of its market dominance: more traffic means more clicks, more clicks mean better search results, better search results will drive more traffic. I try bing and DDG for a week or so every 6 months. I always switch back to google eventually because the results are so much better. Google can only be disrupted if somethi…

I've had the exact opposite reaction to the comparison between Google and DuckDuckGo. I use the latter daily and only rarely revert to Google. Even then I usually don't find the results to be any better and often find them to be worse.

In my estimation, Google's search results have significantly declined in recent years.

Re: Only Google is really allowed to crawl the web

#15

Post by xtracto »

dleslie wrote: post #5

The idea of a public cache available to anyone who wishes to index it is ... kind of compelling. If it was the only indexer allowed, and it was publically governed, then enforcing changes to regulation would be a lot more straightforward. Imagine if indexing public social media profiles was deemed unacceptable, and within days that content disappeared from all search engines. I don't think it'll ever happen, but it's…

That would be a very cool use case for something like STORJ or IPFS.

Re: Only Google is really allowed to crawl the web

#16

Post by Lammy »

henriquez wrote: post #2

I'd like to see some data on their claim that website operators are giving googlebot special privileges. As far as I can tell it would be a huge pain in the ass to block crawler bots from my servers, not that I've tried. I have some weird pages that tend to get crawlers caught in infinite loops, and I try to give them hints with robots.txt but most of the bots don't even respect robots.txt. If I actually wanted to re…

Spoofing your user-agent as googlebot is a common way to bypass paywalls, is (was?) a way to read Quora without creating an account, etc. Publishers obviously need to send their page/article to Google if they want it to be indexed but may not want to send the same page content to a normal user: https://www.256kilobytes.com/content/show/1934/spoofing-your...

This was common even back in the mid-2000s:

https://www.avivadirectory.com/bethebot/

https://developers.google.com/search/blog/2006/09/how-to-ver...

Re: Only Google is really allowed to crawl the web

#17

Post by soheil »

henriquez wrote: post #2

I'd like to see some data on their claim that website operators are giving googlebot special privileges. As far as I can tell it would be a huge pain in the ass to block crawler bots from my servers, not that I've tried. I have some weird pages that tend to get crawlers caught in infinite loops, and I try to give them hints with robots.txt but most of the bots don't even respect robots.txt. If I actually wanted to re…

It's hilarious to think there exists people who think googlebot does not get special treatment from website operators. Here is an experiment you can do in a jiffy, write a script that crawls any major website and see how many URL fetches it takes before your IP gets blocked.

Googlebot has a range of IP addresses that it publicly announces so websites can whitelist them.

Re: Only Google is really allowed to crawl the web

#18

Post by suicas »

henriquez wrote: post #2

I'd like to see some data on their claim that website operators are giving googlebot special privileges. As far as I can tell it would be a huge pain in the ass to block crawler bots from my servers, not that I've tried. I have some weird pages that tend to get crawlers caught in infinite loops, and I try to give them hints with robots.txt but most of the bots don't even respect robots.txt. If I actually wanted to re…

A company I worked for ~7 years ago ran its own focused web crawler (fetching ~10-100m pages per month, targeting certain sections of the web).

There were a surprising number of sites out there that explicitly blocked access to anyone but Google/Bing at the time.

We'd also get a dozen complaints or so a month from sites we'd crawled. Mostly upset about us using up their bandwidth, and telling us that only Google was allowed to crawl them (though having no robots.txt configured to say so).

Re: Only Google is really allowed to crawl the web

#19

Post by jinseokim »

This has been submitted to HN quite a few times.

https://news.ycombinator.com/item?id=25426662 (Most comments; 11 comments)

https://news.ycombinator.com/item?id=25417067 (3 comments)

https://news.ycombinator.com/item?id=25546867 (Most recent; 89 days ago)

https://news.ycombinator.com/item?id=25543859

https://news.ycombinator.com/item?id=25424852

Re: Only Google is really allowed to crawl the web

#20

Post by wunderflix »

Zelphyr wrote: post #14
wunderflix wrote: post #10

Even that won't change much. There is no way Google can be out-googled by other search engines because of its market dominance: more traffic means more clicks, more clicks mean better search results, better search results will drive more traffic. I try bing and DDG for a week or so every 6 months. I always switch back to google eventually because the results are so much better. Google can only be disrupted if somethi…

I've had the exact opposite reaction to the comparison between Google and DuckDuckGo. I use the latter daily and only rarely revert to Google. Even then I usually don't find the results to be any better and often find them to be worse. In my estimation, Google's search results have significantly declined in recent years.

Ha, maybe I should give it a try again :) My 6 months period is almost over again.
Post reply on HN