A lot of sites have IP crawl restrictions, but add exceptions for Googlebot. Could Google or the crawled site legally do something when they find out?
Ask HN: Can I get in trouble for crawling using the Googlebot user agent?
1–10 of 53 posts
Re: Ask HN: Can I get in trouble for crawling using the Googlebot user agent?
#2[deleted]
Re: Ask HN: Can I get in trouble for crawling using the Googlebot user agent?
#3Sure, go join the realms of shady SEOs and malware, if I want to really stop you I'll know you're not coming from a Google IP range. https://www.incapsula.com/blog/was-that-really-a-google-bot-...
However, consider what your ultimate end game is, if it's a website you expect visitors to find through Google or the Play store, good luck once web masters start reporting your misbehaving "Googlebot" crawler.
Re: Ask HN: Can I get in trouble for crawling using the Googlebot user agent?
#4Maybe "Googlebot" is a trademark, or maybe you are violating the usage terms the crawled sites have put in place by masquerading yourself... Could you get in to trouble? _MAYBE_? Seems like a stretch in practice though. I've come across people doing this to sites i've been an admin of relatively often, and unless you're crawling with enough intensity to cause a DoS or doing something nefarious with the content, most site owners would maybe roll their eyes and move on.
Re: Ask HN: Can I get in trouble for crawling using the Googlebot user agent?
#5Depends on the jurisdiction. In the US, the answer is "you really don't want to find out".
In my home country, it's actually quite interesting: fraud usually requires (a) a lie (conveying wrong information with intent), and (b) a financial cost to the other party, and (c) a financial gain for you.
It's debatable at that level, already, because their loss is rather hard to quantify, and probably small. Plus, I believe your financial gain must be directly related to their cost.
And, finally, you actually have to lie to a human being. Lying to a machine doesn't qualify. There was a guy who earned some 5-digit Euros amount by producing fake bottles and feeding them into deposit machines–no crime!
Re: Ask HN: Can I get in trouble for crawling using the Googlebot user agent?
#6Are you only crawling or also scraping the website?
If the sites in question only add an exception for googlebot and not other crawlers (e.g. Yahoo, bing, etc.) I would say that it is against the site owner's consent.
However if the site owner adds this exception also for other crawlers, you could argue that the site owner's intent of only allowing certain crawlers has not been made explicit. In that case you'd have a chance against the claims from the site's owner.
On the other hand Google could possibly sue you for using the user-agent "Googlebot".
The important question here is: would they? If you stay under the radar no one -even the courts- would bother.
PS: I am only a law student, I am not familiar with any laws/regulations/precedents governing this specific issue. I think from the site owner's perspective it's a grey area. From google's perspective brands and ip are established concepts in law. This is a student's very personal opinion at first sight, take it with a grain of salt :).
Re: Ask HN: Can I get in trouble for crawling using the Googlebot user agent?
#7Sure, go join the realms of shady SEOs and malware, if I want to really stop you I'll know you're not coming from a Google IP range. https://www.incapsula.com/blog/was-that-really-a-google-bot-... However, consider what your ultimate end game is, if it's a website you expect visitors to find through Google or the Play store, good luck once web masters start reporting your misbehaving "Googlebot" crawler.
Unless you do it from a Google Cloud instance, that is.
Re: Ask HN: Can I get in trouble for crawling using the Googlebot user agent?
#8You won't get in trouble but if the site uses products from the like of Akamai (Bot Manager) or Shape Security then you'll probably be blocked.
Re: Ask HN: Can I get in trouble for crawling using the Googlebot user agent?
#9Sure, go join the realms of shady SEOs and malware, if I want to really stop you I'll know you're not coming from a Google IP range. https://www.incapsula.com/blog/was-that-really-a-google-bot-... However, consider what your ultimate end game is, if it's a website you expect visitors to find through Google or the Play store, good luck once web masters start reporting your misbehaving "Googlebot" crawler.
You know you have to be able to crawl your own or client sites using screaming frog or deep crawl as google to identify any crawl/ crawl budget issues.
Re: Ask HN: Can I get in trouble for crawling using the Googlebot user agent?
#10If you crawl a site, index it, and then use that for commercial purposes -- all while using Google's trademark to crawl -- yes, you'll probably get a letter from Google.
As for the site owner, it's on them to decide what to do with your traffic. HTTP is an open protocol and extensible. You could send almost anything in your request, as allowed by the protocol. The site owner has opened their service to the HTTP protocol and it's on them to decide what to do with your traffic.