Ask HN: Can I get in trouble for crawling using the Googlebot user agent? • Hacker News Board

Re: Ask HN: Can I get in trouble for crawling using the Googlebot user agent?

#51

Post by jdavis703 » Fri, Apr 21, 2017, 5:41 PM UTC

tomsmeding wrote: post #48

jdavis703 wrote: post #42

Earlier quoted context omitted.

It might be against the terms of service of the website you're crawling, which puts you in violation of the Computer Fraud and Abuse Act (i.e. you're considered to be "hacking" them).

Maybe that's called "hacking" in the relevant document, but I'd classify that as overly broad.

Well however you classify it doesn't matter, what matters is how the government classifies it. There's the tragic story of Aaron Swartz who was caught up in this non-sense: https://en.wikipedia.org/wiki/Aaron_Swartz.

Re: Ask HN: Can I get in trouble for crawling using the Googlebot user agent?

#52

Post by UnpossibleJim » Fri, Apr 21, 2017, 5:41 PM UTC

awinter-py wrote: post #14

I'm not a lawyer and this isn't legal advice; but my instinct is you won't get in trouble. Most important argument: the chrome user-agent contains the word 'mozilla'. Obviously (we argue) google isn't intending these to be accurate and instead are some kind of compatibility mark. Are you committing trademark violation? Given the nature of trademarks, it's not clear that you are. Are you misrepresenting yourself to th…

Speaking of law, and in the modern times this is in the theory of law, not practice, but intent is supposed to count. Now, that being said, the people at the level of the court systems are woefully ignorant of technology and IT more specifically. Intent seems to hold up only those rich enough to push very hard for it and prosecuting attorneys seem to get erections at the thought of putting someone in jail for even the perception of a cyber crime, these days, so.... 50/50 maybe =P

Re: Ask HN: Can I get in trouble for crawling using the Googlebot user agent?

#53

Post by terminalcommand » Fri, Apr 21, 2017, 9:23 PM UTC

jrvidal wrote: post #11

terminalcommand wrote: post #6

Are you only crawling or also scraping the website? If the sites in question only add an exception for googlebot and not other crawlers (e.g. Yahoo, bing, etc.) I would say that it is against the site owner's consent. However if the site owner adds this exception also for other crawlers, you could argue that the site owner's intent of only allowing certain crawlers has not been made explicit. In that case you'd have…

> On the other hand Google could possibly sue you for using the user-agent "Googlebot". Genuinely curious: on what basis? Can you trademark (or similar) a user-agent?

Google is a registered trademark, Googlebot may not be registered. However as it clearly is affiliated with Google, even if Google had not registered it they might pursue claims.

I am not familiar with U.S. law. If the courts in U.S. adopt a strictly formalistic approach on using names in a business context, then the results may be different.

However IMHO, Googlebot is clearly associated with Google and anyone who uses Googlebot as their User Agent is tricking the sites owner's into believing that the request was made from Google.

As for legal grounds, Google could sue on unfair competition.