Since when has it ever been illegal to claim you're someone or something that you're not on the Internet? This is legal without question.
Ask HN: Can I get in trouble for crawling using the Googlebot user agent?
41–50 of 53 posts
Re: Ask HN: Can I get in trouble for crawling using the Googlebot user agent?
#42Since when has it ever been illegal to claim you're someone or something that you're not on the Internet? This is legal without question.
It might be against the terms of service of the website you're crawling, which puts you in violation of the Computer Fraud and Abuse Act (i.e. you're considered to be "hacking" them).
Re: Ask HN: Can I get in trouble for crawling using the Googlebot user agent?
#43I feel like it's a good thing to maintain a certain level of professional ethics, and, while it depends on the specifics of the situation, I'd suggest that falsely claiming to third parties be something you aren't in order to do something they don't want you to do generally falls short of that ethical bar. Say your bot misbehaves and effectively starts DOSing a site with a whole lot of pages, like a small Reddit clon…
Not to detract from your point, but blindly blocking a User Agent because of a bad actor and losing money is not a good solution.
A more robust solution can be coded using information from Google: https://support.google.com/webmasters/answer/80553?hl=en
Re: Ask HN: Can I get in trouble for crawling using the Googlebot user agent?
#44I feel like it's a good thing to maintain a certain level of professional ethics, and, while it depends on the specifics of the situation, I'd suggest that falsely claiming to third parties be something you aren't in order to do something they don't want you to do generally falls short of that ethical bar. Say your bot misbehaves and effectively starts DOSing a site with a whole lot of pages, like a small Reddit clon…
I would hope the people at Reddit are smart enough to check an IP and not just a user-agent.
Re: Ask HN: Can I get in trouble for crawling using the Googlebot user agent?
#45Since when has it ever been illegal to claim you're someone or something that you're not on the Internet? This is legal without question.
1) It's illegal to impersonate law enforcement online.
2) It's illegal to commit fraud by pretending to be someone else online (when, for example, using someone else's credit card or opening a bank account in their name).
3) You can be sued for things that are totally legal, such as blemishing Google's brand my impersonate them and doing things that annoy people.
Re: Ask HN: Can I get in trouble for crawling using the Googlebot user agent?
#46I'm not a lawyer and this isn't legal advice; but my instinct is you won't get in trouble. Most important argument: the chrome user-agent contains the word 'mozilla'. Obviously (we argue) google isn't intending these to be accurate and instead are some kind of compatibility mark. Are you committing trademark violation? Given the nature of trademarks, it's not clear that you are. Are you misrepresenting yourself to th…
What is Google's slogan these days?
Look on my Works, ye Mighty, and despair!
Re: Ask HN: Can I get in trouble for crawling using the Googlebot user agent?
#47May violate a site's TOS... but I don't think you'd ever get in any real trouble... most you'd get is a cease and desist letter... have to waste some time with lawyers... But I think it's on them to block you at the IP level if you are violating the TOS / causing them grief. And look... if a crawl causes them grief then they really need to invest more in DevOps. (Please do what you can to encourage more companies to invest in DevOps!)
"I left my door unlocked and told my friends they could use my living room, but then they put their feet up on my coffee table... Not cool, man!" Pretty much the equivalent situation.
Re: Ask HN: Can I get in trouble for crawling using the Googlebot user agent?
#48Since when has it ever been illegal to claim you're someone or something that you're not on the Internet? This is legal without question.
It might be against the terms of service of the website you're crawling, which puts you in violation of the Computer Fraud and Abuse Act (i.e. you're considered to be "hacking" them).
Maybe that's called "hacking" in the relevant document, but I'd classify that as overly broad.
Re: Ask HN: Can I get in trouble for crawling using the Googlebot user agent?
#49I wonder if most major sites that whitelist Googlebot also have exceptions for Slurp, Bingbot, and other major search engine bots. If not, then it would be interesting to know how these other companies deal with it, or if they just politely back off.
Re: Ask HN: Can I get in trouble for crawling using the Googlebot user agent?
#50If I were going to do this the last place I'd ask this question is HackerNews. But that's just me.