Ask HN: Can I get in trouble for crawling using the Googlebot user agent?

Post by jrvidal » Fri, Apr 21, 2017, 4:10 PM UTC

terminalcommand wrote: post #6

Are you only crawling or also scraping the website? If the sites in question only add an exception for googlebot and not other crawlers (e.g. Yahoo, bing, etc.) I would say that it is against the site owner's consent. However if the site owner adds this exception also for other crawlers, you could argue that the site owner's intent of only allowing certain crawlers has not been made explicit. In that case you'd have…

Post by zackify » Fri, Apr 21, 2017, 4:10 PM UTC

taftster wrote: post #10

If you crawl a site, index it, and then use that for commercial purposes -- all while using Google's trademark to crawl -- yes, you'll probably get a letter from Google. As for the site owner, it's on them to decide what to do with your traffic. HTTP is an open protocol and extensible. You could send almost anything in your request, as allowed by the protocol. The site owner has opened their service to the HTTP proto…

Post by mootothemax » Fri, Apr 21, 2017, 4:12 PM UTC

lend000 wrote: post #7

cube00 wrote: post #3

Sure, go join the realms of shady SEOs and malware, if I want to really stop you I'll know you're not coming from a Google IP range. https://www.incapsula.com/blog/was-that-really-a-google-bot-... However, consider what your ultimate end game is, if it's a website you expect visitors to find through Google or the Play store, good luck once web masters start reporting your misbehaving "Googlebot" crawler.

Unless you do it from a Google Cloud instance, that is.

Post by fbomb » Fri, Apr 21, 2017, 4:18 PM UTC

Post by jimktrains2 » Fri, Apr 21, 2017, 4:19 PM UTC

fbomb wrote: post #17

I handle a number of sites that require a login but need Google to index their content. I verify that Googlebot requests are actually coming from a domain owned by Google. I can't imagine that I'm the only one doing that.