21–30 of 53 posts
Earlier quoted context omitted.
Unless you do it from a Google Cloud instance, that is.
>Unless you do it from a Google Cloud instance, that is. What's the reverse DNS for Google Cloud IPs? Google says to check that Googlebot's IP resolves to either a .google.com or .googlebot.com domain. https://support.google.com/webmasters/answer/80553?hl=en
I'm not a lawyer and this isn't legal advice; but my instinct is you won't get in trouble. Most important argument: the chrome user-agent contains the word 'mozilla'. Obviously (we argue) google isn't intending these to be accurate and instead are some kind of compatibility mark. Are you committing trademark violation? Given the nature of trademarks, it's not clear that you are. Are you misrepresenting yourself to th…
What is Google's slogan these days?
I'm not a lawyer and this isn't legal advice; but my instinct is you won't get in trouble. Most important argument: the chrome user-agent contains the word 'mozilla'. Obviously (we argue) google isn't intending these to be accurate and instead are some kind of compatibility mark. Are you committing trademark violation? Given the nature of trademarks, it's not clear that you are. Are you misrepresenting yourself to th…
What is Google's slogan these days?
If you crawl a site, index it, and then use that for commercial purposes -- all while using Google's trademark to crawl -- yes, you'll probably get a letter from Google. As for the site owner, it's on them to decide what to do with your traffic. HTTP is an open protocol and extensible. You could send almost anything in your request, as allowed by the protocol. The site owner has opened their service to the HTTP proto…
How would you get a letter from Google if you are never scraping google's sites? They would never know?
Earlier quoted context omitted.
What is Google's slogan these days?
'stayin alive, stayin alive'? They've done surprisingly well transitioning to mobile & promoted their android guy to CEO, but haven't been able to diversify revenue away from ads. And people are starting to hate ads.
If you crawl a site, index it, and then use that for commercial purposes -- all while using Google's trademark to crawl -- yes, you'll probably get a letter from Google. As for the site owner, it's on them to decide what to do with your traffic. HTTP is an open protocol and extensible. You could send almost anything in your request, as allowed by the protocol. The site owner has opened their service to the HTTP proto…
How would you get a letter from Google if you are never scraping google's sites? They would never know?
How would Google know? They would start by setting up fictitious websites which would be seemingly unaffiliated to them. If your crawler was to hit the site, you would thus reveal yourself. I wouldn't at all be surprised that Google would have this kind of "honey pot" of sorts sitting out there watching for web crawlers (rogue or otherwise).
Google likely also has business partner relationships with big content producers, which I'm sure they are able to get reports back from regarding their crawling -- to ensure that Google is correctly finding all the content which the site owners want them to.
As an aside, I used to run such a honeypot website. Web crawler behavior is fascinating. I loved being able to find, detect and classify various forms of web crawlers. Some which followed robots.txt, some that didn't, some that went directly to robots.txt and then scraped the pages which were meant to be excluded. I wish I had kept the project going and formalized the results.
Earlier quoted context omitted.
How would you get a letter from Google if you are never scraping google's sites? They would never know?
Even if they do know, what do they have to do with it? Does google have a legal claim to their user-agent string exclusively?
Earlier quoted context omitted.
Unless you do it from a Google Cloud instance, that is.
>Unless you do it from a Google Cloud instance, that is. What's the reverse DNS for Google Cloud IPs? Google says to check that Googlebot's IP resolves to either a .google.com or .googlebot.com domain. https://support.google.com/webmasters/answer/80553?hl=en
Earlier quoted context omitted.
Unless you do it from a Google Cloud instance, that is.
>Unless you do it from a Google Cloud instance, that is. What's the reverse DNS for Google Cloud IPs? Google says to check that Googlebot's IP resolves to either a .google.com or .googlebot.com domain. https://support.google.com/webmasters/answer/80553?hl=en
Like this: http://i.imgur.com/ocR54Yq.jpg
