Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google's robots.txt

google.com

Post reply on HN
Print view View original

41–50 of 94 posts

Re: Google's robots.txt

#41

Post by MichaelApproved »

johnvschmitt wrote: post #36
wupiass wrote: post #16

facebook's... https://www.facebook.com/robots.txt

That really blows my mind. I mean, how can they say that's any kind of "agreement"? I someone writes a curl/wget script wrapper & points it to the top 10 websites, they don't enter into any kind of written contract or agreement.

You're quoting "agreement" as if its literally in their robots file. It's not.

They're telling the public that it does not have permission to crawl the site which try have the right to do. What is the problem with that?

Re: Google's robots.txt

#42

Post by pjscott »

saltysugar wrote: post #39
blossoms wrote: post #8

Unrelated but it looks like www.aol.com's robots.txt is served as text/html http://www.aol.com/robots.txt Is this a common mistake?

Yes, from the server side's mishandling of TXT extension. Probably the server put the MIME type in the HTTP header as "HTML" instead of TXT, and the browser renders the page as such.

From the headers:

    Content-Type: text/html;charset=UTF-8
It also tries to set no less than four cookies.

Re: Google's robots.txt

#44

Post by yeukhon »

MichaelApproved wrote: post #41
johnvschmitt wrote: post #36

Earlier quoted context omitted.

That really blows my mind. I mean, how can they say that's any kind of "agreement"? I someone writes a curl/wget script wrapper & points it to the top 10 websites, they don't enter into any kind of written contract or agreement.

You're quoting "agreement" as if its literally in their robots file. It's not. They're telling the public that it does not have permission to crawl the site which try have the right to do. What is the problem with that?

What is the point of having such silly prohibition? It's silly because anyone can crawl it if they want, Facebook may block such DDoS attack, but why would they bother to put up such sign when they know it's useless?

Re: Google's robots.txt

#47

Post by tonyedgecombe »

judk wrote: post #11
mkonecny wrote: post #10

Earlier quoted context omitted.

It just issues a "HTTP/1.1 302 Moved Temporarily" directed to their homepage. Requesting an invalid file such as "robots.txtsdfa32r523" has the same effect, so they probably don't have a robots file at all.

Huh? No, it is a regular robots.txt file

It redirects requests from the UK.

Re: Google's robots.txt

#48

Post by bmelton »

agwa wrote: post #15
russellbeattie wrote: post #12

Earlier quoted context omitted.

Heh, if only Yandex and Baidu respected robots.txt.

I've found this GitHub project to be an invaluable resource for blocking bad bots: https://github.com/bluedragonz/bad-bot-blocker

That's amazing. Know of anything like it for Nginx?

Re: Google's robots.txt

#49

Post by declan »

johnvschmitt wrote: post #36
wupiass wrote: post #16

facebook's... https://www.facebook.com/robots.txt

That really blows my mind. I mean, how can they say that's any kind of "agreement"? I someone writes a curl/wget script wrapper & points it to the top 10 websites, they don't enter into any kind of written contract or agreement.

The operator of a crawler doesn't need to sign an agreement for the prohibition to be enforceable. See eBay v. Bidder's Edge. This was 14 years ago, folks.
Post reply on HN