Google's robots.txt • Hacker News Board

Re: Google's robots.txt

#61

Post by lesiki » Sun, Feb 23, 2014, 10:06 AM UTC

gergles wrote: post #31

http://www.google.com/baraza/en What a weird little product. It's like Yahoo Answers, but somehow with even less sorting or categorization.

'baraza' is Swahili for forum/meeting place. It was very much Yahoo Answers, targeted at the African market - here in Kenya, most people only have internet access via mobile connections, often using feature phones, hence the minimalistic stylesheet.

Baraza never really took off.

More about it here: http://whiteafrican.com/2010/10/05/google-baraza-qa-for-afri...

Re: Google's robots.txt

#62

Post by sfall » Sun, Feb 23, 2014, 10:32 AM UTC

yeukhon wrote: post #44

MichaelApproved wrote: post #41

Earlier quoted context omitted.

You're quoting "agreement" as if its literally in their robots file. It's not. They're telling the public that it does not have permission to crawl the site which try have the right to do. What is the problem with that?

What is the point of having such silly prohibition? It's silly because anyone can crawl it if they want, Facebook may block such DDoS attack, but why would they bother to put up such sign when they know it's useless?

my guess lawyers

Re: Google's robots.txt

#63

Post by ushi » Sun, Feb 23, 2014, 10:37 AM UTC

Houshalter wrote: post #26

ryanpetrich wrote: post #3

See also: http://www.google.com/humans.txt

This doesn't work if you are using HTTPS everywhere.

  % curl www.google.com/humans.txt
  Google is built by a large team of engineers, designers, researchers,
  robots, and others in many different sites across the globe. It is
  updated continuously, and built with more tools and technologies than
  we can shake a stick at. If you'd like to help us out, see
  google.com/jobs.

Re: Google's robots.txt

#64

Post by daGrevis » Sun, Feb 23, 2014, 11:12 AM UTC

dfc wrote: post #40

Houshalter wrote: post #26

Earlier quoted context omitted.

This doesn't work if you are using HTTPS everywhere.

Weird, it works for me. Iceweasel Aurora / HTTPS Everywhere 4.0-dev

I second this. Latest Chromium.

Re: Google's robots.txt

#65

Post by 0x0 » Sun, Feb 23, 2014, 11:23 AM UTC

Curious as to why someone sat down and added this line to that file:

  Allow: /maps?hq=http://maps.google.com/help/maps/directions/biking/mapleft.kml&ie=UTF8&ll=37.687624,-122.319717&spn=0.346132,0.727158&z=11&lci=bike&dirflg=b&f=d

Re: Google's robots.txt

#66

Post by riffraff » Sun, Feb 23, 2014, 11:28 AM UTC

khyh wrote: post #19

ryanpetrich wrote: post #3

See also: http://www.google.com/humans.txt

How did you find this?

FWIW, I keep finding it in my webserver logs.

Apparently some security scanner tests some php vulnerability based on fopen trying to load that.

Re: Google's robots.txt

#67

Post by ScottWhigham » Sun, Feb 23, 2014, 11:35 AM UTC

anonymfus wrote: post #56

agwa wrote: post #15

Earlier quoted context omitted.

I've found this GitHub project to be an invaluable resource for blocking bad bots: https://github.com/bluedragonz/bad-bot-blocker

>Unless your website is written in Russian or Chinese, you probably don't get any traffic from them. They mostly just waste bandwidth and consume resources. THIS is evil. You could use this argument for banning any new search engine.

What "new search engine" has actually generated actual revenue for any webmaster in the past ten years? You could argue DDG but that's the only one I can think of.

Re: Google's robots.txt

#68

Post by avar » Sun, Feb 23, 2014, 11:46 AM UTC

d99kris wrote: post #18

Glassdoor uses its for recruiting: http://www.glassdoor.com/robots.txt

I love the note at the end of that file about how they can't figure out their own web framework.

Re: Google's robots.txt

#69

Post by usrusr » Sun, Feb 23, 2014, 12:11 PM UTC

johnvschmitt wrote: post #36

wupiass wrote: post #16

facebook's... https://www.facebook.com/robots.txt

That really blows my mind. I mean, how can they say that's any kind of "agreement"? I someone writes a curl/wget script wrapper & points it to the top 10 websites, they don't enter into any kind of written contract or agreement.

It's probably just their way of explaining how those user-agents that do not get the catchall Disallow: / treatment got into that robots.txt file. Also, including some lawyerisms might be quite effective at reminding upstart scrapers that faking the googlebot UA would be even less cool than simply ignoring robots.txt.

Re: Google's robots.txt

#70

Post by MichaelApproved » Sun, Feb 23, 2014, 12:16 PM UTC

yeukhon wrote: post #44

MichaelApproved wrote: post #41

Earlier quoted context omitted.

You're quoting "agreement" as if its literally in their robots file. It's not. They're telling the public that it does not have permission to crawl the site which try have the right to do. What is the problem with that?

What is the point of having such silly prohibition? It's silly because anyone can crawl it if they want, Facebook may block such DDoS attack, but why would they bother to put up such sign when they know it's useless?

They have such a large network, anything they could do to prevent unwanted crawling is probably helpful.