Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Here is the robots.txt of Google

google.com

Post reply on HN
Print view View original

21–30 of 34 posts

Re: Here is the robots.txt of Google

#21

Post by jedc »

sp332 wrote: post #8

What is http://google.com/unclesam ? Edit: must have just mistyped it, works fine.

Since nearly the very beginning of Google proper (aka, not BackRub) they've had a separate search for government sites and for Linux. And also Universities... I vaguely remember both a University of Michigan and a Stanford search. (Larry Page was an undergrad at UMich).

Re: Here is the robots.txt of Google

#24

Post by jacquesm »

Tichy wrote: post #20
pedalpete wrote: post #17

Earlier quoted context omitted.

Though you could end up with a load of URLs, titles, and a summary text, crawling google won't really get you to the important data in building a search engine. No follow-on links, only a subset of the text, no referrals, etc. Love the 'paradox' of the infinite spider loop, hadn't considered that before.

Crawling groups would be interesting. What other archives of usenet are there?

also, google had no problem crawling groups when it was still called dejanews.

Re: Here is the robots.txt of Google

#25

Post by mattmaroon »

jacquesm wrote: post #11
sp332 wrote: post #8

What is http://google.com/unclesam ? Edit: must have just mistyped it, works fine.

it searches the .gov domain, because 'uncle sam' employees can't remember to put site:.gov in front of their searches ? Think of it as a shortcut.

I'd guess over 90% of Google users (and maybe 99%) don't know about "site:" Hell, most people can't even come up with a reasonable definition for "search engine".

Re: Here is the robots.txt of Google

#26

Post by anarcticpuffin »

pedalpete wrote: post #17
brk wrote: post #3

Earlier quoted context omitted.

Yes, but... For the most part "Google" is a condensed version of the web. In theory, you could "spider" a single site (google.com) and build your own search database without having to go out and crawl the web at large. It's an odd paradox, but I think one search site crawling another search site is not a good idea. And there is probably an infinite spider loop hiding in that process somewhere.

Though you could end up with a load of URLs, titles, and a summary text, crawling google won't really get you to the important data in building a search engine. No follow-on links, only a subset of the text, no referrals, etc. Love the 'paradox' of the infinite spider loop, hadn't considered that before.

Most spiders implement a maximum depth in their search though. So you'd really just get an inefficient crawl, not an infinite one.

Re: Here is the robots.txt of Google

#28

Post by digitallogic »

brk wrote: post #3
jacquesm wrote: post #2

I always figured that the company that spiders everybody elses content should have a more relaxed policy towards being spidered itself. After all, google is datamining the web on an ongoing basis, in return it should willingly consent to being mined in return.

Yes, but... For the most part "Google" is a condensed version of the web. In theory, you could "spider" a single site (google.com) and build your own search database without having to go out and crawl the web at large. It's an odd paradox, but I think one search site crawling another search site is not a good idea. And there is probably an infinite spider loop hiding in that process somewhere.

Actually a lot of search sites reguarlly pull down large searchs result sets from eachother to do comparisions to their own, though of course this isn't tecnically 'crawling' in the follow every link sense.

Re: Here is the robots.txt of Google

#29

Post by twoz »

jdrock wrote: post #27
twoz wrote: post #23

Does Yahoo! not have a robots.txt file? http://yahoo.com/robots.txt Sorry, the page you requested was not found.

If that page doesn't exist, then according to specification, they don't and you can crawl any page.

Update: Some Yahoo! subdomains have 'em.

Yes:

    http://search.yahoo.com/robots.txt
    http://groups.yahoo.com/robots.txt
    http://realestate.yahoo.com/robots.txt
No:

    http://maps.yahoo.com/robots.txt
    http://omg.yahoo.com/robots.txt
Post reply on HN