What is http://google.com/unclesam ? Edit: must have just mistyped it, works fine.
it searches the .gov domain, because 'uncle sam' employees can't remember to put site:.gov in front of their searches ?
Think of it as a shortcut.
Here is the robots.txt of Google
11–20 of 34 posts
Re: Here is the robots.txt of Google
#12Earlier quoted context omitted.
Not like anyone writing a spider has to obey the robots.txt file.
Sure, and their IP will be blocked by Google some time around the 10th request.
Next time you're looking at a PC infected with malware have a look at the network traffic using a sniffer, chances are pretty good that you'll see searches to google for the weirdest of terms. Apparently this is to get around the limitation that you mention. I'm assuming the results of such searches will be 'mailed home' through some kind of dead-drop.
Re: Here is the robots.txt of Google
#13We actually have a surprising number of customers come to us at 80legs wanting to crawl google search results. I don't think most of them are trying to reverse engineer google or anything like that. Most probably just want a fast way to find relevant topics to crawl.
They are disappointed when they learn we obey robots.txt, so we have them manually do searches to pull out seed lists for their 80legs crawls. It's a pain, but there's not really a way around it within the rules.
Re: Here is the robots.txt of Google
#14What is http://google.com/unclesam ? Edit: must have just mistyped it, works fine.
this is even weirder..
http://www.google.com/microsoft
Re: Here is the robots.txt of Google
#15What is http://google.com/unclesam ? Edit: must have just mistyped it, works fine.
Re: Here is the robots.txt of Google
#16Earlier quoted context omitted.
Sure, and their IP will be blocked by Google some time around the 10th request.
Next time you're looking at a PC infected with malware have a look at the network traffic using a sniffer, chances are pretty good that you'll see searches to google for the weirdest of terms. Apparently this is to get around the limitation that you mention. I'm assuming the results of such searches will be 'mailed home' through some kind of dead-drop.
Sure, but responding to the parent, that's why people don't crawl Google, not the Robots.txt.
And incidentally, some of those searches are to find forms that it can stuff links into. My sites are constantly getting hit by botnets searching for Drupal comment forms. Luckily it uses a quirky URL format that's easy for mod_security to block.
Re: Here is the robots.txt of Google
#17I always figured that the company that spiders everybody elses content should have a more relaxed policy towards being spidered itself. After all, google is datamining the web on an ongoing basis, in return it should willingly consent to being mined in return.
Yes, but... For the most part "Google" is a condensed version of the web. In theory, you could "spider" a single site (google.com) and build your own search database without having to go out and crawl the web at large. It's an odd paradox, but I think one search site crawling another search site is not a good idea. And there is probably an infinite spider loop hiding in that process somewhere.
Though you could end up with a load of URLs, titles, and a summary text, crawling google won't really get you to the important data in building a search engine.
No follow-on links, only a subset of the text, no referrals, etc.
Love the 'paradox' of the infinite spider loop, hadn't considered that before.
Re: Here is the robots.txt of Google
#18Re: Here is the robots.txt of Google
#19What is http://google.com/unclesam ? Edit: must have just mistyped it, works fine.
this is even weirder.. http://www.google.com/microsoft
I would guess it's the compliment of:
Re: Here is the robots.txt of Google
#20Earlier quoted context omitted.
Yes, but... For the most part "Google" is a condensed version of the web. In theory, you could "spider" a single site (google.com) and build your own search database without having to go out and crawl the web at large. It's an odd paradox, but I think one search site crawling another search site is not a good idea. And there is probably an infinite spider loop hiding in that process somewhere.
Though you could end up with a load of URLs, titles, and a summary text, crawling google won't really get you to the important data in building a search engine. No follow-on links, only a subset of the text, no referrals, etc. Love the 'paradox' of the infinite spider loop, hadn't considered that before.
Crawling groups would be interesting. What other archives of usenet are there?