Aren't /search and /Search considered two different directories when it comes to robots.txt?
Bing search results showing up in Google
11–20 of 131 posts
Re: Bing search results showing up in Google
#12Earlier quoted context omitted.
Why would someone with no authority over your site linking to your site override your robots.txt? Edit: I didn't downvote you, but I think you're wrong because it makes no sense - if I link to your site that shouldn't give search engines a free pass to ignore your wishes and do whatever they want with your content.
I understand a robots.txt "Disallow: /foo" to mean that it must not crawl that page, i.e. look at the links _inside_ that page.
I've always interpreted it to mean they're explicitly not allowed to touch it - no exceptions (unless you actually specified exceptions for them which you can see on the 3rd last example at http://www.robotstxt.org/robotstxt.html).
Re: Bing search results showing up in Google
#13With respect to the author, the conclusion here is very flawed.
If you search for Bing in Google, you get Bing all over page 1. If you search for Google in Bing, you get Google all over page 1. That's not the result of Google capturing click stream data from Google Chrome and copying Bing's results, nor is it the result of Microsoft capturing click stream data from IE8 and copying Google's results. That's just the nature of indexing.
As for robots.txt disallowing those URLs, there is no standard for robots.txt behavior. I have observed some user agents treat it as case insensitive, and others treat it as case sensitive.
Honestly, this isn't even in the same ballpark as the Google accusations made earlier this week, and it smacks of just looking for things to accuse Google of in response to the "Binggate" (ugh, I typed it) drama. Can't we go back to more productive things?
Re: Bing search results showing up in Google
#14Dear Microsoft, case-sensitivity is important. [1] m.bing.com/robots.txt says "/search", not "/Search". All of the crawled [2] urls are "/Search" or "/~/search". Also, wap.bing.com/robots.txt explicitly "Allow:"s several search pages, which are indexed by google. [1] EDIT: Case insensitivity is often important. Above comment notes that some robots are case-insensitive. I suspect Google is not, based on the results. […
Small correction: All crawled urls are "/Search" or "/~/search". Note that www.bing.com/search/ is indexed (link was found on another page), but not crawled (result has no snippet).
Re: Bing search results showing up in Google
#15>never mind that Bing only used its toolbar as a url discovery device, not to 'copy search results'
Yeah, they just happened to discover high quality urls on google. What are the chances?
Re: Bing search results showing up in Google
#16robots.txt applies to the source of the links, not the target. So if, say, http://www.paulgraham.com/ links to http://m.bing.com/search , then http://m.bing.com/robots.txt does not apply to that. EDIT: If you think this is wrong, please explain it instead of just downvoting me, because I think it is pretty unfair that I lose karma for explaining my interpretation.
I think you're confusing robots.txt with the nofollow attribute value: http://en.wikipedia.org/wiki/Nofollow
Re: Bing search results showing up in Google
#17Earlier quoted context omitted.
I understand a robots.txt "Disallow: /foo" to mean that it must not crawl that page, i.e. look at the links _inside_ that page.
I've always interpreted it to mean they're explicitly not allowed to touch it - no exceptions (unless you actually specified exceptions for them which you can see on the 3rd last example at http://www.robotstxt.org/robotstxt.html ).
One counter intuitive thing. They are allowed to link to it in search results (using links that point to it to rank it for insance) but can't use the content of the page.
Re: Bing search results showing up in Google
#18robots.txt applies to the source of the links, not the target. So if, say, http://www.paulgraham.com/ links to http://m.bing.com/search , then http://m.bing.com/robots.txt does not apply to that. EDIT: If you think this is wrong, please explain it instead of just downvoting me, because I think it is pretty unfair that I lose karma for explaining my interpretation.
yup. Anchor text is fair game. Keywords taken from anchor texts are usually a good indication on what the linked page is about.
A page can have a high page rank without ever being crawled - given that other high ranked pages have linked to it with good keywords in the anchor text. Such pages can often be identified by the lack of snippet texts.
Edit: I may have misunderstood you, you can only use the data found in links, not follow them.
Re: Bing search results showing up in Google
#19With respect to the author, the conclusion here is very flawed. If you search for Bing in Google, you get Bing all over page 1. If you search for Google in Bing, you get Google all over page 1. That's not the result of Google capturing click stream data from Google Chrome and copying Bing's results, nor is it the result of Microsoft capturing click stream data from IE8 and copying Google's results. That's just the na…
[deleted]
Re: Bing search results showing up in Google
#20Earlier quoted context omitted.
I understand a robots.txt "Disallow: /foo" to mean that it must not crawl that page, i.e. look at the links _inside_ that page.
I've always interpreted it to mean they're explicitly not allowed to touch it - no exceptions (unless you actually specified exceptions for them which you can see on the 3rd last example at http://www.robotstxt.org/robotstxt.html ).
Interesting, and I've always interpreted it the other way. Looks like you are right. But the language on www.robotstxt.org is pretty vague, too. (What does it mean to "visit" an URL?)
The "RFC" on http://www.robotstxt.org/norobots-rfc.txt (Section 3.2.2) states:
"These lines indicate whether accessing a URL that matches the corresponding path is allowed or disallowed. Note that these instructions apply to any HTTP method on a URL."
And I think both interpretations are in theory (but probably not in practice) valid, at least if a search engine is willing to add a site to its index without accessing it (which is unlikely, but not impossible).