Bing search results showing up in Google

A search engine performs the task of narrowing 1 trillion URLs into the 10 that are most likely to be what you are looking for. Associating a query with those 10 results takes teams of software engineers and datacenters full of computers. A user could not reasonably do this on their own.

Contact data is a list of the people you know. I had that list in my brain before I ever visited Facebook. If I am on different social networking sites, that list is mostly the same, because it is information about me that I already knew. I didn't come to Facebook saying "hey Facebook, find for me the 10 people in the world who would be the best possible friends for me." Instead, I come to Facebook and start telling them the names of my friends.

But forget all that for a second. Imagine that contact data and search engines really were the same thing. Google does not access Facebook's social graph, because Facebook has not allowed them to. For Google's behavior to be like Microsoft's Google would have to work around that by using the "clickstream" of Chrome users who are on Facebook. Who knows, maybe Microsoft is doing this already, since they seem to think that "clickstream" is a valid way to mine the internal databases of other companies.

> Bing didn't weight Google ahead of a single other source of information, so I don't think this really comprises evidence of high weighting at all. To make that claim, someone needs to run a controlled test like the one you describe here.

Strictly speaking, I think you're right, but it was interesting that it would accept mappings that didn't exist elsewhere, rather than requiring the same mapping from at least one other source. But you're right that a more complete test is warranted. And Bing may have already adjusted things to prevent this, so we may never know.

> Although I don't really know what half the fields do, so there could be something clever going on in there.

True, but the mysterious parts I see appear to be constants (I may have overlooked something, though, so feel free to point out any mysterious dynamic bits I'm ignoring). Those should be out-and-out harvestable from actual Bing toolbars. With constants, I strongly suspect that we only need to harvest valid data, rather than figure out how to generate our own.

Worse for Bing, my experience in removing viruses says that "people who install lots of toolbars" and "people who get viruses/malware/joined to a botnet" are sets that overlap so a significant degree, so it might become hard to separate actual user clickstream data from forged clickstream data when they're both coming from the same, infected, computer.

Finally, I'm sorry I forgot to give you credit. If it's any consolation, I think I did give you credit last time I posted your link. Thank you for doing a better technical analysis than I've seen from anyone else on this story.

Even if absolutely everything you say is true, even if "clickstream data belongs to the users" and the users can "share" it with Microsoft, it's still the case that Microsoft is riding on the backs of Google's engineers.

And as I said in my original message, maybe that's even a smart business move. But let's call a spade a spade. Microsoft is copying Google search results.

Is that really what you want to be sticking up for?

The two major issues in this article were:

- Google can see and return links to pages without crawling them. I made a video and a blog post about this a while ago: http://www.mattcutts.com/blog/robots-txt-remove-url/

- URL paths are case sensitive. Bing blocks /search in its robots.txt but not /Search. That's how the /Search urls got crawled.

In a later edit, the author suggests "It would be fairly trivial for bots to test if the server is IIS (if the server identifies itself as such of course) or to try to retrieve Robots.txt and robots.txt, if those come up as equal then the sever can be assumed to be case insensitive."

The issue of case sensitivity in robots.txt is a long, very nuanced topic. Here's just one example to get you started: at least back in 2007 when we were talking about this amongst ourselves at Google, the web server for developer.apple.com was case-insensitive, but their robots.txt had lines like this:

Disallow: /documentation/quicktime/

Disallow: /documentation/Quicktime/

Disallow: /documentation/QUICKTIME/

Disallow: /documentation/macosx/

Why would they do that? Apparently because Apple wanted the canonical link to be /documentation/QuickTime . Back then, at least 21M robots.txt files on the web had mixed-case paths. If Google started interpreting robots.txt files from servers that claimed to be IIS differently... well, I'll leave it as an exercise to the reader to come up with some of the unexpected bugs and behavior that could result.

I know it's really tempting to write a headline like "Bing search results showing up in Google," but I wish the author had done more research instead of going for a gotcha. Any SEO worth his/her salt could have explained what was going on here.

No. It prevents pages from being crawled, but references to uncrawled pages can still be shown. See http://www.mattcutts.com/blog/robots-txt-remove-url/ for how it works.