Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bing search results showing up in Google

jacquesmattheij.com

Post reply on HN
Print view View original

61–70 of 131 posts

Re: Bing search results showing up in Google

#61

Post by xilun0 »

jedsmith wrote: post #46
anonymous246 wrote: post #45

Earlier quoted context omitted.

Hmm, I'm not sure you RTFA. He didn't search for the term "bing", he searched for the term "site:.bing.com/search". So it IS kinda a big deal that they're disrespecting the robots.txt and listing those pages anyway. I have also seen Google listing one of my domains for which I specifically disallowed all spiders (Bing doesn't show those domains FYI). My feeble attempt at separating my personal and professional person…

> So it IS kinda a big deal that they're disrespecting the robots.txt and listing those pages anyway. Did you RMFC? The robots.txt doesn't match what is indexed. As has been exhaustively pointed out in this thread by more than a few commenters, "Search" != "search". On that very search results page, we see #1 which is: > OLAC search - Bing OLAC is an unrelated site, and at one point it apparently linked to "www.bing.…

> "Search" != "search"

This is highly ridiculous. HTTP URLs are not mandated to be case-sensitive (though it's recommended), and clearly lots of sites use them in a case-insensitive way. Robots should either consider robots.txt in a case-insensitive way (even if I'm conscious that lot of them, including major ones, currently don't do that, which is precisely what I consider to be a problem, and this is supported by what happened here -- where google is risking to appear as a fool). The following article has perfectly good arguments in favor of case-insensitivity or even more clever handling: http://www.slicksurface.com/blog/2007-04/be-careful-robotstx...

Also; failing to respect conditions of use of a service because an automatic process is not safe enough is not a completely exonerating excuse for the operator of such process...

Re: Bing search results showing up in Google

#62

Post by seanos »

moultano wrote: post #54
unp3rs0n wrote: post #44

Earlier quoted context omitted.

Let's say tomorrow DDG is the search engine with the largest market share. Then Bing would be getting all the clickstream data from DDG. I hope you do realize that this "algorithm" is not Google specific. Its just a novel ranking technique that incorporates a human user feedback loop and is a pretty well known technique in the information retrieval field.

It would be equally unethical to be copying DDG's results in this fashion.

Equally unethical sure (x=y), but differing opinions on whether that (x) is ethical or unethical. Personally, I see nothing wrong with it.

Re: Bing search results showing up in Google

#64

Post by jfager »

haberman wrote: post #35

Never underestimate the ability of a human being to rationalize. If I was in the Microsoft camp, I'm sure I would also be grasping at straws to explain why it's totally fine for Bing to use Google's search results. It's human nature to rationalize. The bottom line is that Bing's index contains associations that it could never have figured out if Google hadn't figured them out first. How many there are, we cannot know…

The thing that bothers me about all this drama is that the actual offense Google wants everyone to be so worked up about is that Bing doesn't filter Google from its clickstream data.

Bing wrote code that works across the whole web. The whole web includes Google. As a result, Bing gets some info from Google. But they didn't get that info because they copied Google, they got it because they didn't filter Google out - or, said another way, because they ignored Google as someone they needed to special case for clickstream analysis.

I don't work in search, but the idea that you're supposed to special-case your competitors when writing general-purpose tools sounds an awful lot like a unilaterally recognized gentleman's agreement. If it's not illegal, and it doesn't hurt end users, why shouldn't it be considered fair game?

I also think it's odd that throughout this whole thing, nobody has really noticed that the only possible way Google could have spotted this issue is if they're keeping very close tabs on Bing's search results. It's another arbitrary line that Google seems to have unilaterally drawn: it's clearly fine to monitor your competitors results closely, which presumably is going to have an effect on your own results; it's only out of bounds when that effect is directly measurable.

Re: Bing search results showing up in Google

#66

Post by seanos »

haberman wrote: post #35

Never underestimate the ability of a human being to rationalize. If I was in the Microsoft camp, I'm sure I would also be grasping at straws to explain why it's totally fine for Bing to use Google's search results. It's human nature to rationalize. The bottom line is that Bing's index contains associations that it could never have figured out if Google hadn't figured them out first. How many there are, we cannot know…

"The bottom line is that Bing's index contains associations that it could never have figured out if Google hadn't figured them out first."

Actually, it is the users who are figuring out what is relevant not Google since a) Bing are collecting data on the links they click regardless of where Google ranks it (could be on the 20th page of Google search results for all Bing care) and b) Bing are collecting this data from all sites, there has been no evidence that Google search results are treated any differently to any other site.

Re: Bing search results showing up in Google

#67

Post by xilun0 »

amalcon wrote: post #52
floatingatoll wrote: post #8

Dear Microsoft, case-sensitivity is important. [1] m.bing.com/robots.txt says "/search", not "/Search". All of the crawled [2] urls are "/Search" or "/~/search". Also, wap.bing.com/robots.txt explicitly "Allow:"s several search pages, which are indexed by google. [1] EDIT: Case insensitivity is often important. Above comment notes that some robots are case-insensitive. I suspect Google is not, based on the results. […

Naturally, Bing is hosted on a Windows server, which inherits the Windows filesystem eccentricities. Case insensitivity is among those. Because "search" has six letters, that would mean that the robots.txt would need to have 64 entries to completely exclude this directory. That's not even including the tilde thing or any other paths to that directory. And that's for one directory. Lame? Yes. Google's fault? Not in th…

What is the value of processing robots.txt in a case-sensitive way? If urls are to have different status when considering case change, the the site structure is just broken. Plus considering robots.txt in case-sensitive way has already result in lots of errors, this one included, and will result in even more in the future. Plus HTTP is not mandated to use case-sensitive URL (though it's recommended). I can't think of any argument of why robots.txt should be processed in a case-sensitive way (i mean for a good reason -- obviously search engine have a very "good" incentive to handle it that way: the possibility to cheat and index more than they should, with an excuse when they are caught), on the other hand i can think of many for case-insensitive processing...

Re: Bing search results showing up in Google

#69

Post by sid0 »

Natsu wrote: post #51
sid0 wrote: post #23

Earlier quoted context omitted.

Google proved it with the bing sting They didn't prove anything. Every experiment needs a control. Where's Google's?

Google's hypothesis was "Does Bing use Google data in its rankings?" That hypothesis was proven (because there was no other way to get those crazy links except for clickstream data showing users going from a google search for those nonsense terms to those sites). If you want to explore a very different hypothesis, namely "Does Bing single out Google in its weightings of clickstream data?" then I suggest you go here:…

Google's hypothesis was "Does Bing use Google data in its rankings?"

I see. I would contend that that hypothesis is misleading at best, in bad faith at worst.

Re: Bing search results showing up in Google

#70

Post by benologist »

moultano wrote: post #57
benologist wrote: post #27

Earlier quoted context omitted.

They proved that Microsoft uses clickstream data to rank websites, and in 7% of manufactured cases that's all the data they have? I think this is exactly as petty and silly as last week's news, and now they get to spend a week explaining how this occurred and that they do obey robots.txt.

Everyone else in this thread has already explained that microsoft is not properly using robots.txt by having case insensitive urls, hence why these urls were indexed.

The Bing context might suck for you guys, but this is your problem - case insensitivity is every Windows server, not just the Bing website.

Why would anyone hosted on Windows have to specify every possible spelling variation to keep search engines out of a folder or file?

Here's another example:

http://www.ifma.org/robots.txt

These guys are disallowing /pv/

http://www.google.com/search?sourceid=chrome&ie=UTF-8&#3...

You guys are indexing /PV/

Post reply on HN