Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bing search results showing up in Google

jacquesmattheij.com

Post reply on HN
Print view View original

71–80 of 131 posts

Re: Bing search results showing up in Google

#71

Post by anonymous246 »

haberman wrote: post #47
anonymous246 wrote: post #45

Earlier quoted context omitted.

Hmm, I'm not sure you RTFA. He didn't search for the term "bing", he searched for the term "site:.bing.com/search". So it IS kinda a big deal that they're disrespecting the robots.txt and listing those pages anyway. I have also seen Google listing one of my domains for which I specifically disallowed all spiders (Bing doesn't show those domains FYI). My feeble attempt at separating my personal and professional person…

They are not disrespecting robots.txt: http://news.ycombinator.com/item?id=2183519 > I have also seen Google listing one of my domains for which I specifically disallowed all spiders [...] Since I'm trying to be anon here, I won't be able to list the search term. Sorry. Unsubstantiated accusations. If you truly think Google is disregarding robots.txt but you don't want to divulge the original site, you should set up…

Fine. I reason I stated my point was so that others could chime in if they've had a similar experience.

Let me ask you and others this: Is the following robots.txt supposed to exclude all pages from my domain from showing up in Google results? Am I missing something? According to http://www.robotstxt.org/robotstxt.html I think I'm doing the right thing. Same file is returned for www..com/robots.txt and .com/robots.txt. Google lists .com/ in results. I don't think it should be.

    User-agent: *
    Disallow: /

Re: Bing search results showing up in Google

#72

Post by seanos »

joesb wrote: post #41
haberman wrote: post #35

Never underestimate the ability of a human being to rationalize. If I was in the Microsoft camp, I'm sure I would also be grasping at straws to explain why it's totally fine for Bing to use Google's search results. It's human nature to rationalize. The bottom line is that Bing's index contains associations that it could never have figured out if Google hadn't figured them out first. How many there are, we cannot know…

I can not agree more with your second paragraph. Yes, Bing toolbar just rely on people clicking links. But without Google doing all the hardwork here, users will never know about such link, that's why they went to Google. And there will not be any of such link to feed to Bing's system.

You could also argue that the user is doing the hardwork by going to Google (or any other site, since this is not specific to Google) and deciding which links are relevant by clicking on them. The user has opted share this work they have done with Microsoft (by installing the toolbar and agreeing to the terms) to help them get better results on Bing.

Re: Bing search results showing up in Google

#73

Post by andrewcooke »

benologist wrote: post #70
moultano wrote: post #57

Earlier quoted context omitted.

Everyone else in this thread has already explained that microsoft is not properly using robots.txt by having case insensitive urls, hence why these urls were indexed.

The Bing context might suck for you guys, but this is your problem - case insensitivity is every Windows server, not just the Bing website. Why would anyone hosted on Windows have to specify every possible spelling variation to keep search engines out of a folder or file? Here's another example: http://www.ifma.org/robots.txt These guys are disallowing /pv/ http://www.google.com/search?sourceid=chrome&ie=UTF-8&#3...…

Maybe I'm misunderstanding your argument, but I think you're confusing the Windows file system with the URLs that a web service provides.

Re: Bing search results showing up in Google

#74

Post by [deleted] »

jfr wrote: post #59
jedsmith wrote: post #13

With respect to the author, the conclusion here is very flawed. If you search for Bing in Google, you get Bing all over page 1. If you search for Google in Bing, you get Google all over page 1. That's not the result of Google capturing click stream data from Google Chrome and copying Bing's results, nor is it the result of Microsoft capturing click stream data from IE8 and copying Google's results. That's just the na…

Except for the schema and host parts (which are not part of robots.txt anyway) URLs are case sensitive (ref: RFC 3986, sections 6.2.2.2 and 6.2.3). The problem here is that Microsoft servers respond to /search, /Search and /SeaRCh without distinction. They are all distinct URLs. If it was the intended behavior (stupid, but understandable, coming from Microsoft), then robots.txt should contain all variants in capitali…

[deleted]

Re: Bing search results showing up in Google

#75

Post by [deleted] »

ajays wrote: post #40
jsnell wrote: post #36

> never mind that Bing only used its toolbar as a url discovery device That is obviously untrue, and shows that the author does not understand the issue even superficially. The Google experiment showed that Bing was associating urls to search terms for no reason other than that Google had done so. You know, like making a search for mbzrxpgjys return rim.com, a URL which we can safely assume Bing was already quite awa…

the author does not understand the issue even superficially. That, in a nutshell, is it. I don't know why we're spending so much time on this post, as the author has no idea about how search engines work, and what is robots.txt . If he had just looked at Bing's robots.txt and the URLs in Google's results, he would have seen that each and every one of them passed robots.txt . Since he site-restricted the search to ".b…

[deleted]

Re: Bing search results showing up in Google

#76

Post by jeroen »

moultano wrote: post #57
benologist wrote: post #27

Earlier quoted context omitted.

They proved that Microsoft uses clickstream data to rank websites, and in 7% of manufactured cases that's all the data they have? I think this is exactly as petty and silly as last week's news, and now they get to spend a week explaining how this occurred and that they do obey robots.txt.

Everyone else in this thread has already explained that microsoft is not properly using robots.txt by having case insensitive urls, hence why these urls were indexed.

"Be strict in what you emit, liberal in what you accept."

Bing failed the first part, Google the second. Both should fix that.

Re: Bing search results showing up in Google

#77

Post by anonymous246 »

illdave wrote: post #60
anonymous246 wrote: post #45

Earlier quoted context omitted.

Hmm, I'm not sure you RTFA. He didn't search for the term "bing", he searched for the term "site:.bing.com/search". So it IS kinda a big deal that they're disrespecting the robots.txt and listing those pages anyway. I have also seen Google listing one of my domains for which I specifically disallowed all spiders (Bing doesn't show those domains FYI). My feeble attempt at separating my personal and professional person…

There's a bit of a weird myth with robots.txt and the idea that it prevents pages from showing up in search engine indexes. Robots.txt means that the search engine cannot crawl the page - it can still include it in the search results if it sees enough sites linking to it. It can take a guess at what the page title can be, but there's usually no descriptive snippet because it's unable to see what's on the page. If you…

Thanks. I think this may be the reason.

IMHO, Bing is being more reasonable here.

So, I have to let Google see any page which I want to prevent it from showing it to others. If I want Google to show NONE of my pages to others, I should show them ALL my pages. Conveniently, there is no wildcarded noindex, is there? Nonsensical, but since Google has more power here I'll probably have to bend to their whim.

Re: Bing search results showing up in Google

#78

Post by nostrademons »

moultano wrote: post #54
unp3rs0n wrote: post #44

Earlier quoted context omitted.

Let's say tomorrow DDG is the search engine with the largest market share. Then Bing would be getting all the clickstream data from DDG. I hope you do realize that this "algorithm" is not Google specific. Its just a novel ranking technique that incorporates a human user feedback loop and is a pretty well known technique in the information retrieval field.

It would be equally unethical to be copying DDG's results in this fashion.

Highly ironic, though, as DDG uses Yahoo as a backend, which uses Bing, which uses Google, which would use...DDG? I think there's a cycle in that list somewhere...

Re: Bing search results showing up in Google

#79

Post by nostrademons »

jfager wrote: post #64
haberman wrote: post #35

Never underestimate the ability of a human being to rationalize. If I was in the Microsoft camp, I'm sure I would also be grasping at straws to explain why it's totally fine for Bing to use Google's search results. It's human nature to rationalize. The bottom line is that Bing's index contains associations that it could never have figured out if Google hadn't figured them out first. How many there are, we cannot know…

The thing that bothers me about all this drama is that the actual offense Google wants everyone to be so worked up about is that Bing doesn't filter Google from its clickstream data. Bing wrote code that works across the whole web. The whole web includes Google. As a result, Bing gets some info from Google. But they didn't get that info because they copied Google, they got it because they didn't filter Google out - o…

I thought part of the point is that whatever Bing is doing doesn't work across the whole web. They need to associate the URL with a query, and most websites don't have queries. It's not just that Bing has recorded a click on Miley Cyrus's webpage; it's that they've done that and associated it with the query [kecgxjpgqoe].

Re: Bing search results showing up in Google

#80

Post by haberman »

anonymous246 wrote: post #71
haberman wrote: post #47

Earlier quoted context omitted.

They are not disrespecting robots.txt: http://news.ycombinator.com/item?id=2183519 > I have also seen Google listing one of my domains for which I specifically disallowed all spiders [...] Since I'm trying to be anon here, I won't be able to list the search term. Sorry. Unsubstantiated accusations. If you truly think Google is disregarding robots.txt but you don't want to divulge the original site, you should set up…

Fine. I reason I stated my point was so that others could chime in if they've had a similar experience. Let me ask you and others this: Is the following robots.txt supposed to exclude all pages from my domain from showing up in Google results? Am I missing something? According to http://www.robotstxt.org/robotstxt.html I think I'm doing the right thing. Same file is returned for www. .com/robots.txt and .com/robots.t…

> Let me ask you and others this: Is the following robots.txt supposed to exclude all pages from my domain from showing up in Google results?

I believe that robots.txt is a way to prevent your site from being crawled by a robot, but it is not a blacklist against your site appearing in Google search results if it finds a link to your page on a site that does allow robots.

Check out this page: http://www.google.com/support/webmasters/bin/answer.py?hl=en...

Specifically check out the section "I want to completely remove a page from search results." It appears that if you use the "noindex" meta tag, you can prevent the site from showing up in search results even if other pages link to it. The noindex meta tag is documented here: http://www.google.com/support/webmasters/bin/answer.py?answe...

Post reply on HN