Live data from Hacker News

Bing search results showing up in Google

jacquesmattheij.com

81–90 of 131 posts

Re: Bing search results showing up in Google

#81
post #38

Earlier quoted context omitted.

By that standard, Google should also quit trying to integrate invite-your-fb-friends feature, right? Cuz it's essentially facebook who has figured out how to get massive user signups with their real data and it's google trying to leech. Instead of not trying to integrate fb, Google is accusing fb of not opening up the data. So when it's convenient, you want data opened up. When it's not convenient, you scream "copyca…

Google was freely sharing contact data with Facebook, so there's no hypocrisy here. Contact data is not the same as a search engine.

Freely sharing the data in the sphere where you're trying to catch up, while zealously guarding the data in the sphere where you've got a massive lead, is the alleged hypocrisy.

Hard-won bits is hard-won bits, at a suitably abstract level of analysis.

Re: Bing search results showing up in Google

#82
post #67
post #52

Earlier quoted context omitted.

Naturally, Bing is hosted on a Windows server, which inherits the Windows filesystem eccentricities. Case insensitivity is among those. Because "search" has six letters, that would mean that the robots.txt would need to have 64 entries to completely exclude this directory. That's not even including the tilde thing or any other paths to that directory. And that's for one directory. Lame? Yes. Google's fault? Not in th…

What is the value of processing robots.txt in a case-sensitive way? If urls are to have different status when considering case change, the the site structure is just broken. Plus considering robots.txt in case-sensitive way has already result in lots of errors, this one included, and will result in even more in the future. Plus HTTP is not mandated to use case-sensitive URL (though it's recommended). I can't think of…

Webmaster's shoot themselves in the foot a lot. "My site isn't showing up in google" is a frequent complaint on webmaster help forums, and typically the problem is robots.txt or meta noindex. From that standpoint, since most sites do want to be indexed, it makes sense to follow the standard as strictly as possible. It should be hard to remove your site by accident, which case insensitivity would make somewhat easier. Google states explicitly in their robots.txt policies that it is handled in a case-sensitive way.

Re: Bing search results showing up in Google

#83

Earlier quoted context omitted.

The Bing context might suck for you guys, but this is your problem - case insensitivity is every Windows server, not just the Bing website. Why would anyone hosted on Windows have to specify every possible spelling variation to keep search engines out of a folder or file? Here's another example: http://www.ifma.org/robots.txt These guys are disallowing /pv/ http://www.google.com/search?sourceid=chrome&ie=UTF-8&#3...…

Maybe I'm misunderstanding your argument, but I think you're confusing the Windows file system with the URLs that a web service provides.

The problem exists when that web service is on Windows - ASP.NET, Cold Fusion, static html sites, probably a negligible percent of PHP sites etc -

/pv is /PV is /pV is /Pv

Re: Bing search results showing up in Google

#84

Never underestimate the ability of a human being to rationalize. If I was in the Microsoft camp, I'm sure I would also be grasping at straws to explain why it's totally fine for Bing to use Google's search results. It's human nature to rationalize. The bottom line is that Bing's index contains associations that it could never have figured out if Google hadn't figured them out first. How many there are, we cannot know…

I think you're trying too hard to rationalize Google's claims.

With regard to the 1-in-1000 signals, clearly they have that many signals because not all 1000 signals are strong for every query. In Google's case, they "Googlebombed" Bing, but instead of using the "anchor" signal (which most people use to alter Google's results), they chose the "click stream" signal. Google specifically chose scenarios where the other 999 signals weren't being used.

If I had a toolbar installed on millions of machines where people opted-in to send me their click activity, I could start a search engine that used only click stream data to rank results. You'd be seeing the ranking effected by all sorts of sites including Facebook, Google, Amazon, Wikipedia, etc... As it should, the ranking would be effected by the click activity that users perform on the most popular websites. Effectively I'd have crowd sourced my ranking. If Google is returning a result for a particular person's name, but Facebook's click stream activity is returning a fan page for that person's name which is clicked on more frequently than the Google result, then the Facebook result would show up in my search engine for that person's name.

All Google did here was choose a scenario where every other signal wasn't being used by Bing, effectively turning Bing into the click-stream search engine described above. Furthermore, Google chose a scenario where even within this single signal, they were the only input to it. All Google did was googlebomb Bing. Any well trafficked website could do the same.

It's ridiculous the lengths people will go to while trying to discredit Microsoft. Bing has built a generalized system that partially learns rankings through behaviors observed on high-reputation websites. That's it.

And no they shouldn't remove Google from the signal because that'd imply they've done something wrong.

Re: Bing search results showing up in Google

#85

Earlier quoted context omitted.

I don't understand why everyone is using the term "copying the results". I think what Bing did was very smart, they incorporated user clickstream data. One could accuse this method of walking a thin line morally, but I suspect that Google's accusation wouldn't have stood any water as a lawsuit.

What Bing did is very smart! It actually was, but not crediting Google there just makes it look like a cheap shot. You won't quote someone without citing their name now would you? Sadly you are right, they won't be able to push a lawsuit. Not enough grounds for it, however Bing should acknowledge what they did and are doing crediting Google. If Google hadn't caught them, we would all be thinking Bing did it on its ow…

Taking liberty to concoct a scenario. If Walmart asked shoppers to take a photograph of the product layout on display at their favorite shop (which say happens to be Target because its the most popular in town) and used that to make small modifications to its own layout, would you say Walmart needs to credit Target, or that its copying Target? This is an arrangement between Walmart and its shoppers and there is nothing Target can do about it other than making a brouhaha. I don't see why things have to be different in the digital world. We all know how user interfaces historically have been blatantly ripped off.

Re: Bing search results showing up in Google

#87
post #68
post #11

Aren't /search and /Search considered two different directories when it comes to robots.txt?

Yes. RFC 3986, sections 6.2.2.2 and 6.2.3.

I think you got the wrong RFC number:

http://tools.ietf.org/rfc/rfc3938.txt

No mention of robots.txt, nor a section 6.

Re: Bing search results showing up in Google

#88

Earlier quoted context omitted.

Fine. I reason I stated my point was so that others could chime in if they've had a similar experience. Let me ask you and others this: Is the following robots.txt supposed to exclude all pages from my domain from showing up in Google results? Am I missing something? According to http://www.robotstxt.org/robotstxt.html I think I'm doing the right thing. Same file is returned for www. .com/robots.txt and .com/robots.t…

> Let me ask you and others this: Is the following robots.txt supposed to exclude all pages from my domain from showing up in Google results? I believe that robots.txt is a way to prevent your site from being crawled by a robot, but it is not a blacklist against your site appearing in Google search results if it finds a link to your page on a site that does allow robots. Check out this page: http://www.google.com/sup…

So it appears the only way to not appear is to let them crawl to see your NOINDEX tag.

And while it's clear NOINDEX prevents a page from appearing in results, it's not clear that it excludes the page contents from analysis by any of Google's algorithms, once collected. (Is it still used to train the spell-checker, for example?)

Re: Bing search results showing up in Google

#89
post #64

Earlier quoted context omitted.

The thing that bothers me about all this drama is that the actual offense Google wants everyone to be so worked up about is that Bing doesn't filter Google from its clickstream data. Bing wrote code that works across the whole web. The whole web includes Google. As a result, Bing gets some info from Google. But they didn't get that info because they copied Google, they got it because they didn't filter Google out - o…

I thought part of the point is that whatever Bing is doing doesn't work across the whole web. They need to associate the URL with a query, and most websites don't have queries. It's not just that Bing has recorded a click on Miley Cyrus's webpage; it's that they've done that and associated it with the query [kecgxjpgqoe].

People have made that point, but I don't understand it. 'kecgx...' shows up as a parameter in the url of the Google query. Tons of sites include relevant information in parameterized urls; why is it unexpected that Bing would use that information across the whole web? Other people have said that implies that Bing has to have special Google url-parsing code, but that's not true at all - query parameters in urls are standardized. You would have to have special code to understand the specific semantics of Google's query urls, but there's no reason to think Bing needs or wants parameter semantics, they could easily just be interested in making probabilistic associations.

Re: Bing search results showing up in Google

#90
post #66

Never underestimate the ability of a human being to rationalize. If I was in the Microsoft camp, I'm sure I would also be grasping at straws to explain why it's totally fine for Bing to use Google's search results. It's human nature to rationalize. The bottom line is that Bing's index contains associations that it could never have figured out if Google hadn't figured them out first. How many there are, we cannot know…

"The bottom line is that Bing's index contains associations that it could never have figured out if Google hadn't figured them out first." Actually, it is the users who are figuring out what is relevant not Google since a) Bing are collecting data on the links they click regardless of where Google ranks it (could be on the 20th page of Google search results for all Bing care) and b) Bing are collecting this data from…

> Actually, it is the users who are figuring out what is relevant not Google

When you do a Google search, Google chooses ten URLs from over 1 trillion that are in its index. Creating this list takes teams of full-time engineers and data-centers full of servers that crawl the internet constantly.

The user spends, on average, less than ten seconds choosing 1 out of those 10 URLs. In a small minority of cases, they'll click to further pages, in which case maybe they were choosing 1 out of 100.

So Google narrowed the search down from 1 trillion URLs to 10, and the user narrowed it down from 10 URLs to 1. And yet you think it's the user who did the hard work?

Post reply on HN