Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bing search results showing up in Google

jacquesmattheij.com

Post reply on HN
Print view View original

91–100 of 131 posts

Re: Bing search results showing up in Google

#91

Post by anonymous246 »

haberman wrote: post #80
anonymous246 wrote: post #71

Earlier quoted context omitted.

Fine. I reason I stated my point was so that others could chime in if they've had a similar experience. Let me ask you and others this: Is the following robots.txt supposed to exclude all pages from my domain from showing up in Google results? Am I missing something? According to http://www.robotstxt.org/robotstxt.html I think I'm doing the right thing. Same file is returned for www. .com/robots.txt and .com/robots.t…

> Let me ask you and others this: Is the following robots.txt supposed to exclude all pages from my domain from showing up in Google results? I believe that robots.txt is a way to prevent your site from being crawled by a robot, but it is not a blacklist against your site appearing in Google search results if it finds a link to your page on a site that does allow robots. Check out this page: http://www.google.com/sup…

Great. This link helps. Google's site basically says that in addition to specifically excluding their crawler via robots.txt, I also have MANUALLY submit a request to them. As I said in a reply below, this is nonsensical and Bing is being more reasonable, but I'll grit my teeth and do it since Google has more power here.

It definitely seems like Google is exploiting a loophole in spirit of the definition of robots.txt. Robots.txt is an ancient standard, and I don't think it was anticipated at that time that search engines would gain enough confidence about pages' relevance to list them even if they had not indexed/crawled them.

Re: Bing search results showing up in Google

#92

Post by nostrademons »

jfager wrote: post #89
nostrademons wrote: post #79

Earlier quoted context omitted.

I thought part of the point is that whatever Bing is doing doesn't work across the whole web. They need to associate the URL with a query, and most websites don't have queries. It's not just that Bing has recorded a click on Miley Cyrus's webpage; it's that they've done that and associated it with the query [kecgxjpgqoe].

People have made that point, but I don't understand it. 'kecgx...' shows up as a parameter in the url of the Google query. Tons of sites include relevant information in parameterized urls; why is it unexpected that Bing would use that information across the whole web? Other people have said that implies that Bing has to have special Google url-parsing code, but that's not true at all - query parameters in urls are st…

"You would have to have special code to understand the specific semantics of Google's query urls"

That's why people say that it's special-cased. There's no web standard that says the 'q' parameter means that the page is a search engine and the parameter is the query. That was something AltaVista did a long time back (possibly for byte-saving reasons, or possibly because they were lazy) and Google et al copied. Many other search engines use a different system, eg. DDG puts it in the request path, InfoSeek used qt=, Excite used search=.

Re: Bing search results showing up in Google

#93

Post by zaidf »

haberman wrote: post #42
zaidf wrote: post #38

Earlier quoted context omitted.

By that standard, Google should also quit trying to integrate invite-your-fb-friends feature, right? Cuz it's essentially facebook who has figured out how to get massive user signups with their real data and it's google trying to leech. Instead of not trying to integrate fb, Google is accusing fb of not opening up the data. So when it's convenient, you want data opened up. When it's not convenient, you scream "copyca…

Google was freely sharing contact data with Facebook, so there's no hypocrisy here. Contact data is not the same as a search engine.

Contact data is not the same as a search engine.

Elaborate? Bing didn't steal Google's source code. They intercepted some linkstream data with the user's permission.

Re: Bing search results showing up in Google

#94

Post by Herring »

sgk284 wrote: post #84
haberman wrote: post #35

Never underestimate the ability of a human being to rationalize. If I was in the Microsoft camp, I'm sure I would also be grasping at straws to explain why it's totally fine for Bing to use Google's search results. It's human nature to rationalize. The bottom line is that Bing's index contains associations that it could never have figured out if Google hadn't figured them out first. How many there are, we cannot know…

I think you're trying too hard to rationalize Google's claims. With regard to the 1-in-1000 signals, clearly they have that many signals because not all 1000 signals are strong for every query. In Google's case, they "Googlebombed" Bing, but instead of using the "anchor" signal (which most people use to alter Google's results), they chose the "click stream" signal. Google specifically chose scenarios where the other…

>As it should, the ranking would be effected by the click activity that users perform on the most popular websites. Effectively I'd have crowd sourced my ranking.

Not exactly. Those sites work hard to present relevant links to the crowd. If the sites have competent engineers, your record of clicks cannot add much value -- if just because you only have a subset of all the clicks on that site. So you see google says they're seeing increasing overlap with bing over time.

Also see http://news.ycombinator.com/item?id=2184025

Re: Bing search results showing up in Google

#95

Post by jfager »

nostrademons wrote: post #92
jfager wrote: post #89

Earlier quoted context omitted.

People have made that point, but I don't understand it. 'kecgx...' shows up as a parameter in the url of the Google query. Tons of sites include relevant information in parameterized urls; why is it unexpected that Bing would use that information across the whole web? Other people have said that implies that Bing has to have special Google url-parsing code, but that's not true at all - query parameters in urls are st…

"You would have to have special code to understand the specific semantics of Google's query urls" That's why people say that it's special-cased. There's no web standard that says the 'q' parameter means that the page is a search engine and the parameter is the query. That was something AltaVista did a long time back (possibly for byte-saving reasons, or possibly because they were lazy) and Google et al copied. Many o…

Why do you think Bing cares or has to know that "q=" means a Google query term? My point is that they don't have to have any semantic information about parameter keys to be able to derive probabilistic associations between parameter values and clicks.

If you consistently see pages with 'foo' as a parameter value to any parameter key, and clicks on those pages consistently go to site bar, it's completely reasonable to start associating foo with bar, regardless of what the parameter keys are.

Re: Bing search results showing up in Google

#96

Post by jfr »

nostrademons wrote: post #87
jfr wrote: post #68

Earlier quoted context omitted.

Yes. RFC 3986, sections 6.2.2.2 and 6.2.3.

I think you got the wrong RFC number: http://tools.ietf.org/rfc/rfc3938.txt No mention of robots.txt, nor a section 6.

Sorry, the correct number was 3986.

RFC 3986 - Uniform Resource Identifier (URI): Generic Syntax

Re: Bing search results showing up in Google

#97

Post by jiji »

sajidnizami wrote: post #48
unp3rs0n wrote: post #33

Earlier quoted context omitted.

I don't understand why everyone is using the term "copying the results". I think what Bing did was very smart, they incorporated user clickstream data. One could accuse this method of walking a thin line morally, but I suspect that Google's accusation wouldn't have stood any water as a lawsuit.

What Bing did is very smart! It actually was, but not crediting Google there just makes it look like a cheap shot. You won't quote someone without citing their name now would you? Sadly you are right, they won't be able to push a lawsuit. Not enough grounds for it, however Bing should acknowledge what they did and are doing crediting Google. If Google hadn't caught them, we would all be thinking Bing did it on its ow…

Actually, Bing should credit the users providing all that clickstream data and all the websites they are using. It just happens that Google is one of many... what's your point?

Re: Bing search results showing up in Google

#98

Post by [deleted] »

xilun0 wrote: post #61
jedsmith wrote: post #46

Earlier quoted context omitted.

> So it IS kinda a big deal that they're disrespecting the robots.txt and listing those pages anyway. Did you RMFC? The robots.txt doesn't match what is indexed. As has been exhaustively pointed out in this thread by more than a few commenters, "Search" != "search". On that very search results page, we see #1 which is: > OLAC search - Bing OLAC is an unrelated site, and at one point it apparently linked to "www.bing.…

> "Search" != "search" This is highly ridiculous. HTTP URLs are not mandated to be case-sensitive (though it's recommended), and clearly lots of sites use them in a case-insensitive way. Robots should either consider robots.txt in a case-insensitive way (even if I'm conscious that lot of them, including major ones, currently don't do that, which is precisely what I consider to be a problem, and this is supported by w…

[deleted]

Re: Bing search results showing up in Google

#99

Post by angusgr »

Natsu wrote: post #51
sid0 wrote: post #23

Earlier quoted context omitted.

Google proved it with the bing sting They didn't prove anything. Every experiment needs a control. Where's Google's?

Google's hypothesis was "Does Bing use Google data in its rankings?" That hypothesis was proven (because there was no other way to get those crazy links except for clickstream data showing users going from a google search for those nonsense terms to those sites). If you want to explore a very different hypothesis, namely "Does Bing single out Google in its weightings of clickstream data?" then I suggest you go here:…

Good points. I'm the one who took the packet capture you've linked, and I'd be really interested to see what happens if someone runs the test you describe.

That said, I'd say that the fact that they weight Google highly enough that they'd take their word for it that a clearly irrelevant term should be mapped to some site is strong evidence, I think.

The evidence was that this occurred in 9% of the cases they tested. For terms that don't exist anywhere else on the net (ie the only place they could have been found anywhere in Bing's dataset was in the google query URL.)

Bing didn't weight Google ahead of a single other source of information, so I don't think this really comprises evidence of high weighting at all. To make that claim, someone needs to run a controlled test like the one you describe here.

spammers/SEO types will make it useless soon enough

Yeah, one thing that surprised me greatly is how easily it looks like this clickstream data could be injected. Although I don't really know what half the fields do, so there could be something clever going on in there.

Re: Bing search results showing up in Google

#100

Post by angusgr »

sid0 wrote: post #69
Natsu wrote: post #51

Earlier quoted context omitted.

Google's hypothesis was "Does Bing use Google data in its rankings?" That hypothesis was proven (because there was no other way to get those crazy links except for clickstream data showing users going from a google search for those nonsense terms to those sites). If you want to explore a very different hypothesis, namely "Does Bing single out Google in its weightings of clickstream data?" then I suggest you go here:…

Google's hypothesis was "Does Bing use Google data in its rankings?" I see. I would contend that that hypothesis is misleading at best, in bad faith at worst.

I think the problem wasn't so much the hypothesis, but the spin on the conclusions.
Post reply on HN