Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bing search results showing up in Google

jacquesmattheij.com

Post reply on HN
Print view View original

121–130 of 131 posts

Re: Bing search results showing up in Google

#121

Post by Matt_Cutts »

mc32 wrote: post #120
Matt_Cutts wrote: post #108

The two major issues in this article were: - Google can see and return links to pages without crawling them. I made a video and a blog post about this a while ago: http://www.mattcutts.com/blog/robots-txt-remove-url/ - URL paths are case sensitive. Bing blocks /search in its robots.txt but not /Search. That's how the /Search urls got crawled. In a later edit, the author suggests "It would be fairly trivial for bots t…

>"...but I wish the author had done more research instead of going for a gotcha." That's a bit of irony right there, isn't it? --wow, Matt, I guess it cuts both ways, eh?

Anything in particular you're talking about? I've said a lot of stuff online over the last 10 years. :)

Re: Bing search results showing up in Google

#122

Post by mc32 »

Matt_Cutts wrote: post #121
mc32 wrote: post #120

Earlier quoted context omitted.

>"...but I wish the author had done more research instead of going for a gotcha." That's a bit of irony right there, isn't it? --wow, Matt, I guess it cuts both ways, eh?

Anything in particular you're talking about? I've said a lot of stuff online over the last 10 years. :)

It's no secret that a fair number of commenters throughout the controversy have sided with Google's or Bing's side.

Some people have thought that the initial story came to a premature conclusion (where, incidentally, Bing cried foul). Now we have Jacques, it appears, arriving at premature conclusion as well and we have Google crying foul. To me, that's the essence of irony.

I'm sure both Google Search and Bing have your reasons for the way the dirty laundry has been brought out in public.

Re: Bing search results showing up in Google

#123

Post by angusgr »

Natsu wrote: post #104
angusgr wrote: post #99

Earlier quoted context omitted.

Good points. I'm the one who took the packet capture you've linked, and I'd be really interested to see what happens if someone runs the test you describe. That said, I'd say that the fact that they weight Google highly enough that they'd take their word for it that a clearly irrelevant term should be mapped to some site is strong evidence, I think. The evidence was that this occurred in 9% of the cases they tested.…

> Bing didn't weight Google ahead of a single other source of information, so I don't think this really comprises evidence of high weighting at all. To make that claim, someone needs to run a controlled test like the one you describe here. Strictly speaking, I think you're right, but it was interesting that it would accept mappings that didn't exist elsewhere, rather than requiring the same mapping from at least one…

Finally, I'm sorry I forgot to give you credit.

Ah, that's no problem. If you'd copied it to your domain and linked it there I might have minded, but if you're posting a link to my site I consider that plenty of credit.

Re: Bing search results showing up in Google

#124

Post by Natsu »

Matt_Cutts wrote: post #121
mc32 wrote: post #120

Earlier quoted context omitted.

>"...but I wish the author had done more research instead of going for a gotcha." That's a bit of irony right there, isn't it? --wow, Matt, I guess it cuts both ways, eh?

Anything in particular you're talking about? I've said a lot of stuff online over the last 10 years. :)

I don't know, but I would assume that they're one of the people who don't think that what Bing did is a big deal. If I understand the arguments properly, most either believe that the clickstream data belongs to the user (who gave it away), or they believe that you haven't actually proven that they're singling out Google in their clickstream data.

Anyhow, I've already posted in this story how anyone who wants to could go do their own experiment (it appears that Bing's data isn't too hard to fake, that goes double for anyone who can reverse engineer the toolbar). Similarly, I'm not convinced that Bing doing this is going to ruin search any time soon, mostly because I can see spammers/blackhat SEO types renting botnets to feed Bing all the bogus clickstream data they want.

It appears to be a simple http request with some time zones, link text, and an identifier or two that can be harvested from actual toolbars. If they don't bother to spam them, it's because they believe that Bing is irrelevant. And when I say "irrelevant" I mean "even less relevant than a low-traffic wiki for a free game that's currently under a massive assault from spambots."

Which, incidentally, might be one good reason for your team to look more at keeping wiki-spam out of your index. Some spam results from that wiki (in spite of having rel=no-follow) were seen in Google's index and stayed there until the admins caught on and cleaned things up.

Re: Bing search results showing up in Google

#125

Post by Natsu »

jfager wrote: post #95
nostrademons wrote: post #92

Earlier quoted context omitted.

"You would have to have special code to understand the specific semantics of Google's query urls" That's why people say that it's special-cased. There's no web standard that says the 'q' parameter means that the page is a search engine and the parameter is the query. That was something AltaVista did a long time back (possibly for byte-saving reasons, or possibly because they were lazy) and Google et al copied. Many o…

Why do you think Bing cares or has to know that "q=" means a Google query term? My point is that they don't have to have any semantic information about parameter keys to be able to derive probabilistic associations between parameter values and clicks. If you consistently see pages with 'foo' as a parameter value to any parameter key, and clicks on those pages consistently go to site bar, it's completely reasonable to…

> Why do you think Bing cares or has to know that "q=" means a Google query term?

If that's true, then they should also be associating the sites linked with all the other weird parameter values in a search query, which would spam them to heck. Here are all the params from a search I just did on google:

q=test&ie=utf-8&oe=utf-8&aq=t&rls={moz:distributionID}:{moz:locale}:{moz:official}&client=firefox

They'd start making a lot of strange associations between random sites and "utf-8" if that were true, because that parameter shows up in just about every Google search done in English.

It's also a perfectly normal thing for programmers to search for, so they'd clutter up their index with millions of sites that had nothing whatsoever to do with utf-8.

So to make any real use out of that, they had to understand what the parameters in there actually mean, rather than associating ALL of them with whatever site was next in the clickstream.

Though I grant you, that does not disprove the alternate hypothesis that they were dumb and polluted their index with loads of irrelevant crap.

And I admit that I found weak support for that hypothesis by trying to see if utf-8 was linked to rim.com (one of the tests sites, if memory serves):

http://www.bing.com/search?q=utf-8+rim&go=&form=QBRE

Those results appear to be crap, though I'm not sure that any sensible results exist that they could return.

Re: Bing search results showing up in Google

#126

Post by yaix »

Shouldn't the headline be "Bing explicitly allowing some results pages to showing up in other search engines".

The wap.bing.com/robots.txt blocks all /search/ and then explicitly allows a few. What ever the reason is for that.

Very weak article, IMHO.

Re: Bing search results showing up in Google

#127

Post by jfager »

Natsu wrote: post #125
jfager wrote: post #95

Earlier quoted context omitted.

Why do you think Bing cares or has to know that "q=" means a Google query term? My point is that they don't have to have any semantic information about parameter keys to be able to derive probabilistic associations between parameter values and clicks. If you consistently see pages with 'foo' as a parameter value to any parameter key, and clicks on those pages consistently go to site bar, it's completely reasonable to…

> Why do you think Bing cares or has to know that "q=" means a Google query term? If that's true, then they should also be associating the sites linked with all the other weird parameter values in a search query, which would spam them to heck. Here are all the params from a search I just did on google: q=test&ie=utf-8&oe=utf-8&aq=t&rls={moz:distributionID}:{moz:locale}:{moz:official}&client=firefox They'd start makin…

If that's true, then they should also be associating the sites linked with all the other weird parameter values in a search query

If it's a parameter value people will ever actually use Bing to search for ('utf-8'), there's probably plenty of other signal to help them figure out which results to return. If it's not ('kecgxjpgqoe'), we already know they sometimes return crap, thanks to Google's little experiment.

They'd start making a lot of strange associations between random sites and "utf-8" if that were true, because that parameter shows up in just about every Google search done in English.

If it shows up as a parameter for every search, why do you think Bing's algorithm would decide it was a good indicator for a particular url? P(foo.com|utf-8) wouldn't be any different from P(bar.com|utf-8), making 'utf-8' basically worthless as a discriminator. I'm pretty sure the folks at Bing understand the concept of conditional probability.

It's also a perfectly normal thing for programmers to search for, so they'd clutter up their index with millions of sites that had nothing whatsoever to do with utf-8

I see no justification for the idea that url parameters are somehow a more difficult challenge in this regard than the mass of crap that is content on the web, which we know they crawl and index at a massive scale.

Re: Bing search results showing up in Google

#128

Post by Herring »

seanos wrote: post #115
haberman wrote: post #90

Earlier quoted context omitted.

> Actually, it is the users who are figuring out what is relevant not Google When you do a Google search, Google chooses ten URLs from over 1 trillion that are in its index. Creating this list takes teams of full-time engineers and data-centers full of servers that crawl the internet constantly. The user spends, on average, less than ten seconds choosing 1 out of those 10 URLs. In a small minority of cases, they'll c…

Individually, it negligible. Cumulatively, the time spent by the many thousands (millions?) of users who are clicking links they deem relevant for queries all over the web (Bing are collecting data from all sites) is huge... My point is that the assertion that Google are doing all the work in creating the data used by Bing is clearly false.

those millions of users are all searching on platforms where 99%+ of the work was already done by the site's own engineers. Cmon, this isn't rocket science.

Re: Bing search results showing up in Google

#129

Post by Natsu »

jfager wrote: post #127
Natsu wrote: post #125

Earlier quoted context omitted.

> Why do you think Bing cares or has to know that "q=" means a Google query term? If that's true, then they should also be associating the sites linked with all the other weird parameter values in a search query, which would spam them to heck. Here are all the params from a search I just did on google: q=test&ie=utf-8&oe=utf-8&aq=t&rls={moz:distributionID}:{moz:locale}:{moz:official}&client=firefox They'd start makin…

If that's true, then they should also be associating the sites linked with all the other weird parameter values in a search query If it's a parameter value people will ever actually use Bing to search for ('utf-8'), there's probably plenty of other signal to help them figure out which results to return. If it's not ('kecgxjpgqoe'), we already know they sometimes return crap, thanks to Google's little experiment. They…

It would only show up as a parameter "relevant" to whatever sites were next in the clickstream. Remember, not every page transition is Google -> other site.

They'd also be gobbling up tons of random things from forums and whatnot (which should appear on long tail searches, if we knew where to look), most of which spam the heck out of you with random parameters, forum names, and whatnot.

> I see no justification for the idea that url parameters are somehow a more difficult challenge in this regard than the mass of crap that is content on the web, which we know they crawl and index at a massive scale.

Which is why I see no reason to assume that they don't understand or attempt to understand the actual meaning of parameters passed to one of the biggest sites on the internet.

Re: Bing search results showing up in Google

#130

Post by jfager »

Natsu wrote: post #129
jfager wrote: post #127

Earlier quoted context omitted.

If that's true, then they should also be associating the sites linked with all the other weird parameter values in a search query If it's a parameter value people will ever actually use Bing to search for ('utf-8'), there's probably plenty of other signal to help them figure out which results to return. If it's not ('kecgxjpgqoe'), we already know they sometimes return crap, thanks to Google's little experiment. They…

It would only show up as a parameter "relevant" to whatever sites were next in the clickstream. Remember, not every page transition is Google -> other site. They'd also be gobbling up tons of random things from forums and whatnot (which should appear on long tail searches, if we knew where to look), most of which spam the heck out of you with random parameters, forum names, and whatnot. > I see no justification for t…

You said 'utf-8' shows up as a parameter in almost every English Google search, and suggested that this would cause weird associations between 'utf-8' and random pages.

I pointed out that there's no reason to expect this to be true, because Bing engineers are likely smart enough to realize that if the probability of clicking on to foo.com is not significantly different than the probability of clicking on to bar.com given the presence of the 'utf-8' parameter, then 'utf-8' is a pretty poor discriminator between foo.com and bar.com, and probably shouldn't be used to determine search results.

It doesn't matter that not every page transition is Google -> another site. You still wouldn't need to special-case Google to determine an association with a parameter is useful or useless - the same code could build that model for any site with params.

They'd also be gobbling up tons of random things from forums and whatnot (which should appear on long tail searches, if we knew where to look), most of which spam the heck out of you with random parameters, forum names, and whatnot.

How do you know that they don't? Google pointed out some longtail results that look bad, and you yourself pointed some out in a previous comment.

Which is why I see no reason to assume that they don't understand or attempt to understand the actual meaning of parameters passed to one of the biggest sites on the internet.

You're misunderstanding what I'm saying. I don't assume that they don't; I'm just saying there's no evidence that they do, and that assertions of wrongdoing based on the belief that they do are just irresponsible speculation. I have no knowledge of what Bing actually does, but neither do the vast majority of the people on the internet who are talking about this, many of whom assume the worst based on a mistaken notion of what's technically necessary to see the results that Google demonstrated.

Post reply on HN