Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bing search results showing up in Google

jacquesmattheij.com

Post reply on HN
Print view View original

111–120 of 131 posts

Re: Bing search results showing up in Google

#111

Post by gojomo »

mukyu wrote: post #107
gojomo wrote: post #103

Earlier quoted context omitted.

Google has said they do not use clickstream data for ranking from Google toolbar. Please provide a reference. I've been looking for this statement and haven't found it. When Googlers are directly asked, they pointedly don't answer or say they don't know. Amit Singhal's statement was carefully worded to be ambiguous on this matter, and Google has apparently confirmed that page-load-time data (at the very least) from t…

You are being disingenuous. That you cannot find them explicitly denying something does not therefore make it true nor provide any evidence that it is true. “Absolutely not. The PageRank feature sends back URLs, but we’ve never used those URLs or data to put any results on Google’s results page. We do not do that, and we will not do that,” said Singhal. http://www.toprankblog.com/2006/04/matt-cutts-on-toolbar-dat...…

"Put any results" is vague, perhaps intentionally finessed, language – as I (and the Sullivan quote) already highlighted in the grandparent comment. It could mean, especially in the context of the Bing allegations, that Google Toolbar data never adds a new URL to the index or a result-set, but is still used for relative ranking of already-known URLs.

In the link you provide, Matt Cutts says: "I’m not going to say definitively that Google doesn’t/won’t use toolbar data (or other signals) in ranking." And: "I’m not going to say whether Google uses a particular signal in our ranking." Cutts simply says Toolbar data could be problematic because it could be gamed. Well, links can too – didn't stop Google from building its empire on that impure signal. This seems to me more of the same finesse that creates the impression of a denial without a denial.

Further, it's clear from previous Google statements that page-load-timing from the Toolbar is used to affect rankings. That alone invalidates the 'strong' interpretation of Amit Singhal's statement. So Singhal means something other than "Toolbar data never effects search rankings". What does he mean? Just requoting that vague statement doesn't clear anything up.

And since everything Google does with this data is a closely-guarded secret, how can we be sure of anything, short of awaiting (and then trusting) definitive Google statements? And I can't yet find any clear statement about Toolbar data usage – even though lots of people seem to think they have seen them. (I think general warm feelings towards Google are creating this mistaken impression.)

I don't expect a clear statement; I believe Toolbar clicktrails are a big part of Google' secret sauce. But it means Google could be using equivalent techniques to Bing's, and simply have better filters against the blatant dominance of a single website or only 20 clickers on any result sets.

Re: Bing search results showing up in Google

#112

Post by prodigal_erik »

anonymous246 wrote: post #91
haberman wrote: post #80

Earlier quoted context omitted.

> Let me ask you and others this: Is the following robots.txt supposed to exclude all pages from my domain from showing up in Google results? I believe that robots.txt is a way to prevent your site from being crawled by a robot, but it is not a blacklist against your site appearing in Google search results if it finds a link to your page on a site that does allow robots. Check out this page: http://www.google.com/sup…

Great. This link helps. Google's site basically says that in addition to specifically excluding their crawler via robots.txt, I also have MANUALLY submit a request to them. As I said in a reply below, this is nonsensical and Bing is being more reasonable, but I'll grit my teeth and do it since Google has more power here. It definitely seems like Google is exploiting a loophole in spirit of the definition of robots.tx…

As I recall, the spirit of robots.txt was not about appropriateness of search results so much as "this URL space can generate an unbounded graph, please don't DoS my server by trying to exhaustively traverse it."

Re: Bing search results showing up in Google

#113

Post by haberman »

zaidf wrote: post #109
haberman wrote: post #106

Earlier quoted context omitted.

Even if absolutely everything you say is true, even if "clickstream data belongs to the users" and the users can "share" it with Microsoft, it's still the case that Microsoft is riding on the backs of Google's engineers. And as I said in my original message, maybe that's even a smart business move. But let's call a spade a spade. Microsoft is copying Google search results. Is that really what you want to be sticking…

Microsoft is copying Google search results. Clickstream data != copying search results.

This is the lie that you keep telling yourselves. Rationalize it however you want, but show a normal person the screenshots on this page and ask them if there is copying going on: http://www.mattcutts.com/blog/google-bing/

Maybe you call it "clickstream," maybe you think this has something to do with the user (even though the users had no idea they were aiding in this), this web of rationalizations cannot get around this fact:

Bing puts things in its search index that came directly from Google's algorithm.

Re: Bing search results showing up in Google

#114

Post by prodigal_erik »

xilun0 wrote: post #61
jedsmith wrote: post #46

Earlier quoted context omitted.

> So it IS kinda a big deal that they're disrespecting the robots.txt and listing those pages anyway. Did you RMFC? The robots.txt doesn't match what is indexed. As has been exhaustively pointed out in this thread by more than a few commenters, "Search" != "search". On that very search results page, we see #1 which is: > OLAC search - Bing OLAC is an unrelated site, and at one point it apparently linked to "www.bing.…

> "Search" != "search" This is highly ridiculous. HTTP URLs are not mandated to be case-sensitive (though it's recommended), and clearly lots of sites use them in a case-insensitive way. Robots should either consider robots.txt in a case-insensitive way (even if I'm conscious that lot of them, including major ones, currently don't do that, which is precisely what I consider to be a problem, and this is supported by w…

An HTTP client must always assume URL paths and query strings are case-sensitive. You can't rely on the bing.com web servers always returning the same resource for http://www.bing.com/search?q=test and http://www.bing.com/Search?q=test just because the default filesystem for their OS is case-insensitive today. Actually it's unlikely the main entry point for their search engine is a file named "search" in some top-level directory. The only way for an author to tell you many URLs are equivalent is to use "301 Moved Permanently" responses to the canonical one from all the others, and Bing doesn't do that.

Re: Bing search results showing up in Google

#115

Post by seanos »

haberman wrote: post #90
seanos wrote: post #66

Earlier quoted context omitted.

"The bottom line is that Bing's index contains associations that it could never have figured out if Google hadn't figured them out first." Actually, it is the users who are figuring out what is relevant not Google since a) Bing are collecting data on the links they click regardless of where Google ranks it (could be on the 20th page of Google search results for all Bing care) and b) Bing are collecting this data from…

> Actually, it is the users who are figuring out what is relevant not Google When you do a Google search, Google chooses ten URLs from over 1 trillion that are in its index. Creating this list takes teams of full-time engineers and data-centers full of servers that crawl the internet constantly. The user spends, on average, less than ten seconds choosing 1 out of those 10 URLs. In a small minority of cases, they'll c…

Individually, it negligible. Cumulatively, the time spent by the many thousands (millions?) of users who are clicking links they deem relevant for queries all over the web (Bing are collecting data from all sites) is huge...

My point is that the assertion that Google are doing all the work in creating the data used by Bing is clearly false.

Re: Bing search results showing up in Google

#116

Post by mukyu »

gojomo wrote: post #111
mukyu wrote: post #107

Earlier quoted context omitted.

You are being disingenuous. That you cannot find them explicitly denying something does not therefore make it true nor provide any evidence that it is true. “Absolutely not. The PageRank feature sends back URLs, but we’ve never used those URLs or data to put any results on Google’s results page. We do not do that, and we will not do that,” said Singhal. http://www.toprankblog.com/2006/04/matt-cutts-on-toolbar-dat...…

"Put any results" is vague, perhaps intentionally finessed, language – as I (and the Sullivan quote) already highlighted in the grandparent comment. It could mean, especially in the context of the Bing allegations, that Google Toolbar data never adds a new URL to the index or a result-set, but is still used for relative ranking of already-known URLs. In the link you provide, Matt Cutts says: "I’m not going to say def…

I do not think it would matter if Sergey and Page personally delivered a stone slab engraved with a statement that Google does not use their toolbar's clicks. As you say, you would have to rely on trusting their statements. You seem to require extraordinary evidence that they do not and have no evidence to back up that they actually do use it, and yet cling to belief that they do.

Matt spent 5 paragraphs on the subject, and yet you have laser focus on one statement that is not even contradicted by other evidence and try to derive whatever you want from it.

Here are some other comments from the panel at bigthink, but I'm sure you will find enough wiggle in them to assert that Google faked the moon landing.

Matt Cutts: "I'm not sure that users realize...when they search on Google and click... those results appear--those clicks appear to be encrypted and sent to Microsoft which then appear to be used in Google's [sic] rankings?"

[stuff about EULAs]

Harry Shum: "Everyone does this Matt you know.."

Matt: "Google... I want to categorically deny that Google does this."

Harry: "That Google does what?"

Matt: "We don't use clicks on Bing's users in Google's rankings."

http://bigthink.com/series/62#!selected_item=4845 around 25:00

Re: Bing search results showing up in Google

#117

Post by zmmmmm »

unp3rs0n wrote: post #33
jedsmith wrote: post #13

With respect to the author, the conclusion here is very flawed. If you search for Bing in Google, you get Bing all over page 1. If you search for Google in Bing, you get Google all over page 1. That's not the result of Google capturing click stream data from Google Chrome and copying Bing's results, nor is it the result of Microsoft capturing click stream data from IE8 and copying Google's results. That's just the na…

I don't understand why everyone is using the term "copying the results". I think what Bing did was very smart, they incorporated user clickstream data. One could accuse this method of walking a thin line morally, but I suspect that Google's accusation wouldn't have stood any water as a lawsuit.

It's a subtle point, but the whole point of this issue is that Bing actually does copy more than just "clickstream data". Think about what "clickstream" data is - user clicks a link, the link and the text of the link get sent to Microsoft. But in Google's text the search term was not in the link text, nor was it in the target page the user went to. So then how does the search term end up in Bing's index? They have to get it from somewhere - where? The logical conclusion is they get it from the URL of the page the click is performed on, or from the referrer header when the link is clicked (essentially these are the same thing). But how do you do that generically? The search URL contains the query terms in a format that is unique to Google. The only way Bing can be seeing it is if they deliberately are parsing out the Google search terms by specifically targeting how Google encodes them.

So rather than just generically "incorporating clickstream data" this is actually using a special procedure to extract search terms from clicks that are determined to be google searches and then putting those search terms into the bing index.

Re: Bing search results showing up in Google

#118

Post by zaidf »

haberman wrote: post #113
zaidf wrote: post #109

Earlier quoted context omitted.

Microsoft is copying Google search results. Clickstream data != copying search results.

This is the lie that you keep telling yourselves. Rationalize it however you want, but show a normal person the screenshots on this page and ask them if there is copying going on: http://www.mattcutts.com/blog/google-bing/ Maybe you call it "clickstream," maybe you think this has something to do with the user (even though the users had no idea they were aiding in this), this web of rationalizations cannot get around…

...that came directly from Google's algorithm

That is a lie.

1. User searches for specific string 2. Google pulls up the results 3. User clicks on the result

Bing makes a corelation between the user's search string and the URL they end up on and takes a note of it. There is very little algorithm involved here because there is only one listing for that string. The algorithm is primarily used to rank pages. In order for Google to convince me that Bing is copying them, they would need to show a consistent before and after of Bing copying the search results and its order. Emphasis on resultS--in plural.

Neither are present in the Google honeypot. There is only one search result returned; and there is no question of the order of search results cuz of that. Two items central to the algorithm are entirely missing.

Going back to what Microsoft does copy:

1. it copies the user's input--imo this belongs to the user

2. it does not copy the order of search results nor scrape the returned search results returned by google

3. it copies the url the user ends up on, which is effectively a user's browsing history that the user has opted in to share with Bing.

Re: Bing search results showing up in Google

#119

Post by haberman »

seanos wrote: post #115
haberman wrote: post #90

Earlier quoted context omitted.

> Actually, it is the users who are figuring out what is relevant not Google When you do a Google search, Google chooses ten URLs from over 1 trillion that are in its index. Creating this list takes teams of full-time engineers and data-centers full of servers that crawl the internet constantly. The user spends, on average, less than ten seconds choosing 1 out of those 10 URLs. In a small minority of cases, they'll c…

Individually, it negligible. Cumulatively, the time spent by the many thousands (millions?) of users who are clicking links they deem relevant for queries all over the web (Bing are collecting data from all sites) is huge... My point is that the assertion that Google are doing all the work in creating the data used by Bing is clearly false.

> My point is that the assertion that Google are doing all the work in creating the data used by Bing is clearly false.

Clearly it's not 100%, but what percentage is it?

The user ends up visiting 1 out 1T possible URLs.

Google narrowed it down from 1T to 10 URLs (11 orders of magnitude).

The user narrowed it down from 10 to 1 URLs (1 order of magnitude).

Even if you measure this logarithmically, Google did 91% of the work, and the user did 9%. And the Microsoft people are arguing that it's totally justified because of that 9%.

Or you could measure it linearly, and argue that Google did 99.999999999% of the work because that's the percentage of the results they eliminated.

Either way, this is an extremely weak justification for taking a result of which 91% came from your direct competitor (when measured in the most charitable way).

Re: Bing search results showing up in Google

#120

Post by mc32 »

Matt_Cutts wrote: post #108

The two major issues in this article were: - Google can see and return links to pages without crawling them. I made a video and a blog post about this a while ago: http://www.mattcutts.com/blog/robots-txt-remove-url/ - URL paths are case sensitive. Bing blocks /search in its robots.txt but not /Search. That's how the /Search urls got crawled. In a later edit, the author suggests "It would be fairly trivial for bots t…

>"...but I wish the author had done more research instead of going for a gotcha."

That's a bit of irony right there, isn't it?

--wow, Matt, I guess it cuts both ways, eh?

Post reply on HN