Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Challenging the Bing It On Challenge

freakonomics.com

Post reply on HN
Print view View original

41–50 of 63 posts

Re: Challenging the Bing It On Challenge

#41

Post by VLM »

dangero wrote: post #32

I don't think even the Freakonomics followup is valid because there's a huge difference between telling someone to run some searches and actual search engine use. Here's what I might type if someone asked me to try their search engine: "Miley Cyrus". The funny thing is that I'm not sure if I would like Google's "Miley Cyrus" results better than Bing's. I'm not sure I could tell the difference. Here's the kind of stuf…

The most surprising thing is there might be any correlation between performance of top end search engines at such widely divergent areas of human experience, low quantity weird software errors and high quantity pointless hollywood blather. Certainly a junky demo search engine could be beaten all around by any top end search engine. But I'm surprised at the top end they are competitive in both areas.

It would be like finding out olympic class weight lifters tend to also be olympic class marathon runners. Or the best commuter vehicle and best 100 ton mining truck happen to be the same vehicle.

Re: Challenging the Bing It On Challenge

#42

Post by GFischer »

dangero wrote: post #32

I don't think even the Freakonomics followup is valid because there's a huge difference between telling someone to run some searches and actual search engine use. Here's what I might type if someone asked me to try their search engine: "Miley Cyrus". The funny thing is that I'm not sure if I would like Google's "Miley Cyrus" results better than Bing's. I'm not sure I could tell the difference. Here's the kind of stuf…

I'm at work, and I wanted to try the challenge, but BingItOn redirects me to Bing.

Maybe it's not available outside the U.S.?

Bing's results are way less relevant to me than Google's, but Google does have the advantage of a lot more personal data, and a specific subdomain for my country (.uy) which often returns more relevant queries (OTOH for programming I often have to force the U.S. website).

That's the reason I don't use DuckDuckGo either, the results are vastly inferior for my use case (even though I want to like it).

Re: Challenging the Bing It On Challenge

#43

Post by BIair »

tl;dr if you take the Bing challenge and make Bing your default search, try other variations on Bing as you would have on Google.

I tried the Bing challenge, and changed my default search to Bing almost a year ago. According to Google I was performing 10k - 20k searches per month, so Bing Rewards was an appealing incentive. Over a years time, those are some decent rewards.

Bing results are very good. The notable exception I've noticed is for technical, geeky searches that are probably most likely to occur with this crowd.

The biggest problem is one of "branding" and confidence. At first I found myself searching Bing, and if I didn't find the result I wanted, I'd switch to Google. The results were often very similar. But instead of returning to Bing to perform another search, I'd do it on Google. As a Google user, when I didn't find the results I wanted, I'd try a new search... I didn't try Bing. When I noticed this and changed my behavior, I became more satisfied with Bing's results.

Re: Challenging the Bing It On Challenge

#45

Post by mattwallaert »

GFischer wrote: post #42
dangero wrote: post #32

I don't think even the Freakonomics followup is valid because there's a huge difference between telling someone to run some searches and actual search engine use. Here's what I might type if someone asked me to try their search engine: "Miley Cyrus". The funny thing is that I'm not sure if I would like Google's "Miley Cyrus" results better than Bing's. I'm not sure I could tell the difference. Here's the kind of stuf…

I'm at work, and I wanted to try the challenge, but BingItOn redirects me to Bing. Maybe it's not available outside the U.S.? Bing's results are way less relevant to me than Google's, but Google does have the advantage of a lot more personal data, and a specific subdomain for my country (.uy) which often returns more relevant queries (OTOH for programming I often have to force the U.S. website). That's the reason I d…

You're correct, Bing It On is US only (although I believe there is also a Chinese version). As for Uruguay, we're always working to improve our international coverage - check back soon!

(Note: I work for Bing.)

Re: Challenging the Bing It On Challenge

#46

Post by mattwallaert »

yid wrote: post #3
pyrocat wrote: post #2

It's great that they did this study. I wish more people would publicly challenge many of the outlandish claims in advertising. But it seems rather hypocritical to knock Microsoft for a sample size of "nearly 1,000" when their own study "obtained 1,008 Bing It On challenge responses from the MTurk platform and narrowed our analysis to 985 respondents who submitted screen shots for 4925 searches."

Not in this case, no. They were trying to replicate Microsoft's results, which entails using a similar sample size.

Except they split the sample into three, thereby reducing the sample size (and power) in thirds. What matters is that there are still large enough to run significance tests on, both in the original claim studies and Ayers' study.

(Note: I work for Bing.)

Re: Challenging the Bing It On Challenge

#47

Post by mattwallaert »

justinsb wrote: post #10

I tried the BingItOn.com challenge: 5/5 for Google for me. But I saw that Microsoft was trying to push me towards "popular" queries (what Miley Cyrus said today about the government shutdown). So I think this is an issue of demographics and sampling: for mass-market 'leisure' queries I could easily see Bing beating Google if that's what Bing is focusing on. Equally, using Mechanical Turk is probably a biased sample (…

That's a remarkably clear-headed analysis; thanks for that. We at Bing don't always get the courtesy of people actually thinking deeply about what we do. =]

(Note: I work for Bing.)

Re: Challenging the Bing It On Challenge

#48

Post by mattwallaert »

rlu wrote: post #30
rlu wrote: post #28

For what it's worth, I'm pretty sure that most of us/the HN crowd can't fairly do the bingiton challenge. At least for me, it's really quite easy to tell them apart just based on subtleties. I.E. Not the quality of the content itself but just how its presented. And if there are ever images involved, it gets even easier because Google and Bing display them completely differently (and honestly I think Bing does a more…

Here's me "blindly" picking Bing 5 times: http://i.imgur.com/NAuhhhV.png This is why I'm skeptical of anyone here saying "I got Bing 100%" or "I got Google 100%". It's extremely easy for us to game.

This is why we don't track the Bing It On challenge itself and instead do lab studies; on the website, it is fairly easy to game if you're trying to. Though every once in awhile, having seen it a million times, I'm still wrong about which one I think is which, particularly since Google changed their results to look more like ours.

(Note: I work for Bing.)

Re: Challenging the Bing It On Challenge

#49

Post by mattwallaert »

sailfast wrote: post #24

While the results returned are of use, Bing It On removes any of the right-hand side Google content pulled from Wikipedia and other databases which differentiate Google results quite a bit more than the Bing results (especially for quick returns on top hits). Selecting only a portion of a page is not really a fair comparison. That said, the effort did get me to take a closer look at the two and evaluate which one I p…

It is getting increasingly hard to do an apples-to-apples comparison (social sidebar, snapshot, etc.). The best we can do is try to narrow down to just the web results, for the moment.

(Note: I work for Bing.)

Re: Challenging the Bing It On Challenge

#50

Post by mattwallaert »

VLM wrote: post #40
coldcode wrote: post #27

Earlier quoted context omitted.

Sample size is immaterial unless you clearly can see and evaluate the criteria under which they were chosen. 8000 of 9000 dentists choose Brand X toothpaste means nothing if they asked 100,000 people and found 8,000 willing to say they liked Brand X. By itself the sample size that is indicated may have little to do with the actual test.

I was going to suggest the same thing, that given about 1/5 of people use Bing (why?) then you simply sample 5000 people, select the 1000 or so Bing users... However, it turns out there is an easier way. Simply select your sample from existing heavy Bing users. Oddly enough, they'll probably prefer Bing.

And this is precisely why a 3rd party does our testing: so that these sort of manipulations don't creep in.

(Note: I work for Bing.)

Post reply on HN