To Break Google’s Monopoly on Search, Make Its Index Public • Hacker News Board

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#361

Post by nojvek » Mon, Jul 15, 2019, 4:59 PM UTC

nostrademons wrote: post #309

Ex-Google-Search engineer here, having also done some projects since leaving that involve data-mining publicly-available web documents. This proposal won't do very much. Indexing is the (relatively) easy part of building a search engine. CommonCrawl already indexes the top 3B+ pages on the web and makes it freely available on AWS. It costs about $50 to grep over it, $800 or so to run a moderately complex Hadoop job.…

This ^ times a 1000.

Google simply has the best search product. They invest in it like crazy.

I’ve tried bing multiple times. It’s slow, it spams msn ads in your face on the homepage. Microsoft just doesn’t get the value of a clean UX.

DuckDuckGo results are pretty irrelevant the last time I tried them. There is nothing that comes close to their usability. To make the switchover, it has to be much much better than Google. Chances are that if something is, Google will buy them.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#362

Post by ahallock » Mon, Jul 15, 2019, 5:02 PM UTC

An API is not the same as making the index public. I don't think the author has a grasp on all the technical details.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#363

Post by AznHisoka » Mon, Jul 15, 2019, 5:03 PM UTC

colinmhayes wrote: post #197

AznHisoka wrote: post #187

Earlier quoted context omitted.

I'd wager any startup that tries to crawl a few sites like Amazon, Yelp, Linkedin, etc will be blocked. Google, however gets a pass because they're Google. So yes, I believe their huge index, and ability to crawl any site at will is a huge, huge advantage for them.

I built a search engine that was able to crawl Amazon and Yelp. The toughest sites were reddit and facebook.

at scale? millions of pages a week? And now? I wrote a crawler that could crawl Amazon as early as a year ago too, but now it doesn't work.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#364

Post by root_axis » Mon, Jul 15, 2019, 5:05 PM UTC

bodono wrote: post #352

root_axis wrote: post #330

Earlier quoted context omitted.

When I type "Donald Trump R" (or Ru or Russ etc) no autocomplete results contain the word Russia despite plenty of news coverage. When I type "Donald Trump Epstein" no autocomplete results. Must be a conspiracy by google to protect the president... or drawing conclusions based on individually cherry-picked autocomplete results is like drawing conclusions from numerology.

The same thing happens with Russia for "barack obama rus" "hillary clinton rus", "george bush rus", etc. - none autocomplete for me despite the fact that there was lots of news items relating those politicians and russia during their careers. However Clinton is the only one that doesn't appear to autocomplete for email, suggesting that it is specific to her. When I type "donald trump epstei" I get "theo epstein donal…

None of those people had a gigantic Russia scandal though, Trump did, so you must still account for this unexplained aberration. If anything, the Trump/Russia scandal had more coverage than the Hillary e-mail scandal, so it's an even more difficult aberration to explain.

Also, if I type "hillary e" I get "Hillary Emails PDF" as an autocomplete suggestion. If I type "clinton e" I get several email suggestions "clinton email PDF" ,"clinton email film", "clinton email FOIA", "clinton email download".

Please include these confounding results in your analysis

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#365

Post by scott_s » Mon, Jul 15, 2019, 5:05 PM UTC

buzzerbetrayed wrote: post #331

scott_s wrote: post #289

Earlier quoted context omitted.

I got curious and tried the names of a bunch of public figures. Some of " e" yielded "email" as the suggestion. But these did not: elizabeth holmes, tom jones, tom cruise, brad pitt, gwyneth paltrow, roger federer, will smith, jimmy carter. Since Hillary Clinton is not unique, then it's not proof that her results are treated differently.

Why would you expect “brad Pitt email” to be something that auto completes? You would, on the other hand expect “Hillary Clinton email” to auto complete because there was a huge controversy about it. I’m not saying google is manipulating auto complete intentionally (though they might be), I’m just saying your counter examples are irrelevant. It would be like “Donald trump Russia” NOT auto completing then someone sayi…

The poster claimed Hillary Clinton was unique, meaning the only person that applied to. For me, she was not. Since she's not unique, then her being unique can't be used as evidence.

Claiming that she's unusual, since you expect it to work for her based on stories written about her is a different claim.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#366

Post by henvic » Mon, Jul 15, 2019, 5:05 PM UTC

There is no barrier of entry to indexing the web. It is antiethical to force Google to make its index public through regulation. Google has its problems and we should attack them but not like this. It is not a monopoly. It is just the largest player by attending public wishes.

Let other indexes such as DuckDuckGo win through the market.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#367

Post by [deleted] » Mon, Jul 15, 2019, 5:07 PM UTC

dehrmann wrote: post #280

tryptophan wrote: post #103

Earlier quoted context omitted.

>2) 20 years of experience fighting SEO spam. Tangential - but does anyone else feel that google results are useless a lot of the time? If you search for something, you will get 100% SEO optimized shitty ad-ridden blog/commercial pages giving surface level info about what you searched about. I find for programming/IT topics its pretty good, but for other topics it is horrible. Unless you are very specific with your s…

w3schools still outranks mdn a lot.

[deleted]

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#368

Post by eaenki » Mon, Jul 15, 2019, 5:09 PM UTC

toast0 wrote: post #251

eaenki wrote: post #100

Earlier quoted context omitted.

Ye and anyway you could return x10 worse results than Google's current results and still become the new dominant search engine if you've got infinite $B a year to outbid Google to be the default on browsers and operating systems and a big salesforce to onboard the advertisers. Man this is the exact playbook they used to become the search engine in the first place. They literally were Yahoo's search bar at some point…

Microsoft has infinite money, pays users with rewards directly, and has barely gotten any marketshare with Bing. In order to compete, you would need good results and good performance for your organic results and your ad program in addition to a huge budget for traffic aquisition and patience and ability to execute on strategy consistently over multiple years. Also, keep in mind that people will judge based on perceiv…

As you can clearly see for yourself, Microsoft is not as interested in acquiring market share for Bing. Not as much as to outbid Google's contracts with Firefox and Apple to be the default. Bing is clearly a second thought for Microsoft even tho it's big. That's probably because they wouldn't get the same bang for their buck anyway. They have far less advertisers on board. Anyway, according to Microsoft, they own 33% of the US market. That is quite the market share. Especially once you consider it has mostly been acquired thru sneaky toolbar installations and IE.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#369

Post by bodono » Mon, Jul 15, 2019, 5:13 PM UTC

root_axis wrote: post #364

bodono wrote: post #352

Earlier quoted context omitted.

The same thing happens with Russia for "barack obama rus" "hillary clinton rus", "george bush rus", etc. - none autocomplete for me despite the fact that there was lots of news items relating those politicians and russia during their careers. However Clinton is the only one that doesn't appear to autocomplete for email, suggesting that it is specific to her. When I type "donald trump epstei" I get "theo epstein donal…

None of those people had a gigantic Russia scandal though, Trump did, so you must still account for this unexplained aberration. If anything, the Trump/Russia scandal had more coverage than the Hillary e-mail scandal, so it's an even more difficult aberration to explain. Also, if I type "hillary e" I get "Hillary Emails PDF" as an autocomplete suggestion. If I type "clinton e" I get several email suggestions "clinton…

Actually both Obama and Clinton had major stories relating to Russia including the failed 'reset', the annexation of Crimea during the Obama administration, obama's "red line" of using chemical weapons in syria which was circumvented by russia, and Obama telling Mitt Romney that Russia was not a threat in the 2012 presidential debates. So yes, they have huge stories relating to russia. And your autocomplete results just bolster the evidence that 'hillary clinton email' has been made a special case and is not organic!

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#370

Post by bduerst » Mon, Jul 15, 2019, 5:15 PM UTC

briandear wrote: post #138

nova22033 wrote: post #106

Caveat: The author is not a technologists Robert Epstein (born June 19, 1953) is an American psychologist, professor, author, and journalist. He earned his Ph.D. in psychology at Harvard University in 1981, was editor in chief of Psychology Today, He has also made some questionable claims about google manipulating search results to favor Hillary Clinton. https://en.wikipedia.org/wiki/Robert_Epstein#cite_note-15 His r…

> He has also made some questionable claims about google manipulating search results to favor Hillary Clinton. Despite it being off topic, can we define why those claims are questionable? Is their data proving those claims wrong? Because with all the Google political controversies over the past few years, and given the political donation history of Google employees, it’s highly plausible that search results are manip…

The "Manipulating instant search results in favor of Hillary Clinton" claim has been independently debunked and anyone still standing behind it are only signalling their technical illiteracy and/or political agenda for playing a victim card. [1][2][3]

That's not really off-topic - The fact that the author still supports the claim calls into question their ability to make further claims about the subject.

[1] https://www.snopes.com/fact-check/google-manipulate-hillary-...

[2] https://www.vox.com/2016/6/10/11903028/hillary-clinton-googl...

[3] https://mashable.com/2016/06/10/clinton-google-search/