Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

To Break Google’s Monopoly on Search, Make Its Index Public

bloomberg.com

Post reply on HN
Print view View original

261–270 of 630 posts

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#261

Post by Kovah »

dkyc wrote: post #81
noahl wrote: post #61

Okay, this is a relatively serious proposal to require Google to allow API access to its search index, with the premise that it would democratize the search engine ecosystem. There are some issues with the regulations he proposes (you have to allow throttling to prevent DDoS attacks, and you can't let anyone with API access add content to prevent garbage results), but it's roughly feasible. The main problem is, I thi…

> Yes, Google has a huge index, but most queries aren't in the long tail. I'm not quite sure about that. 15% of Google searches per day are unique, as in, Google has never seen them before. [1]. That's quite an insane number. [1] https://searchengineland.com/google-reaffirms-15-searches-ne...

Good point and 15% is indeed much, but the question would be what "unique" means. If it means that the exact same character sequence appeared for the first time, it doesn't mean that the users searches for a term that has never been searched for.

I mean with the newest advantages like machine learning it's more and more possible to _semantically_ link queries. If that's the case, those 15% could become 5% truly unique searches or even less.

"how dumb is trump" and "how dumb is donald trump" are two different searches but they semantically belong together because they mean the same.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#262

Post by tantalor »

ngngngng wrote: post #199
tantalor wrote: post #172

Earlier quoted context omitted.

> but with Google Analytics... No: "Search does not use Google Analytics for ranking" https://www.youtube.com/watch?v=LLmO1GE4GvI

I think you misunderstand what I'm saying. I'm not saying Google analytics will get you ranked higher or lower. Just that Google can use the data from analytics to tell if their results were what the user wanted.

> tell if their results were what the user wanted

That's ranking.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#263

Post by Sander_Marechal »

aantix wrote: post #247
slashdev wrote: post #240

Earlier quoted context omitted.

Storage and bandwidth are cheaper than ever before, people scrape a billion pages for much more mundane purposes these days, even for academic papers. Having a full text index on that is more involved but hardly impossible. You're completely right that it's not at all Google's secret sauce. Bing has clearly indexed much more than that, plus invested a ton in actually returning good results from their index. And still…

The Common Crawl corpus is already available and stored on S3 - so analyzing billions of web pages is literally already available with an AWS account and a simple map reduce job. I'd actually advocate for making public an anonymized list of actual search queries. Domain specific search engines could evolved based on the demand of what has already been searched for.

Anonymizong search queries is extremely hard, if not impossible. See https://en.wikipedia.org/wiki/AOL_search_data_leak for example.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#265

Post by kjeetgill »

dkyc wrote: post #81
noahl wrote: post #61

Okay, this is a relatively serious proposal to require Google to allow API access to its search index, with the premise that it would democratize the search engine ecosystem. There are some issues with the regulations he proposes (you have to allow throttling to prevent DDoS attacks, and you can't let anyone with API access add content to prevent garbage results), but it's roughly feasible. The main problem is, I thi…

> Yes, Google has a huge index, but most queries aren't in the long tail. I'm not quite sure about that. 15% of Google searches per day are unique, as in, Google has never seen them before. [1]. That's quite an insane number. [1] https://searchengineland.com/google-reaffirms-15-searches-ne...

I think they mean that the results are still from the top pages of the internet. They mean long tail of visited pages, not long tail of searches.

A unique search query could still land you on Wikipedia.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#266

Post by scarface74 »

Loughla wrote: post #256
scarface74 wrote: post #126

Earlier quoted context omitted.

Two points. The slap on the wrist from the DOJ didn’t lead to decreasing dominance on Microsoft. Google/Facebook/Apple competing did. A search engine is not life or death. Google doesn’t stop anyone from creating a website and reaching consumers other ways.

>A search engine is not life or death. Google doesn’t stop anyone from creating a website and reaching consumers other ways. As someone outside of the tech bubble, this statement always confuses me. I see it once a week or so on this site. Someone will say 'well, company x is doing y, what's to stop you from just disrupting them and working around?" Because people get their information in consistent, and predictable…

Its not about “disrupting” anyone. There are other forms of advertising than Google - there is Facebook, Amazon, commercials on TV, guerrilla marketing, getting “1000 true fans” and let your product spread organically, etc. if you are completely dependent on Google you are just one algorithm change from becoming invisible and you may need to rethink your business plan. It's kind of silly to start a business with your only method customer acquisition is depending on Google.

I’ve worked for B2B companies that actually had a sales team.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#267

Post by [deleted] »

noahl wrote: post #61

Okay, this is a relatively serious proposal to require Google to allow API access to its search index, with the premise that it would democratize the search engine ecosystem. There are some issues with the regulations he proposes (you have to allow throttling to prevent DDoS attacks, and you can't let anyone with API access add content to prevent garbage results), but it's roughly feasible. The main problem is, I thi…

[deleted]

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#268

Post by mrep »

bodono wrote: post #236
root_axis wrote: post #209

Earlier quoted context omitted.

This is not "research" period. Using one arbitrary search comparison to draw conclusions about the nature of a system that processes billions of queries a day is pretty weak. Additionally, I don't get the same results you do. "hillary clinton e" does not bring up emails, nor does "donald trump e" bring up emails (the first results I see are election, education, england visit, ex wife). I'm not ruling out the possibil…

Try "hillary clinton emai". From a fresh chrome session in NYC I get nothing, not a single autocomplete result. On the other hand "donald trump emai" gets: * donald trump email * donald trump email address * donald trump email list * donald trump email newsletter * donald trump email list signup And just to drive the point home I tried "root_axis emai" and got "root_axis email". Try anyone else and you get similar re…

Seems to go both ways though as I'm not getting any auto completion results for "donald trump stormy daniels" either. I'm guessing they scrub things that are highly sensationalized in the news.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#269

Post by sanderjd »

swebs wrote: post #227
_jal wrote: post #185

Earlier quoted context omitted.

Do you have an example of Vox being dishonest?

https://mediabiasfactcheck.com/vox/

I'm not sure this answers the question. Clearly Vox has a progressive (maybe originally neo-liberal?) editorial bias, but that may or may not mean that they have dishonestly distorted facts. There is a big difference between editorial bias and dishonest reporting!

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#270

Post by anonuser123456 »

I think a distinction needs to be made between monopoly and 'best product'. Search is winner take all in many respects. As a user, I don't particularly care about the second best (because I don't pay).

The best way to fix the situation is to get users to pay. And this could be done. Users can pay negative dollars (e.g. get paid). Why would a search engine want to pay it's users? Why would a retailer want to pay me to look at their ad? Why wouldn't they?

Example:. My wife and I buy a not insignificant amount of stuff from LuLuLemon. So I have a deal for Lululemon. Pay DuckDuck half what you pay Google to show me your stuff. And DuckDuck, pay me half what they pay you. Right now DuckDuck gets zero. Google probably makes 20+ bucks a month off me. In this math, DuckDuck nets 5$!

There is a natural equalibrium here. Advertisers quickly start paying zero to people that don't buy stuff, so users can't farm ads. But relevant customers get discounts.

So now I actually care about making the trade off between second best but effectively cheaper or best but more expensive. E.g. a functioning market.

To do this though, you have to start tracking users in some way. But Google already does this, so the situation doesn't change. Some form of surveillance is required to improve relevance. The difference here is, I can choose who surveils me and to what extent rather than being stick with the winner take all.

Post reply on HN