Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

To Break Google’s Monopoly on Search, Make Its Index Public

bloomberg.com

Post reply on HN
Print view View original

241–250 of 630 posts

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#241

Post by gtfratteus »

This is idiotic for three reasons:

1. Google is not a monopoly in search. Bing and DuckDuckGo are perfectly acceptable alternatives. 2. Google search is so much more than its index. Opening up the index would do little to nothing to improve the competition. 3. Even if the author did know what he was talking about, his proposal would boil down to saying McDonalds needs to release the recipe for their secret sauce because they sell too many burgers. It's an ethically repugnant proposal.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#242

Post by debatem1 »

GordonS wrote: post #200
detritus wrote: post #78

Earlier quoted context omitted.

> Indexing the top billion pages or so won't take as long as people think. This is what makes me wonder why we don't have a LOT of competing search engines. Perhaps i'm vastly under-estimating the technology and difficulty (I could well be - it's not my domain) but it surely it can't be THAT hard to spawn Google-like weighted crawl-based search results? It's a long-since solved problem - heck, pageRank's first iterat…

Aside from the quality issues that others have already mentioned, I think that simply gaining traction for a new search engine is incredibly difficult - people typically use whatever is the default in their browser, or/and Google/Baidu/Yandex (which are surely the best known in their respective regions). Consider DuckDuckGo, which sells itself on privacy, but after more than a decade has only 0.18% market share. With…

I don't think this is correct. For years, the #3 search query on Bing in the US was "Google", and globally it used to be a double-digit percentage of all Bing queries. That suggests to me that people with a default Bing search engine had learned in droves to click their way to the preferred engine regardless of what the default was, and did so without being technically skilled enough to change the default once and for all. I don't know how large a group the latter is, but it seems hard to argue that the two together are small.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#243

Post by sprash »

There is already commoncrawl[1]. Nobody managed to build a useful search engine with it. Apparently it takes much more to produce meaningful search results. Making googles index public won't solve anything. What we really need is a viable competitor to google.

[1]: https://commoncrawl.org/

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#244

Post by Sebguer »

djsumdog wrote: post #173
briandear wrote: post #138

Earlier quoted context omitted.

> He has also made some questionable claims about google manipulating search results to favor Hillary Clinton. Despite it being off topic, can we define why those claims are questionable? Is their data proving those claims wrong? Because with all the Google political controversies over the past few years, and given the political donation history of Google employees, it’s highly plausible that search results are manip…

What about Project Veritas? People claim the statements by Google employed were taken out of context, but I've gone back, listened to them, looked at the videos, and it's hard to think in what context anything they said is acceptable. Even if the specific engineers and managers in the video clips don't have the level of authority to make the changes they're talking about, it's still chilling that their attitude could…

Project Veritas has historically operated by guiding people into saying something ridiculous, either by themselves acting ridiculous (and convincing the person they're talking to that they're crazy) or just driving the conversation to ridiculous areas.

Maybe this Google 'expose' is the first time they're not guilty of that, but anyone who still finds them credible after their last several blatant mischaracterizations is far more forgiving than I would be.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#245

Post by seisvelas »

This seems a bit silly - it's not like Google's search results are that much better than Bing or DuckDuckGo (or better at all most of the time). Google has Google Colab, Chrome, and Android, and the whole Docs ecosystem, and they've integrated those things together pretty well while still being loosely coupled enough to switch out any part of the ecosystem with a competitor's product.

Is there a reason to break up Google other than that they are doing well? Other search engines seem to have no problem establishing their own niche and doing well.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#246

Post by Excel_Wizard »

root_axis wrote: post #209
ar-jan wrote: post #170

Earlier quoted context omitted.

Google's claim that the algorithm is generic is demonstrably false. Type in "hillary clinton e" and there is no suggestion for "email", type "donald trump e" and email is the first suggestion. Given the news content that we know is out there, that can only be the result of adjusting the results for clinton specifically (if anything, we would not expect "email" to be autocompleted for trump). This is not research that…

This is not "research" period. Using one arbitrary search comparison to draw conclusions about the nature of a system that processes billions of queries a day is pretty weak. Additionally, I don't get the same results you do. "hillary clinton e" does not bring up emails, nor does "donald trump e" bring up emails (the first results I see are election, education, england visit, ex wife). I'm not ruling out the possibil…

"hillary clinton emails" is a topic that is widely published around the web. Autocomplete should pick up on this and recommend it. I even went to the trouble of typing "hillary clinton emai" and no autocomplete suggestions were brought up.

It is a rather suspicious result. Suspicious enough that it is hard not to imagine a deliberate act is behind it. I admit that I don't have any proof.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#247

Post by aantix »

slashdev wrote: post #240
noahl wrote: post #61

Okay, this is a relatively serious proposal to require Google to allow API access to its search index, with the premise that it would democratize the search engine ecosystem. There are some issues with the regulations he proposes (you have to allow throttling to prevent DDoS attacks, and you can't let anyone with API access add content to prevent garbage results), but it's roughly feasible. The main problem is, I thi…

Storage and bandwidth are cheaper than ever before, people scrape a billion pages for much more mundane purposes these days, even for academic papers. Having a full text index on that is more involved but hardly impossible. You're completely right that it's not at all Google's secret sauce. Bing has clearly indexed much more than that, plus invested a ton in actually returning good results from their index. And still…

The Common Crawl corpus is already available and stored on S3 - so analyzing billions of web pages is literally already available with an AWS account and a simple map reduce job.

I'd actually advocate for making public an anonymized list of actual search queries.

Domain specific search engines could evolved based on the demand of what has already been searched for.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#248

Post by wutbrodo »

aurbano wrote: post #229
GordonS wrote: post #195

Earlier quoted context omitted.

Heh, yes, they do. Which is a reminder that devs are not "typical" users. As a developer, I search using keywords; for example, if I was looking for property for sale in Inverness, I might search for "property Inverness", whereas I've seen and heard "typical" users use something like "find me a 2 bedroom house with a garden for sale in the North of Inverness" - much more verbose, and containing stop words and phrases…

I do the same as you, but was just thinking that if most users search using full sentences then Google will spend most effort optimizing for that, so maybe we're the ones getting the worse results?

No, the optimization they do for the low-quality query is more than balanced out by the higher clarity and relevance of a well-phrased query. There are often extraneous words that aren't simple stop words, and they're not 100% successful at removing these extraneous ones.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#249

Post by arch-ninja »

tomsmeding wrote: post #40
arch-ninja wrote: post #23

No sure if relevant but I've hated google's search for a unique reason: I find it horribly slow, especially compared to what it used to be. It's so slow, I've designed + partially implemented an alternative for my own use: https://github.com/Jeffrey-P-McAteer/dindex I've only tested with 1000 records, but the query times are all <200ms.

How does your performance scale? If it scales linearly in records, you're going to have trouble if you're only below 200ms on 1k records. If it scales logarithmically, there are still a couple orders of magnitude between a comprehensive index like Google's and your 1k record index, which is still going to give you trouble. In short, 200ms is slow, not fast. EDIT: Just to clarify, I'm not saying it's necessarily easy…

Thanks for the insight; indexing the entire web is not a goal of dindex. Instead, it's designed to work like DNS where you can host your own server + have it federate queries to other servers if it doesn't have records which match a query.

Basically it moves control to the client; if the client wants to query servers across the globe there's nothing stopping them, but by default you only query servers in your config file. Organizations like the new york times, universities, and governments would host servers for the same reason they host their own email infrastructure.

It also moves a ton of control to publishers; dindex doesn't crawl the web unless a client explicitly wants to.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#250

Post by sidibe »

rpiguy wrote: post #88
taf2 wrote: post #54

So although everyone likes to believe google is a monopoly it’s far from it. You have choices- bing, biadu , yandex, DuckDuckGo... there is also nothing about googles search position that prevents you from building a competitor. What we do have is peter thiel backing an administration that’s anti google, Russia, China that are anti google. Why? it’s a source of truth that challenges their lies. We also have an emerge…

Google is a profit-seeking corporation that needs to expand into China to generate more money and bends the knee as much as it can to make inroads there. Far from being a source of truth they follow all Chinese censorship laws (why, because they would be blocked from operating there and unable to make money off of the Chinese). China likes everything about Google other than the fact that it is not a domestic corporat…

>Google is not the victim of misinformation.

>Google is a profit-seeking corporation that needs to expand into China to generate more money and bends the knee as much as it can to make inroads there. Far from being a source of truth they follow all Chinese censorship laws (why, because they would be blocked from operating there and unable to make money off of the Chinese). China likes everything about Google other than the fact that it is not a domestic corporation.

Whew the irony

Post reply on HN