Live data from Hacker News

To Break Google’s Monopoly on Search, Make Its Index Public

bloomberg.com

121–130 of 630 posts

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#121
> Google is especially worrisome because it has maintained an unopposed monopoly on search worldwide for nearly a decade. It controls 92 percent of search, with the next largest competitor, Microsoft’s Bing, drawing only 2.5%.

1. The definition of the word "monopoly" is not applicable to this situation.

2. Property expropriation reduces the competition within the society, while also being unethical.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#122
post #61

Okay, this is a relatively serious proposal to require Google to allow API access to its search index, with the premise that it would democratize the search engine ecosystem. There are some issues with the regulations he proposes (you have to allow throttling to prevent DDoS attacks, and you can't let anyone with API access add content to prevent garbage results), but it's roughly feasible. The main problem is, I thi…

> Indexing the top billion pages or so won't take as long as people think. This is what makes me wonder why we don't have a LOT of competing search engines. Perhaps i'm vastly under-estimating the technology and difficulty (I could well be - it's not my domain) but it surely it can't be THAT hard to spawn Google-like weighted crawl-based search results? It's a long-since solved problem - heck, pageRank's first iterat…

It is because people just stick with their best usually instead of using a variety of search engines. It becomes rather winner takes all.

Google for general search. Duckduckgo fir general if you want something a bit more private but not extreme enough to run your own spiders. Bing mostly for porn search - not being snarky some people do consider it to have better results.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#123
post #61

Okay, this is a relatively serious proposal to require Google to allow API access to its search index, with the premise that it would democratize the search engine ecosystem. There are some issues with the regulations he proposes (you have to allow throttling to prevent DDoS attacks, and you can't let anyone with API access add content to prevent garbage results), but it's roughly feasible. The main problem is, I thi…

>2) 20 years of experience fighting SEO spam. Tangential - but does anyone else feel that google results are useless a lot of the time? If you search for something, you will get 100% SEO optimized shitty ad-ridden blog/commercial pages giving surface level info about what you searched about. I find for programming/IT topics its pretty good, but for other topics it is horrible. Unless you are very specific with your s…

Google signed an armistice in the Great Spamsite War some time around '08 or '09, to the effect that spam can have all the search results aside from those pointing at a few top, trusted sites, so long as they provide any content at all. Bad content is fine. Farmed content is fine. Content that was probably machine-generated is fine. Just content. Play the game, make sure your markov chain article generator or mechanical turks post every day, throw some Google ads on your page, and G will happily put your spamsite garbage at result #3.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#124

Caveat: The author is not a technologists Robert Epstein (born June 19, 1953) is an American psychologist, professor, author, and journalist. He earned his Ph.D. in psychology at Harvard University in 1981, was editor in chief of Psychology Today, He has also made some questionable claims about google manipulating search results to favor Hillary Clinton. https://en.wikipedia.org/wiki/Robert_Epstein#cite_note-15 His r…

The comments he made are not just questionable, they're outright wrong (and a great example of the problem with cherry-picking data):

https://www.vox.com/2016/6/10/11903028/hillary-clinton-googl...

https://www.politifact.com/punditfact/statements/2016/jun/23...

(Disclosure: I work at Google, but this opinion is my own)

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#125
post #61

Okay, this is a relatively serious proposal to require Google to allow API access to its search index, with the premise that it would democratize the search engine ecosystem. There are some issues with the regulations he proposes (you have to allow throttling to prevent DDoS attacks, and you can't let anyone with API access add content to prevent garbage results), but it's roughly feasible. The main problem is, I thi…

>2) 20 years of experience fighting SEO spam. Tangential - but does anyone else feel that google results are useless a lot of the time? If you search for something, you will get 100% SEO optimized shitty ad-ridden blog/commercial pages giving surface level info about what you searched about. I find for programming/IT topics its pretty good, but for other topics it is horrible. Unless you are very specific with your s…

It has gotten better over the years in some ways even if it feels like it also got worse. I recall pages of "ads and useful lookimh search result keywords" being more common in the past.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#126

Earlier quoted context omitted.

And when it all came down to it, it didn’t matter. Chrome didn’t become dominant on the desktop because of government intervention and Safari didn’t become important because of DOJ either. Apple and Google competed. Government intervention is rarely the right answer. A bunch of people who are both beholden to lobbyists and ignorant to technology won’t produce the outcome people think it will. Besides, the last thing…

That which the market does not naturally provide... does one just morally expect?

Two points.

The slap on the wrist from the DOJ didn’t lead to decreasing dominance on Microsoft. Google/Facebook/Apple competing did.

A search engine is not life or death. Google doesn’t stop anyone from creating a website and reaching consumers other ways.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#127
Fun timing to read this, this last weekend I was playing around with making my own search engine to understand better how ElasticSearch and Lucene work.

It occurred to me that the two most powerful things Google has to work with are records of clicks, and the time users spent on the webpages Google returned. I've argued against Google monopoly before because I can throw together a web crawler and search engine in a weekend, so it's not like it's a hard market to enter.

> According to W3Techs, Google Analytics is being used by 52.9 percent of all websites on the internet

This is the real problem though. When a search engine sees a new query, it uses everything it's got to assert which pages the user wants, but with Google Analytics, they can test their assertions constantly to see if a user actually wanted that web page. Then your future queries could be compared against previous queries that were validated by a user spending several minutes of active time on the returned page.

I'm sure Google's algorithm is great and all, but I really think this is what sets them apart.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#128
post #115

Making the index available would be very valuable for scientific research. when I was a googler, we used the index (and google scholar's index) to do all sorts of interesting science projects (DNA search, gene search, etc) But we couldn't publish the results for various reasons. If the index (or a fragment of it) was available sitting in parquet files (or possibly a better format for indices) you could easily sit aro…

There are tons of projects with serious sizable open source index, for ex, http://commoncrawl.org/. Even crawling 15B pages on AWS is not super expensive on your own using well established OSS tool chain. Bing already provides APIs if you don’t want to do crawling.

Web index, while important, is by no means most important part of search engine (which I had say is relevance). The OP article is written by someone who has little clue that search engine involves massive amount of technologies besides index and even relevance - everything from spell correction to recommendations to query rewriting to answers to segment specific searches like images/video/maps/local/product/news/entertainment/events, so on and on and on. This is above and beyond the gigantic infrastructure needed to run all these at scale, speed and cost effectively. All of these needs to work harmoniously with each other and designed keeping in mind weaknesses and strength of each component. For example, index for news must be refreshed almost in real time and relevance needs higher emphasis on location.

Search is not free and no one has figured out if someone just can provide index and someone else can provide relevance and everything can just work as efficiently as before (data point: each query consumes 0.3 Watt-hour of energy at Google).

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#129
post #70

Earlier quoted context omitted.

> the value of Google is the algo. The index is worth little. But it's worth something. Giving DuckDuckGo direct access to Google's index, including the ability to train models on said index, would improve the competitive landscape.

Or, you know, they could innovate by themselves instead of relying on Google to do all the heavy lifting?

Yeah it seems like this would spawn a lot of weak competitors who only exist because of the protections and would die off the second they're cut off from the API.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#130

I disagree. To break the Monopoly a new & innovative way to search needs to rise, not just a clone or "Google wanna be" like Bing/DuckDuckGo etc. So, somebody gotta do what Google did 20 years ago.

I don't necessarily think re-inventing the wheel every so often is the way to go on some things. When it comes to search, you want to type in your query, and search. It doesn't really get much simpler than having a search box, and I am having a hard time understanding how that can be innovated upon. I am however not saying it CAN'T be done, I just don't see why or how it would be.
Post reply on HN