Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

To Break Google’s Monopoly on Search, Make Its Index Public

bloomberg.com

Post reply on HN
Print view View original

301–310 of 630 posts

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#301

Post by hiram112 »

tryptophan wrote: post #103
noahl wrote: post #61

Okay, this is a relatively serious proposal to require Google to allow API access to its search index, with the premise that it would democratize the search engine ecosystem. There are some issues with the regulations he proposes (you have to allow throttling to prevent DDoS attacks, and you can't let anyone with API access add content to prevent garbage results), but it's roughly feasible. The main problem is, I thi…

>2) 20 years of experience fighting SEO spam. Tangential - but does anyone else feel that google results are useless a lot of the time? If you search for something, you will get 100% SEO optimized shitty ad-ridden blog/commercial pages giving surface level info about what you searched about. I find for programming/IT topics its pretty good, but for other topics it is horrible. Unless you are very specific with your s…

100% agree. For technical queries, as long as a StackExchange comes up, Google is still okay.

But for increasingly more basic searches about a product I'm interested in or a medication or anything else non-complicated that would have gotten me a clean list of decent, non-paid results even 5 years, I'm now getting half a page of sponsored BS and then another half a page of 'created content' written by a bot or shyster explicitly for gaming Google's SEO.

Not only has Google lost almost all their good will (i.e. Don't be evil), but their products aren't even that good anymore, at least not so much better than alternatives where the negatives of using Google outweigh the difference in quality.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#302

Post by arm64future »

tomweingarten wrote: post #124
nova22033 wrote: post #106

Caveat: The author is not a technologists Robert Epstein (born June 19, 1953) is an American psychologist, professor, author, and journalist. He earned his Ph.D. in psychology at Harvard University in 1981, was editor in chief of Psychology Today, He has also made some questionable claims about google manipulating search results to favor Hillary Clinton. https://en.wikipedia.org/wiki/Robert_Epstein#cite_note-15 His r…

The comments he made are not just questionable, they're outright wrong (and a great example of the problem with cherry-picking data): https://www.vox.com/2016/6/10/11903028/hillary-clinton-googl... https://www.politifact.com/punditfact/statements/2016/jun/23... (Disclosure: I work at Google, but this opinion is my own)

do an image search for "european people art"

what's up with that

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#303

Post by caust1c »

madez wrote: post #287
dkyc wrote: post #81

Earlier quoted context omitted.

> Yes, Google has a huge index, but most queries aren't in the long tail. I'm not quite sure about that. 15% of Google searches per day are unique, as in, Google has never seen them before. [1]. That's quite an insane number. [1] https://searchengineland.com/google-reaffirms-15-searches-ne...

> 15% of Google searches per day are unique, as in, Google has never seen them before. That is impossible, and therefore wrong (I'm wrong, please see below). To know if a search is unique, as in Google has never seen them before, Google must be able to decide if a query it receives was seen before or not. Even if we assume Google needed only one bit for each message it has ever seen, and assuming it only saw 15% of n…

This is somewhat a faulty analysis. One could easily use a high accuracy bloom filter to store whether a search has definitely not been seen before, and that would be an estimate on the lower bound of the error margin.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#304

Post by [deleted] »

Strilanc wrote: post #297
madez wrote: post #287

Earlier quoted context omitted.

> 15% of Google searches per day are unique, as in, Google has never seen them before. That is impossible, and therefore wrong (I'm wrong, please see below). To know if a search is unique, as in Google has never seen them before, Google must be able to decide if a query it receives was seen before or not. Even if we assume Google needed only one bit for each message it has ever seen, and assuming it only saw 15% of n…

How are you computing that number? It's definitely wrong. Assume Google receives 1 trillion queries per year, and has been around for 20 years. Using a bloom filter you can achieve a 1% error rate with ~10 bits per item. So a 200 terabyte bloom filter would be more than sufficient to estimate the number of unique queries.

[deleted]

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#305

Post by dfabulich »

madez wrote: post #287
dkyc wrote: post #81

Earlier quoted context omitted.

> Yes, Google has a huge index, but most queries aren't in the long tail. I'm not quite sure about that. 15% of Google searches per day are unique, as in, Google has never seen them before. [1]. That's quite an insane number. [1] https://searchengineland.com/google-reaffirms-15-searches-ne...

> 15% of Google searches per day are unique, as in, Google has never seen them before. That is impossible, and therefore wrong (I'm wrong, please see below). To know if a search is unique, as in Google has never seen them before, Google must be able to decide if a query it receives was seen before or not. Even if we assume Google needed only one bit for each message it has ever seen, and assuming it only saw 15% of n…

Where are you getting these numbers? Google says they get ~2 trillion searches per year. 40 trillion searches over 20 years (way too many) would be 2^44 searches. https://searchengineland.com/google-now-handles-2-999-trilli...

(And they don’t even need to store all searches for all time for this, thanks to Bloom filters.)

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#306

Post by Angostura »

buildzr wrote: post #295
Angostura wrote: post #198

Earlier quoted context omitted.

Hmmm - Error: System.InvalidOperationException: The workflow with id=15f08b34-33f5-4063-8dea-d4ca6212c0d6 is no longer available. is not atypical.

Does that actually work? I must be old school, I always delete such IDs before searching, but then again I used Google back when it actually did what you told it instead of misinterpreting everything for you.

It doesn't seem to have any particular effect on the results that come up. I always used to delete them, and still do sometimes but Google seems to pretty much ignore them in practice.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#307

Post by learnfromstory »

martin_drapeau wrote: post #283

The article mentions a precedent: "in the 1956 consent decree in the U.S. in which AT&T agreed to share all its patents with other companies free of charge." Today the picture is completely different. Google, Amazon, Facebook and Apple are global - their infrastructure is virtual - not physical lines and cables. If a government wants to cause harm to one of those companies, they can decide to "move" somewhere else. I…

Uh no. Google has hundreds of billions of dollars invested in real non-virtual assets, majority of which are in the USA.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#308

Post by MiroF »

GordonS wrote: post #205
tryptophan wrote: post #103

Earlier quoted context omitted.

>2) 20 years of experience fighting SEO spam. Tangential - but does anyone else feel that google results are useless a lot of the time? If you search for something, you will get 100% SEO optimized shitty ad-ridden blog/commercial pages giving surface level info about what you searched about. I find for programming/IT topics its pretty good, but for other topics it is horrible. Unless you are very specific with your s…

Yes, I feel like Google search results have very gradually become more irrelevant and spammy over the past decade or so. There are 2 issues, I think. Firstly, the SE-optimised spam, which has become very good as masquerading as genuine content. Secondly, Google has dumbed search syntax down a bit, and often seems to outright ignore double quoted phrases, presumably thinking it knows better than I what I want. As a de…

is there a way to turn this " ignore thing off? drives me nuts

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#309

Post by nostrademons »

Ex-Google-Search engineer here, having also done some projects since leaving that involve data-mining publicly-available web documents.

This proposal won't do very much. Indexing is the (relatively) easy part of building a search engine. CommonCrawl already indexes the top 3B+ pages on the web and makes it freely available on AWS. It costs about $50 to grep over it, $800 or so to run a moderately complex Hadoop job.

(For comparison, when I was at Google nearly all research & new features were done on the top 4B pages, and the remaining 150B+ pages were only consulted if no results in the top 4B turned up. Difficulty of running a MapReduce over that corpus was actually a little harder than running a Hadoop job over CommonCrawl, because there's less documentation available.)

The comments here that PageRank is Google's secret sauce also aren't really true - Google hasn't used PageRank since 2006. The ones about the search & clickthrough data being important are closer, but I suspect that if you made those public you still wouldn't have an effective Google competitor.

The real reason Google's still on top is that consumer habits are hard to change, and once people have 20 years of practice solving a problem one way, most of them are not going to switch unless the alternative isn't just better, it's way, way better. Same reason I still buy Quilted Northern toilet paper despite knowing that it supports the Koch brothers and their abhorrent political views, or drink Coca-Cola despite knowing how unhealthy it is.

If you really want to open the search-engine space to competition, you'd have to break Google up and then forbid any of the baby-Googles from using the Google brand or google.com domain name. (Needless to say, you'd also need to get rid of Chrome & Toolbar integration.) Same with all the other monopolies that plague the American business landscape. Once you get to a certain age, the majority of the business value is in the brand, and so the only way to keep the monopoly from dominating its industry again is to take away the brand and distribute the productive capacity to successor companies on relatively even footing.

Post reply on HN