Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

To Break Google’s Monopoly on Search, Make Its Index Public

bloomberg.com

Post reply on HN
Print view View original

461–470 of 630 posts

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#461

Post by saalweachter »

Strilanc wrote: post #297
madez wrote: post #287

Earlier quoted context omitted.

> 15% of Google searches per day are unique, as in, Google has never seen them before. That is impossible, and therefore wrong (I'm wrong, please see below). To know if a search is unique, as in Google has never seen them before, Google must be able to decide if a query it receives was seen before or not. Even if we assume Google needed only one bit for each message it has ever seen, and assuming it only saw 15% of n…

How are you computing that number? It's definitely wrong. Assume Google receives 1 trillion queries per year, and has been around for 20 years. Using a bloom filter you can achieve a 1% error rate with ~10 bits per item. So a 200 terabyte bloom filter would be more than sufficient to estimate the number of unique queries.

A Bloom filter is just way overkill.

If you have a list of 20 trillion query strings, and each query string is on average < 100 bytes, you're looking at a three line MapReduce and < 1 PiB of disk to create a table which has the frequency of every query ever issued. Add a counter to your final reduce to count how often the # times seen is 1.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#462

Post by bduerst »

M2Ys4U wrote: post #326
taf2 wrote: post #54

So although everyone likes to believe google is a monopoly it’s far from it. You have choices- bing, biadu , yandex, DuckDuckGo... there is also nothing about googles search position that prevents you from building a competitor. What we do have is peter thiel backing an administration that’s anti google, Russia, China that are anti google. Why? it’s a source of truth that challenges their lies. We also have an emerge…

Being a monopoly isn't actually the issue. Having a dominant market position is . This is where European competition law works well. Google definitely has a dominant market position.

EU antitrust law doesn't even require dominant market position to break the law.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#463

Post by d1zzy »

detritus wrote: post #78
noahl wrote: post #61

Okay, this is a relatively serious proposal to require Google to allow API access to its search index, with the premise that it would democratize the search engine ecosystem. There are some issues with the regulations he proposes (you have to allow throttling to prevent DDoS attacks, and you can't let anyone with API access add content to prevent garbage results), but it's roughly feasible. The main problem is, I thi…

> Indexing the top billion pages or so won't take as long as people think. This is what makes me wonder why we don't have a LOT of competing search engines. Perhaps i'm vastly under-estimating the technology and difficulty (I could well be - it's not my domain) but it surely it can't be THAT hard to spawn Google-like weighted crawl-based search results? It's a long-since solved problem - heck, pageRank's first iterat…

"indexing" is only part of the problem, it's a batch job. I find being able to respond to searches across a huge data set in the order of milliseconds (while having planet scale fail over) be a lot more challenging to implement.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#464

Post by bduerst »

celticmusic wrote: post #66
taf2 wrote: post #54

So although everyone likes to believe google is a monopoly it’s far from it. You have choices- bing, biadu , yandex, DuckDuckGo... there is also nothing about googles search position that prevents you from building a competitor. What we do have is peter thiel backing an administration that’s anti google, Russia, China that are anti google. Why? it’s a source of truth that challenges their lies. We also have an emerge…

ah yes, the only reason you could ever have for being anti google, et al, is that you want to lie... You started with a decent premise, that google isn't an actual monopoly because it has "competitors" (and yes, those quotes are intentional). But then you go off the deep end and completely lost me to the point that I didn't even finish the entirety of your post. There are many many reasons that someone could be again…

Google isn't a monopoly, or monopsony for that matter. Competitors are a click away.

Having a popular product != having control over the supply of a market. Market dominance through having a better product than the competition is how the free market works.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#465

Post by blackflame7000 »

d1zzy wrote: post #460
tsss wrote: post #353

Earlier quoted context omitted.

(2) is one of the most important points. We have to stop Google from cross-financing new products from other revenue streams so they can no longer undercut or buy all competitors. Google Maps is a good example. They ran it for super cheap a long time to drive out competitors and now rack up the prices. In contrast to most people here, I think breaking up Amazon is far more important to Facebook, Microsoft, Apple and…

But you have to acknowledge that without the cross-financing those "markets" wouldn't even exist. Before Google Maps we had a few online map services and they were terrible. Google Maps redefined what it means to have free access to web based interactive global maps, it changed how people find things and it was all payed by the ad business. Later on some monetizing efforts were made for it and competitors started to…

> "Before Google Maps we had a few online map services and they were terrible. Google Maps redefined what it means to have free access to web based interactive global maps"

This is not true. MapQuest revolutionized things almost 10 years earlier than Google Maps. Google search is what allowed Google Maps to overtake MapQuest. Also, Android providing real-time traffic data of all their users gave them the winning formula.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#466

Post by linuxftw »

rpiguy wrote: post #337
linuxftw wrote: post #317

Earlier quoted context omitted.

I think it matters in the context of liability. For instance, if you're a newspaper, and you publish "Politician X is a rapist" and you don't have a way to back up that allegation in court you're liable. As a platform, you're not liable for the content your users post. If you are allowing people to say "Politician X is a rapist" but not allowing people to say "Politician Y is a rapist", you've crossed into the land o…

I am personally all for freedom of information and also would like the concept of protected groups to go away, however... Liable is incredibly hard to prove in court, and usually cases are taken in civil court not criminal court. Can a platform ban specific groups? I see where your argument is coming from. Commerce is allowed by the Constitution for the common good (to serve everyone). Could Walmart, for example, say…

I think specifically in the case of platform vs publisher, there is federal law that gives some sort of extra legal protection to platforms. In order to get these protections, they have to behave a certain way. I'm not sure about this, but it's an argument I've heard.

> Liable is incredibly hard to prove in court, and usually cases are taken in civil court not criminal court.

Going to court is expensive, even if you win. Thus, the protections for platforms allow an avenue of easy dismissal of cases, preventing from go very far in the court system in the first place.

> Google, etc. (you can't compel us to disseminate what we deem as immoral, anymore than you can make a Christian baker bake a gay wedding cake).

Sure, but I think then they would qualify as a publisher and not a platform according to some. Compare the results from Google to Bing for the query 'Trump is a'. Did Google hand-tune their algorithm for their results? Did they tune their algorithm to weight more heavily organizations that share their political bias? EG, Prefer CNN over Fox News (or vice versa). Do some sites have insider knowledge to better game the system?

I can remember a time when there were no news results in a google search. It was a better internet.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#467

Post by entropicdrifter »

rasz wrote: post #448
losvedir wrote: post #412

Earlier quoted context omitted.

Yeah, this is why I still use and like myactivity.google.com, as creepy as it is. It's helped me re-find so many interesting half-remembered sites and videos and songs I'd previously come across.

Why would you rely on google spying instead of your own browser history?

cross platform support, maybe?

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#468

Post by zubspace »

foota wrote: post #449
zubspace wrote: post #65

From the article: "But what about those nasty filter bubbles that trap people in narrow worlds of information? Making Google’s index public doesn’t solve that problem, but it shrinks it to nonthreatening proportions. At the moment, it’s entirely up to Google to determine which bubble you’re in, which search suggestions you receive, and which search results appear at the top of the list; that’s the stuff of worldwide…

> I don't want Google results! I want different web crawlers ordering the results according to my taste Okay, great, we'll see how you browse and use that to determine... > without tracking each and every page impression of me. Give me that and I'll switch in a heartbeat. Uh? How exactly do you want them to do that?

The search engine basically owns an index of all websites. I assume, that crawling websites and storing an index is a solved problem. Not trivial on that scale and expensive. But conceptually solved.

What google does really well are two things:

1) It ranks those pages according to the well-known PageRank algorithm, although the details are kept secret. They also learned how to punish sites, which try to mislead the algorithm (Duplicate content, excessive SEO, link farms, etc)

2) What they also do well is returning search results in a very short time, based on a relevancy of your search term combined with the precalculated ranking. It also looks at your history in this step and the data it tracks about you.

My preferred search engine would open up both bullet points, giving you the option how pages are ranked and how relevancy is determined. For example: Rank personal blog posts higher or rank news higher or only use an inbound link count. Or you could get fancy and crowd source ranking (similar to hacker news). You could also configure search term relevancy. For example: did you ever search google for something specific and it automatically adjusted your term to something similar, more popular?

You could open up all those decisions and give the user options. Google on the other hand keeps everything secret and uses your personal data in unknown ways. I would love to have a search engine which is configurable. Yeah, not an easy thing to do. But it would be awesome.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#469

Post by latexr »

reroute1 wrote: post #455
latexr wrote: post #426

Earlier quoted context omitted.

> Also people forget that the creepy stuff Google does is super useful . For the same reasons you’re exalting them, I have non-technical friends who asked me how Google knows so much about them (and suggestions on how to avoid it) because they found it too creepy. I don’t think people forget Google’s results are useful; some just think they’re more creepy than valuable. You seem to have picked your side in that (im)b…

It is actually possible to operate without relying on Google or any other big tech firm. Who is forcing you into these privacy dilemmas? All of their services are a choice you are making. You don't need to accept any of it if you don't want to.

> You don't need to accept any of it if you don't want to.

Tell that to the people who had their privacy violated by Street View[1]. And the people who specifically disabled location services on their Android devices but were still tracked[2]. Or all the people who have no idea what Google Analytics is and never consented to it, but are profiled by it everyday.

> All of their services are a choice you are making.

I do my best to avoid privacy invading companies, and as a technical user I find it tiring and know I deal with consequences (e.g. broken websites). It perplexes me that comments like yours still pop up. We’re not the only segment of the population that exists; non-technical users are the majority, and they have the same right to privacy as we do, with a modicum of transparency. If even technical people are regularly tripped by privacy invasions we didn’t know about, what chances do non-technical users have?

[1]: https://www.nytimes.com/2013/03/13/technology/google-pays-fi...

[2]: https://qz.com/1131515/google-collects-android-users-locatio...

Post reply on HN