Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

To Break Google’s Monopoly on Search, Make Its Index Public

bloomberg.com

Post reply on HN
Print view View original

421–430 of 630 posts

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#421

Post by kingludite »

utefan001 wrote: post #219
dkyc wrote: post #81

Earlier quoted context omitted.

> Yes, Google has a huge index, but most queries aren't in the long tail. I'm not quite sure about that. 15% of Google searches per day are unique, as in, Google has never seen them before. [1]. That's quite an insane number. [1] https://searchengineland.com/google-reaffirms-15-searches-ne...

Sharing for anyone who didn't know there is a very good dataset you can use now. If you don't have a nvme ssd in your computer, I highly recommend getting one for fast i/o. http://commoncrawl.org/ http://commoncrawl.org/the-data/ http://index.commoncrawl.org/ related.. Mark's blog is amazing and worth more than any data science degree imho. https://tech.marksblogg.com/petabytes-of-website-data-spark-... https://tech.…

wow, thanks.

[edit] in my experience yacy works really well. You have it crawl the sites you frequently visit and their external links and it quickly accumulates to something more accurate than google.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#422

Post by malvosenior »

bduerst wrote: post #389
malvosenior wrote: post #386

Earlier quoted context omitted.

None of those blog posts debunk the idea that Google manipulates search results to favor particular political parties. Mashable has a statement from Google (the other two don't), saying that they don't but why would they say if they were? None of these debunk the claims made by Google employees in the Project Veritas videos.

Except you're wrong in that you can't logically prove something doesn't exist. You can't prove that pink unicorns don't exist, just as much as you can't prove that political bias in the search results don't exist. All you can do is disprove the claims of their existence. Someone tries to claim there's a pink unicorn in the garage, and you can check the garage and say that the it is pink unicorn free. Someone tries to…

> Now, if you suspect that the instant search results are politically biased, then the burden of proof is on you to provide evidence of that existence

The proof is a senior Google employee admitting to bias and manipulating results in the Project Veritas video. There's also plenty of anecdotal evidence you can see for yourself as a user. In addition to that I know many people who work at Google and the vast majority of them have extreme political bias.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#423

Post by izacus »

Sebguer wrote: post #244
djsumdog wrote: post #173

Earlier quoted context omitted.

What about Project Veritas? People claim the statements by Google employed were taken out of context, but I've gone back, listened to them, looked at the videos, and it's hard to think in what context anything they said is acceptable. Even if the specific engineers and managers in the video clips don't have the level of authority to make the changes they're talking about, it's still chilling that their attitude could…

Project Veritas has historically operated by guiding people into saying something ridiculous, either by themselves acting ridiculous (and convincing the person they're talking to that they're crazy) or just driving the conversation to ridiculous areas. Maybe this Google 'expose' is the first time they're not guilty of that, but anyone who still finds them credible after their last several blatant mischaracterizations…

Jen Gennai had this to say in her article https://medium.com/@gennai.jen/this-is-not-how-i-expected-mo... :

" ... Project Veritas has edited the video to make it seem that I am a powerful executive who was confirming that Google is working to alter the 2020 election. On both counts, this is absolute, unadulterated nonsense, of course. In a casual restaurant setting, I was explaining how Google’s Trust and Safety team (a team I used to work on) is working to help prevent the types of online foreign interference that happened in 2016. Google has been very public about the work that our teams have done since 2016 on this, so it’s hardly a revelation. The video then goes on to stitch together a series of debunked conspiracy theories about our search results, and our other products. ... "

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#424

Post by ineedasername »

evrydayhustling wrote: post #393
noahl wrote: post #61

Okay, this is a relatively serious proposal to require Google to allow API access to its search index, with the premise that it would democratize the search engine ecosystem. There are some issues with the regulations he proposes (you have to allow throttling to prevent DDoS attacks, and you can't let anyone with API access add content to prevent garbage results), but it's roughly feasible. The main problem is, I thi…

> it's roughly feasible What do folks even mean by "Google's index"?? Google results combine tons of signals, including personal histories for each user. Sharing metadata for the top billion urls wouldn't cover half the functionality, or make a competitive engine. And on the other hand, there may not be a single other organization in the world prepared to manage a replica of the entire data plane that impacts seatch.…

Thanks, this is mainly what I came here to say. And I just don't see even the vaguely defined "index" as the crown jewel. If anything, it's "relevant results", which is something quite different.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#426

Post by latexr »

com2kid wrote: post #345
noahl wrote: post #61

Okay, this is a relatively serious proposal to require Google to allow API access to its search index, with the premise that it would democratize the search engine ecosystem. There are some issues with the regulations he proposes (you have to allow throttling to prevent DDoS attacks, and you can't let anyone with API access add content to prevent garbage results), but it's roughly feasible. The main problem is, I thi…

> 1) a record of searches and user clicks for the past 20 years From what I can tell, Google cares a lot more about recency. When I switch over to a new framework or language, search results are pretty bad for the first week, horrible actually as Google thinks I am still using /other language/. I have to keep appending the language / framework name to my queries. After a week or so? The results are pure magic. I can…

> Also people forget that the creepy stuff Google does is super useful.

For the same reasons you’re exalting them, I have non-technical friends who asked me how Google knows so much about them (and suggestions on how to avoid it) because they found it too creepy.

I don’t think people forget Google’s results are useful; some just think they’re more creepy than valuable. You seem to have picked your side in that (im)balance, and other people prefer the other side.

There’s also the relevant consideration that no matter how useful they may be, they should have no right to impose themselves on you. By this I mean that one should be free to refuse their creepiness, understanding the price is their usefulness. Yet, Google is the subject of privacy violations all the time, and they are caught time and again lying about what they collect on users.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#428

Post by marcosdumay »

tantalor wrote: post #262
ngngngng wrote: post #199

Earlier quoted context omitted.

I think you misunderstand what I'm saying. I'm not saying Google analytics will get you ranked higher or lower. Just that Google can use the data from analytics to tell if their results were what the user wanted.

> tell if their results were what the user wanted That's ranking.

That's validating the rank. Ranking is done in another step.

If they use the data for ranking, they'll lose the huge amount of value it has for validating.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#429

Post by bduerst »

malvosenior wrote: post #422
bduerst wrote: post #389

Earlier quoted context omitted.

Except you're wrong in that you can't logically prove something doesn't exist. You can't prove that pink unicorns don't exist, just as much as you can't prove that political bias in the search results don't exist. All you can do is disprove the claims of their existence. Someone tries to claim there's a pink unicorn in the garage, and you can check the garage and say that the it is pink unicorn free. Someone tries to…

> Now, if you suspect that the instant search results are politically biased, then the burden of proof is on you to provide evidence of that existence The proof is a senior Google employee admitting to bias and manipulating results in the Project Veritas video. There's also plenty of anecdotal evidence you can see for yourself as a user. In addition to that I know many people who work at Google and the vast majority…

This is precisely the hand-waving I was talking about. An employee rambling in a bar does not constitute evidence of search result manipulation, especially when the person recording it is known for stretching information, inciting people to commit voter fraud, and crossing the U.S. Mexico border dressed as deceased Osama Bin Laden to prove some point.

Likewise, the handful of people you know having political views, in an organization of 85,000 people, is not evidence that that organization's search results being biased.

If there is so much anecdotal evidence then it should be easy for you to prove, right? Or are you afraid of being disproven?

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#430

Post by Razengan »

nojvek wrote: post #361
nostrademons wrote: post #309

Ex-Google-Search engineer here, having also done some projects since leaving that involve data-mining publicly-available web documents. This proposal won't do very much. Indexing is the (relatively) easy part of building a search engine. CommonCrawl already indexes the top 3B+ pages on the web and makes it freely available on AWS. It costs about $50 to grep over it, $800 or so to run a moderately complex Hadoop job.…

This ^ times a 1000. Google simply has the best search product. They invest in it like crazy. I’ve tried bing multiple times. It’s slow, it spams msn ads in your face on the homepage. Microsoft just doesn’t get the value of a clean UX. DuckDuckGo results are pretty irrelevant the last time I tried them. There is nothing that comes close to their usability. To make the switchover, it has to be much much better than Go…

> Google simply has the best search product.

The best available doesn't necessarily mean the best possible. And Google is far from it, and it's getting worse, not better.

Post reply on HN