To Break Google’s Monopoly on Search, Make Its Index Public • Hacker News Board

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#451

Post by nova22033 » Mon, Jul 15, 2019, 7:20 PM UTC

malvosenior wrote: post #422

bduerst wrote: post #389

Earlier quoted context omitted.

Except you're wrong in that you can't logically prove something doesn't exist. You can't prove that pink unicorns don't exist, just as much as you can't prove that political bias in the search results don't exist. All you can do is disprove the claims of their existence. Someone tries to claim there's a pink unicorn in the garage, and you can check the garage and say that the it is pink unicorn free. Someone tries to…

> Now, if you suspect that the instant search results are politically biased, then the burden of proof is on you to provide evidence of that existence The proof is a senior Google employee admitting to bias and manipulating results in the Project Veritas video. There's also plenty of anecdotal evidence you can see for yourself as a user. In addition to that I know many people who work at Google and the vast majority…

There's also plenty of anecdotal evidence you can see for yourself as a user. In addition to that I know

> anecdotal evidence > I know

So much hard data here...real hacker news material...

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#452

Post by w_s_l » Mon, Jul 15, 2019, 7:22 PM UTC

The index word suddenly gave me an idea: the entire internet transformed into tabular data, searchable and open to public.

Even if we got Google's data, there's a whole lot of scraping to transform irregular and disparate. Typically you would have to google a keyword, look through the search results, visit different websites (research mode), and then consolidate separate sources of truths to build your own understanding. Building a scraper for each website in the search results and displaying a complete table data of the website. Why click through 100s of pages of profiles or data when you can have it all in one view? Why bother with HTML, when all a data-centric individual desires is data. Getting to the data is so tedious and a long journey. Scrape the website, clean the data, make it available for consumption, schedule & consolidate updates. For instance, a hedge fund that scrapes certain group of websites to execute market orders for an automated trading system.

A tabular data focused search engine would return tabular data, there would be no HTML medium, just straight up raw data. For instance, instead of seeing the comments rendered in a normal browser, imagine a tabular data that describes all the username, post time, comment minus the hierarchy.

To build a focused crawler quickly, I came up with Web Scraping Language (https://scrapeit.netlify.com), and essentially what I want to do is hire people to write WSL to scrape the web and then sell a subscription to data-centric customers.

What do ya say HN?

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#453

Post by based2 » Mon, Jul 15, 2019, 7:24 PM UTC

add Bing to the Index

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#454

Post by [deleted] » Mon, Jul 15, 2019, 7:28 PM UTC

exolymph wrote: post #443

bduerst wrote: post #394

Earlier quoted context omitted.

Right, and further observations show that it's not unique to Hillary Clinton. This means you can discount the claim since it uses cherry picked data.

But we still haven't answered the actual question: Why doesn't "email" come up in autocomplete for Hillary Clinton, when it clearly should?

[deleted]

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#455

Post by reroute1 » Mon, Jul 15, 2019, 7:29 PM UTC

latexr wrote: post #426

com2kid wrote: post #345

Earlier quoted context omitted.

> 1) a record of searches and user clicks for the past 20 years From what I can tell, Google cares a lot more about recency. When I switch over to a new framework or language, search results are pretty bad for the first week, horrible actually as Google thinks I am still using /other language/. I have to keep appending the language / framework name to my queries. After a week or so? The results are pure magic. I can…

> Also people forget that the creepy stuff Google does is super useful . For the same reasons you’re exalting them, I have non-technical friends who asked me how Google knows so much about them (and suggestions on how to avoid it) because they found it too creepy. I don’t think people forget Google’s results are useful; some just think they’re more creepy than valuable. You seem to have picked your side in that (im)b…

It is actually possible to operate without relying on Google or any other big tech firm. Who is forcing you into these privacy dilemmas? All of their services are a choice you are making. You don't need to accept any of it if you don't want to.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#456

Post by malvosenior » Mon, Jul 15, 2019, 7:29 PM UTC

nova22033 wrote: post #451

malvosenior wrote: post #422

Earlier quoted context omitted.

> Now, if you suspect that the instant search results are politically biased, then the burden of proof is on you to provide evidence of that existence The proof is a senior Google employee admitting to bias and manipulating results in the Project Veritas video. There's also plenty of anecdotal evidence you can see for yourself as a user. In addition to that I know many people who work at Google and the vast majority…

There's also plenty of anecdotal evidence you can see for yourself as a user. In addition to that I know > anecdotal evidence > I know So much hard data here...real hacker news material...

You conveniently leave off:

"The proof is a senior Google employee admitting to bias and manipulating results in the Project Veritas video."

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#457

Post by bduerst » Mon, Jul 15, 2019, 7:33 PM UTC

exolymph wrote: post #443

bduerst wrote: post #394

Earlier quoted context omitted.

Right, and further observations show that it's not unique to Hillary Clinton. This means you can discount the claim since it uses cherry picked data.

But we still haven't answered the actual question: Why doesn't "email" come up in autocomplete for Hillary Clinton, when it clearly should?

Yes we have. Further observation shows Google removes autocomplete for controversial items, like "Russian Investigation" for Donald Trump. If that example doesn't answer your question then you have confirmation bias.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#458

Post by vl » Mon, Jul 15, 2019, 7:33 PM UTC

nostrademons wrote: post #403

parentheses wrote: post #385

Earlier quoted context omitted.

I don't necessarily agree. The hard part of search is building the index and differentiating _real_ promotion from the _fake_. There's a lot of SEO manipulation that Google does a good job avoiding.

Webspam is a really big problem, yes. It's very unlikely that you'd be able to catch up or keep up in that regard without Google's resources. Building the index itself is relatively easy. There are some subtleties that most people don't think about (eg. dupe detection and redirects are surprisingly complicated, and CJK segmentation is a pre-req for tokenizing), but things like tokenizing, building posting lists, and…

You concentrating on the literal interpretation of the phrase “give access to the index”. This is non-technical article which didn’t go into details, just read it as “give access to index & ranking”.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#459

Post by bduerst » Mon, Jul 15, 2019, 7:35 PM UTC

malvosenior wrote: post #433

bduerst wrote: post #429

Earlier quoted context omitted.

This is precisely the hand-waving I was talking about. An employee rambling in a bar does not constitute evidence of search result manipulation, especially when the person recording it is known for stretching information, inciting people to commit voter fraud, and crossing the U.S. Mexico border dressed as deceased Osama Bin Laden to prove some point. Likewise, the handful of people you know having political views, i…

You asked for proof and I gave it to you. I'm sorry you don't like the source, but unless you want to address what Jen Gennai said, I don't think you have much of a point.

Jen Gennai herself points out how Veritas is lying about what she said: https://medium.com/@gennai.jen/this-is-not-how-i-expected-mo...

James Okeefe is publicly known for selectively editing video to make false representations, and this duck is no different.

You specifically claimed that there's plenty of anecdotal evidence. Where is it?

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#460

Post by d1zzy » Mon, Jul 15, 2019, 7:43 PM UTC

tsss wrote: post #353

dalbasal wrote: post #188

Earlier quoted context omitted.

So... I think there are two issues with this. (1) This doesn't actually reduce market share, since each of these are basically different market categories. (2) Almost all the revenue is from search. That company is the revenue generating arm for the other ones.

(2) is one of the most important points. We have to stop Google from cross-financing new products from other revenue streams so they can no longer undercut or buy all competitors. Google Maps is a good example. They ran it for super cheap a long time to drive out competitors and now rack up the prices. In contrast to most people here, I think breaking up Amazon is far more important to Facebook, Microsoft, Apple and…

But you have to acknowledge that without the cross-financing those "markets" wouldn't even exist.

Before Google Maps we had a few online map services and they were terrible. Google Maps redefined what it means to have free access to web based interactive global maps, it changed how people find things and it was all payed by the ad business. Later on some monetizing efforts were made for it and competitors started to appear, mostly trying to catch up and copy what Google Maps did, but without the huge cash infusion of the ad business none of this would have happened.

A decade later, people take these things for granted and just want to split services up. I guess it makes sense from their point of view but to me it's not that clear what should happen while still allowing for the type of creativity and speed of development that allowed things like Google Maps to appear because I'm afraid "the next big" thing that could redefine our lives (and improve them) would be slowed down or simply made non-feasible.