To Break Google’s Monopoly on Search, Make Its Index Public • Hacker News Board

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#531

Post by lrem » Mon, Jul 15, 2019, 10:41 PM UTC

majormajor wrote: post #436

com2kid wrote: post #345

Earlier quoted context omitted.

> 1) a record of searches and user clicks for the past 20 years From what I can tell, Google cares a lot more about recency. When I switch over to a new framework or language, search results are pretty bad for the first week, horrible actually as Google thinks I am still using /other language/. I have to keep appending the language / framework name to my queries. After a week or so? The results are pure magic. I can…

I always see posts like this here, and then I try it, and I get a page full of "array length" results for Javascript, while everything in the last year that I've searched for has been Java or Kotlin... Same when I owned a Pixel after hearing about Google Now and their ML magic there. Nothing more magical than an iPhone in terms of suggestions. The camera was amazing, but not all this supposed contextual stuff.

Wild guess: in a surge of privacy consciousness you told Google to stay the heck away from your data. These checkboxes stick forever and couple years down the line some magic feature won't be able to learn from your data. E.g. despite working there, I still haven't figured out how to let Photos recognize people in my pictures, something that definitely is on by default.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#532

Post by b52_ » Mon, Jul 15, 2019, 10:42 PM UTC

nostrademons wrote: post #528

frabcus wrote: post #520

Earlier quoted context omitted.

"Remember, search touches (nearly) every indexed document on every query" - wait, why does that happen? Doesn't it only touch ones with at least one of the search terms in, or stemmed/varied words relating to some of the terms? And does that via an index?

I struggled with how to word that in a way that's both true, understandable, and doesn't give away any proprietary information. Added "indexed" to clarify but I didn't fix up the numbers, so they're likely an overestimate. Basically, yes, it uses an index and touches only documents that appear in one of the relevant posting lists. However, after stemming, spell-correcting, synonyms, and a number of other expansions I…

> Each one of these needs to be scored (well, sorta - there are various tricks you can use to avoid scoring some docs, which again I'm not at liberty to discuss)

You mean like Wand or BMW?

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#533

Post by nordsieck » Mon, Jul 15, 2019, 10:44 PM UTC

nothis wrote: post #511

nordsieck wrote: post #342

Earlier quoted context omitted.

> What I have a problem with is literally every other service they provide. Google shouldn't own the most popular browser, the most popular video streaming platform, the most popular map service, the most popular operating system and (one of the?) most popular email services. So you're saying they're giving away too much good free stuff and need to be stopped?

I mean, obviously it’s not “free” to society.

> I mean, obviously it’s not “free” to society.

My mistake. I had forgotten that Google is a taxpayer funded organization.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#534

Post by jamiek88 » Mon, Jul 15, 2019, 10:46 PM UTC

chrshawkes wrote: post #257

whalabi wrote: post #213

Earlier quoted context omitted.

Now I feel bad for putting gibberish like jsjsjdkktkwoapaoalf in my address bar and searching Google to test if my internet is working..

I do that all the time, I wonder how common that is?

I would think it’s pretty common. For a lot of people google is the internet. Or at least the reference. If google isn't working it’s almost certain it’s your end. I don’t think anyone else has that reputation for availability amongst the general public.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#535

Post by Fion » Mon, Jul 15, 2019, 10:56 PM UTC

nostrademons wrote: post #309

Ex-Google-Search engineer here, having also done some projects since leaving that involve data-mining publicly-available web documents. This proposal won't do very much. Indexing is the (relatively) easy part of building a search engine. CommonCrawl already indexes the top 3B+ pages on the web and makes it freely available on AWS. It costs about $50 to grep over it, $800 or so to run a moderately complex Hadoop job.…

Page Rank is a synonym for link juice. So when you say Google hasn't used page rank since 2006, can you confirm that you are talking about link juice as opposed the the old toolbar representation of page rank? And assuming you do mean link juice, well why do links still work so well for seo?

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#536

Post by rifung » Mon, Jul 15, 2019, 10:59 PM UTC

crowbahr wrote: post #476

Razengan wrote: post #430

Earlier quoted context omitted.

> Google simply has the best search product. The best available doesn't necessarily mean the best possible. And Google is far from it, and it's getting worse, not better.

I've definitely noticed a decline in quality from Google results over the past few years in particular. I don't know if that's because SEO has gotten control of the results of if Google's algo is shoving lower quality up higher for revenue but it's become difficult. Using a bit of Google-fu I'm usually able to find what I need quickly but it's still more of a hassle than it used to be.

I work at Google but not on Search.

My guess is that it's because Google Search now also has to cater to queries from Assistant. Being required to handle web, mobile, and assistant probably necessitated tradeoffs in quality of one over another.

More generally I feel like as the company gets bigger it just gets much harder to handle all the complexity and keep things focused.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#537

Post by CSMastermind » Mon, Jul 15, 2019, 11:02 PM UTC

neilobremski wrote: post #524

appleshore wrote: post #493

Earlier quoted context omitted.

If there were viable alternatives, people would shift over time. If I type in “ Pentagon” on Google, the first link is LinkedIn. DuckDuckGo doesn’t even list it at all. There’s countless examples where DuckDuckGo just can’t find basic information. DDG is just unreliable beyond it’s silly name.

What does a viable alternative look like? I've been using Bing for the past few months; it's not great or terrible but is it "viable" enough for people to shift to over time? Or is it not viable because it's backed by a major corporation? I'm sure there are search quirks with each engine but I've seen issues with Google too and yet it's the "devil we know" ... so people unconsciously work around them.

I've used Bing for years now. The only time I go back to Google is if I'm searching for something super specific (normally programming related). Bing takes care of most of my search needs.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#538

Post by phreeza » Mon, Jul 15, 2019, 11:04 PM UTC

foota wrote: post #449

zubspace wrote: post #65

From the article: "But what about those nasty filter bubbles that trap people in narrow worlds of information? Making Google’s index public doesn’t solve that problem, but it shrinks it to nonthreatening proportions. At the moment, it’s entirely up to Google to determine which bubble you’re in, which search suggestions you receive, and which search results appear at the top of the list; that’s the stuff of worldwide…

> I don't want Google results! I want different web crawlers ordering the results according to my taste Okay, great, we'll see how you browse and use that to determine... > without tracking each and every page impression of me. Give me that and I'll switch in a heartbeat. Uh? How exactly do you want them to do that?

I guess at least in principle, the search engines could return many more results than actually needed, along with some feature vector per result. Then a model could be trained on the users machine to sort the results according to their preferences

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#539

Post by jibbed123 » Mon, Jul 15, 2019, 11:07 PM UTC

noahl wrote: post #61

Okay, this is a relatively serious proposal to require Google to allow API access to its search index, with the premise that it would democratize the search engine ecosystem. There are some issues with the regulations he proposes (you have to allow throttling to prevent DDoS attacks, and you can't let anyone with API access add content to prevent garbage results), but it's roughly feasible. The main problem is, I thi…

No, most queries are in the long tail

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#540

Post by blackflame7000 » Mon, Jul 15, 2019, 11:11 PM UTC

joshuamorton wrote: post #489

blackflame7000 wrote: post #465

Earlier quoted context omitted.

> "Before Google Maps we had a few online map services and they were terrible. Google Maps redefined what it means to have free access to web based interactive global maps" This is not true. MapQuest revolutionized things almost 10 years earlier than Google Maps. Google search is what allowed Google Maps to overtake MapQuest. Also, Android providing real-time traffic data of all their users gave them the winning form…

Free scrolling was pretty revolutionary. As was, like, an app that could reroute live instead of relying on pre-printed paper instructions. Gmaps had both before MapQuest.

You are right that traffic was revolutionary and that's why google maps became the defacto standard. However, in context with the original post, this is exactly why it's unfair. Google has an android that gives them user location data which they then use as a competitive advantage in another space to eliminate all competition. If Android were 1 business and GoogleMaps another, then people like MapQuest could also negotiate deals with Android to get user data and then it's a matter of who has the best platform that wins. That's what is best for the consumer as well. In the current structure, there is no way that a small business like MapQuest could build a smartphone to ascertain user data and nor should they have to. They should only have to build the best map application to succeed in the online mapping space. Having to also succeed in location data aggregation eliminates competition. It's designed so the giants can eat the small guys at will without them being able to fight back.