To Break Google’s Monopoly on Search, Make Its Index Public • Hacker News Board

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#391

Post by inlined » Mon, Jul 15, 2019, 5:38 PM UTC

asark wrote: post #123

tryptophan wrote: post #103

Earlier quoted context omitted.

>2) 20 years of experience fighting SEO spam. Tangential - but does anyone else feel that google results are useless a lot of the time? If you search for something, you will get 100% SEO optimized shitty ad-ridden blog/commercial pages giving surface level info about what you searched about. I find for programming/IT topics its pretty good, but for other topics it is horrible. Unless you are very specific with your s…

Google signed an armistice in the Great Spamsite War some time around '08 or '09, to the effect that spam can have all the search results aside from those pointing at a few top, trusted sites, so long as they provide any content at all. Bad content is fine. Farmed content is fine. Content that was probably machine-generated is fine. Just content. Play the game, make sure your markov chain article generator or mechani…

“Farmed content is fine”. I thought that was one of the major (intentional) victims of the Panda update. https://moz.com/learn/seo/google-panda

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#392

Post by nostrademons » Mon, Jul 15, 2019, 5:41 PM UTC

bogomipz wrote: post #380

nostrademons wrote: post #309

Ex-Google-Search engineer here, having also done some projects since leaving that involve data-mining publicly-available web documents. This proposal won't do very much. Indexing is the (relatively) easy part of building a search engine. CommonCrawl already indexes the top 3B+ pages on the web and makes it freely available on AWS. It costs about $50 to grep over it, $800 or so to run a moderately complex Hadoop job.…

>"Indexing is the (relatively) easy part of building a search engine. CommonCrawl already indexes the top 3B+ pages on the web and makes it freely available on AWS." Interesting I would have thought that crawling at this scale and finishing in a reasonable amount of time would still be somewhat challenging. Might you have any suggested reading for how this is done in practice? >"It costs about $50 to grep over it, $8…

Crawling is tricky but it's been commoditized. CommonCrawl does it for free for you. If you need pages that aren't in the index then you need to deal with all the crawling issues, but its index is about as big as the one most Google research was done on when I was there.

$50 gets you basically a Hadoop job that can run a regular expression over the plain text in a reasonably-efficient programing language (I tested with both Kotlin and Rust and they were in that ballpark). $800 was for a custom MapReduce I wrote that did something moderately complex - it would look at an arbitrary website, determine if it was a forum page, and then develop a strategy for extracting parsed & dated posts from the page and crawling it in the future.

A straight inverted index (where you tokenize the plaintext and store a posting list of documents for each term) would likely be more towards the $50 end of the spectrum - this is a classic information retrieval exercise that's both pretty easy to program (you can do it in a half day or so) and not very computationally intensive. It's also pretty useless for a real consumer search engine - there's a reason Google replaced all the keyword-based search engines we used in the 80s. There's also no reason you would do it today, when you have open-source products like ElasticSearch that'd do it for you and have a lot more linguistic smarts built in. (Straight ElasticSearch with no ranking tweaks is also nowhere near as good as Google.)

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#393

Post by evrydayhustling » Mon, Jul 15, 2019, 5:42 PM UTC

noahl wrote: post #61

Okay, this is a relatively serious proposal to require Google to allow API access to its search index, with the premise that it would democratize the search engine ecosystem. There are some issues with the regulations he proposes (you have to allow throttling to prevent DDoS attacks, and you can't let anyone with API access add content to prevent garbage results), but it's roughly feasible. The main problem is, I thi…

> it's roughly feasible

What do folks even mean by "Google's index"?? Google results combine tons of signals, including personal histories for each user. Sharing metadata for the top billion urls wouldn't cover half the functionality, or make a competitive engine. And on the other hand, there may not be a single other organization in the world prepared to manage a replica of the entire data plane that impacts seatch. The proposal is somewhere between underspecified and nonsense.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#394

Post by bduerst » Mon, Jul 15, 2019, 5:42 PM UTC

ar-jan wrote: post #255

root_axis wrote: post #209

Earlier quoted context omitted.

This is not "research" period. Using one arbitrary search comparison to draw conclusions about the nature of a system that processes billions of queries a day is pretty weak. Additionally, I don't get the same results you do. "hillary clinton e" does not bring up emails, nor does "donald trump e" bring up emails (the first results I see are election, education, england visit, ex wife). I'm not ruling out the possibil…

That's the point: this is not research, but whatever is going on at Google, the explanation has to account for examples like these. It's simply one observation that you cannot discount. I just tried searching again a few times with new private windows, and "email" alternates between first and fourth suggestion for trump. But the more important point is the absence of the suggestion for clinton: we know it's been in t…

Right, and further observations show that it's not unique to Hillary Clinton. This means you can discount the claim since it uses cherry picked data.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#395

Post by ar-jan » Mon, Jul 15, 2019, 5:42 PM UTC

root_axis wrote: post #364

bodono wrote: post #352

Earlier quoted context omitted.

The same thing happens with Russia for "barack obama rus" "hillary clinton rus", "george bush rus", etc. - none autocomplete for me despite the fact that there was lots of news items relating those politicians and russia during their careers. However Clinton is the only one that doesn't appear to autocomplete for email, suggesting that it is specific to her. When I type "donald trump epstei" I get "theo epstein donal…

None of those people had a gigantic Russia scandal though, Trump did, so you must still account for this unexplained aberration. If anything, the Trump/Russia scandal had more coverage than the Hillary e-mail scandal, so it's an even more difficult aberration to explain. Also, if I type "hillary e" I get "Hillary Emails PDF" as an autocomplete suggestion. If I type "clinton e" I get several email suggestions "clinton…

I can believe Google is attempting to de-emphasize scandals on both sides. As an aside, I do get "russia investigation" as autosuggestion for trump.

I think the bigger problem is that these adjustment appear to be done manually. We know that's what they're doing to avoid racist autocomplete phrases (and reasonably so, I think most people would agree). Having such judgments made on specific political topics/events/scandals will inevitably result in political bias, especially in an organization whose workforce is so politically skewed to one side.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#396

Post by jonas21 » Mon, Jul 15, 2019, 5:43 PM UTC

beatgammit wrote: post #194

detritus wrote: post #78

Earlier quoted context omitted.

> Indexing the top billion pages or so won't take as long as people think. This is what makes me wonder why we don't have a LOT of competing search engines. Perhaps i'm vastly under-estimating the technology and difficulty (I could well be - it's not my domain) but it surely it can't be THAT hard to spawn Google-like weighted crawl-based search results? It's a long-since solved problem - heck, pageRank's first iterat…

Querying an index isn't a solved problem, building it is. It's easy to gather the necessary data, but it's hard to know which parts of that data are the most relevant for finding good content and avoiding bad content. Is it more relevant if key words show up in links or titles than in the body of the text? If so, SEO spam sites will include a bunch of keywords in links and titles. Is it more relevant if keywords show…

> The only way I can think of a competitor stepping in is if they target an underserved demographic and focus data collection and monetization there, and DuckDuckGo is close by targeting privacy conscious power users.

The irony there is that DuckDuckGo can't collect much of that data precisely because of their privacy focus.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#397

Post by BluffFace » Mon, Jul 15, 2019, 5:45 PM UTC

First, Google doesn't have a monopoly on 'Search'. Competitors provide entirely comparable products and are not that far off from what Google provides. Other industries? Maybe. Search? Not in the slightest.

Second, that is a terrible idea. It would be a fucking crisis if Google's entire index was made public. I don't think the author understands the scope and nature of what Google indexes, as well as the protections Google has implemented.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#398

Post by jmichaelhudson » Mon, Jul 15, 2019, 5:51 PM UTC

I am boycotting google and also we should begin injecting it with as much junk and incorrect data as we possibly can.

There is no reason for humans to think we are too big to break this awful company.

They said they didn't track everyone's map location, but they did.

This is espionage. That is against the law.

Arrest everyone involved. But yes also confiscating their data for public good is a good idea, so maybe it is not a total loss.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#399

Post by reaperducer » Mon, Jul 15, 2019, 5:54 PM UTC

tryptophan wrote: post #103

noahl wrote: post #61

Okay, this is a relatively serious proposal to require Google to allow API access to its search index, with the premise that it would democratize the search engine ecosystem. There are some issues with the regulations he proposes (you have to allow throttling to prevent DDoS attacks, and you can't let anyone with API access add content to prevent garbage results), but it's roughly feasible. The main problem is, I thi…

>2) 20 years of experience fighting SEO spam. Tangential - but does anyone else feel that google results are useless a lot of the time? If you search for something, you will get 100% SEO optimized shitty ad-ridden blog/commercial pages giving surface level info about what you searched about. I find for programming/IT topics its pretty good, but for other topics it is horrible. Unless you are very specific with your s…

does anyone else feel that google results are useless a lot of the time?

Google doesn't make money from you finding what you're looking for. Google makes money from you searching for what you're looking for.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#400

Post by nostrademons » Mon, Jul 15, 2019, 5:55 PM UTC

mrtksn wrote: post #388

nostrademons wrote: post #309

Ex-Google-Search engineer here, having also done some projects since leaving that involve data-mining publicly-available web documents. This proposal won't do very much. Indexing is the (relatively) easy part of building a search engine. CommonCrawl already indexes the top 3B+ pages on the web and makes it freely available on AWS. It costs about $50 to grep over it, $800 or so to run a moderately complex Hadoop job.…

I think it is possible to make way, way better search engine because Google Search is no longer as good as it used to, at least for me. I can no longer find anything remotely good quality, I discover new and quality stuff from social media like Twitter and HN. The search results seem to be too general and too mainstream. Nothing new to discover, just a shortcut to the few websites like Reddit, StackOverflow for more…

The reason for that is because Google's building for a mainstream audience, because the mainstream (by definition) is much bigger than any niche. They increase aggregate happiness (though not your specific happiness) a lot more by doing so.

It's probably possible to build a search engine for a specific vertical that's better than Google. However, you face a few really big problems that make this not worthwhile:

1) Speaking from experience, it's very difficult to define what "better" means when you don't have exemplars of what queries are likely and what the results should be. The reason search engines are a product is that they let us find things we didn't know existed before; if we don't know they exist, how can we tweak the search engine to return them?

2) People go to a search engine because it has the answers for their question, no matter what their question is. If you had a specific search engine for games, and another for celebrities, and another for flights, and another for hotels, and another for books, and another for power tools, and another for current events, and another for technical documentation, and another for punditry, and another for history, and another to settle arguments on the Internet, then pretty soon you'd need a search engine to find the appropriate search engine. We call this "Google", and as a consumer, it's really convenient if they just give us the answer directly rather than directing us to another search engine where we need to refine our query again.

3) Google makes basically 80% of their revenue from searches for commercial products or services (insurance, lawyers, therapists, SaaS, flowers, etc.) The remainder is split between AdSense, Cloud, Android, Google Play, GFiber, YouTube, DoubleClick, etc. (may be a bit higher now). Many queries don't even run ads at all - when was the last time you saw an ad on a technical programming query, or a navigational query like [facebook login]? All of these are cross-subsidized by the commercial queries, because there's a benefit to Google from it being the one place you go to look for answers. If you build a niche site just to give good answers to programming queries or celebrity searches or current events, there's no business model there.