Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

To Break Google’s Monopoly on Search, Make Its Index Public

bloomberg.com

Post reply on HN
Print view View original

551–560 of 630 posts

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#551

Post by adventured »

websitejanitor wrote: post #530
nostrademons wrote: post #309

Ex-Google-Search engineer here, having also done some projects since leaving that involve data-mining publicly-available web documents. This proposal won't do very much. Indexing is the (relatively) easy part of building a search engine. CommonCrawl already indexes the top 3B+ pages on the web and makes it freely available on AWS. It costs about $50 to grep over it, $800 or so to run a moderately complex Hadoop job.…

>The comments here that PageRank is Google's secret sauce also aren't really true - Google hasn't used PageRank since 2006. That's quite a claim considering they were reporting PageRank in their toolbar until 2016, and toolbar PageRank was visible in Google Directory until 2011. Are you talking about PageRank from the original patent?

It is a seemingly incorrect claim. Google has semi-recently, publicly said they still use PageRank as one of their signals.

https://searchengineland.com/google-has-confirmed-they-are-r...

https://twitter.com/methode/status/829755916895535104

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#552

Post by mr_toad »

appleshore wrote: post #493
nostrademons wrote: post #309

Ex-Google-Search engineer here, having also done some projects since leaving that involve data-mining publicly-available web documents. This proposal won't do very much. Indexing is the (relatively) easy part of building a search engine. CommonCrawl already indexes the top 3B+ pages on the web and makes it freely available on AWS. It costs about $50 to grep over it, $800 or so to run a moderately complex Hadoop job.…

If there were viable alternatives, people would shift over time. If I type in “ Pentagon” on Google, the first link is LinkedIn. DuckDuckGo doesn’t even list it at all. There’s countless examples where DuckDuckGo just can’t find basic information. DDG is just unreliable beyond it’s silly name.

Neither DDG or Google return any LinkedIn results for me unless I also add LinkedIn to the search, in which case I get the same results for both search engines.

Google knows what you want before you even ask. You might find that convenient, I find it unsettling.

I guess it’s not as bad as Facebook; at least Google doesn’t spoon feed you.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#553

Post by striking »

bamboozled wrote: post #547
striking wrote: post #522

Earlier quoted context omitted.

I find DDG has pretty acceptable or even good results most of the time. The real power is in the "bangs", though; you can use the `!` to immediately jump to the first search result without seeing a search page, or use `!g` to switch to Google for this particular query, among others. It enables a sort of power-user usage that one wouldn't get with Google.

I don’t really get the logic, just use a good search engine in the first place ?

I'm saying that DDG can be "good enough", and that not having to click around on a results page can save you time if you know what you're doing.

I understand that for some people that's not enough of a time savings to make a difference, but I know DDG well enough to be able to `!` things and almost always immediately get to a successful result. I treat it as an extension of my brain at this point.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#554

Post by knowaveragejoe »

fjp wrote: post #149
drevil-v2 wrote: post #140

Earlier quoted context omitted.

Speaking of cherry picking, I wouldn't use vox as reliable source for bipartisan data.

Or politifact...

I've seen plenty of politifact and snopes fact-checks that go against the conspiracies that you guys seem to think are underway in those types of organizations.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#555

Post by knowaveragejoe »

malvosenior wrote: post #422
bduerst wrote: post #389

Earlier quoted context omitted.

Except you're wrong in that you can't logically prove something doesn't exist. You can't prove that pink unicorns don't exist, just as much as you can't prove that political bias in the search results don't exist. All you can do is disprove the claims of their existence. Someone tries to claim there's a pink unicorn in the garage, and you can check the garage and say that the it is pink unicorn free. Someone tries to…

> Now, if you suspect that the instant search results are politically biased, then the burden of proof is on you to provide evidence of that existence The proof is a senior Google employee admitting to bias and manipulating results in the Project Veritas video. There's also plenty of anecdotal evidence you can see for yourself as a user. In addition to that I know many people who work at Google and the vast majority…

The "proof" is extremely shaky. Veritas has a record of creating these types of videos where they draw conclusions out of thin air. Here's what the person in particular has to say:

https://medium.com/@gennai.jen/this-is-not-how-i-expected-mo...

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#556

Post by jjeaff »

nojvek wrote: post #361
nostrademons wrote: post #309

Ex-Google-Search engineer here, having also done some projects since leaving that involve data-mining publicly-available web documents. This proposal won't do very much. Indexing is the (relatively) easy part of building a search engine. CommonCrawl already indexes the top 3B+ pages on the web and makes it freely available on AWS. It costs about $50 to grep over it, $800 or so to run a moderately complex Hadoop job.…

This ^ times a 1000. Google simply has the best search product. They invest in it like crazy. I’ve tried bing multiple times. It’s slow, it spams msn ads in your face on the homepage. Microsoft just doesn’t get the value of a clean UX. DuckDuckGo results are pretty irrelevant the last time I tried them. There is nothing that comes close to their usability. To make the switchover, it has to be much much better than Go…

I disagree. It works great for me. Maybe once every few days I will use !g when I can't find something, but I rarely end up finding it on Google either.

I read somewhere that someone used a skin to make ddg look identical to Google. After doing that, they never even thought about using Google again.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#557

Post by tills13 »

mrtksn wrote: post #388
nostrademons wrote: post #309

Ex-Google-Search engineer here, having also done some projects since leaving that involve data-mining publicly-available web documents. This proposal won't do very much. Indexing is the (relatively) easy part of building a search engine. CommonCrawl already indexes the top 3B+ pages on the web and makes it freely available on AWS. It costs about $50 to grep over it, $800 or so to run a moderately complex Hadoop job.…

I think it is possible to make way, way better search engine because Google Search is no longer as good as it used to, at least for me. I can no longer find anything remotely good quality, I discover new and quality stuff from social media like Twitter and HN. The search results seem to be too general and too mainstream. Nothing new to discover, just a shortcut to the few websites like Reddit, StackOverflow for more…

I see comments like this all the time. Am I alone in that search results, for me, have gotten significantly _better_ since a couple years ago?

I can't help but think it's partially due to people using tools _specifically designed_ to make Google's job harder (FF SandBoxes, uBlock, etc) and not understanding the implications of using them... and then blaming Google for returning "bad" results.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#558

Post by RaceWon »

beatgammit wrote: post #194
detritus wrote: post #78

Earlier quoted context omitted.

> Indexing the top billion pages or so won't take as long as people think. This is what makes me wonder why we don't have a LOT of competing search engines. Perhaps i'm vastly under-estimating the technology and difficulty (I could well be - it's not my domain) but it surely it can't be THAT hard to spawn Google-like weighted crawl-based search results? It's a long-since solved problem - heck, pageRank's first iterat…

Querying an index isn't a solved problem, building it is. It's easy to gather the necessary data, but it's hard to know which parts of that data are the most relevant for finding good content and avoiding bad content. Is it more relevant if key words show up in links or titles than in the body of the text? If so, SEO spam sites will include a bunch of keywords in links and titles. Is it more relevant if keywords show…

> Querying an index isn't a solved problem, building it is...

You didn't just hit the nail on the head; you drove it all the way in with a single blow. Bravo.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#559

Post by riku_iki »

nostrademons wrote: post #491
Eridrus wrote: post #475

Earlier quoted context omitted.

Sure, it costs $50 to grep it, but how much does it cost to host an in-memory index with all the data? This is not a proposal to just share the crawl data, but the actual searchable index, presumably at arms length cost both internally & externally. The same ideas could be extended to the Knowledge Graph, etc. IMO the goal here should not be to kill Google, but to keep Google on their toes by removing barriers to com…

The data was about 55TB of compressed HTML last I looked, so that's about 70 r5a.24xlarge instances, each going for $5.424/hour, so about $350/hour or $250K/month. That's not cheap, and definitely not something you'd put on your personal credit card, but it's well within the range of a seed-funded startup. Sizes may vary a bit depending upon the exact index format, but that should be a rough ballpark. With batch jobs…

> Knowledge Graph is already public > https://developers.google.com/freebase/

That dump is outdated, not supported, and very incomplete comparing to what google has now.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#560

Post by Karunamon »

latexr wrote: post #474
reroute1 wrote: post #473

Earlier quoted context omitted.

Street view is debatably invasive. I understand this might seem hand wavy to someone really concerned about privacy issues, but 1.generally speaking I would think VERY few people care about an image of their property being on street views. 2. It's not really illegal to take pictures so even from a legal standpoint it seems like a gray area. 3. I understand there can be individual reasons for not wanting this, but it…

The first sentence of the linked New York Times story: > Google on Tuesday acknowledged to state officials that it had violated people’s privacy during its Street View mapping project when it casually scooped up passwords, e-mail and other personal information from unsuspecting computer users. That answers your first three paragraphs. There’s no “if” to their lying and privacy invasions. They’ve been caught and admit…

That snippet of the NYT story omits critical context: The data they captured were random wifi packets (probably for use in Skyhook-type location fixes by way of mapping out where APs are). Sounds like they were doing the equivalent of a wardrive and captured more than the AP advertisement message.

This is information that Google doesn't have any need for (noise) and didn't want in the first place.

They also self-reported the failure, where they could have just nuked it and we wouldn't be having this conversation.

Post reply on HN