Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

To Break Google’s Monopoly on Search, Make Its Index Public

bloomberg.com

Post reply on HN
Print view View original

601–610 of 630 posts

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#601

Post by joantorres »

repolfx wrote: post #597
joantorres wrote: post #592

Earlier quoted context omitted.

Why do you say there is no business model in a search niche? StackOverflow and pleny of listing sites (Tripadvisor, Yelp, Zillow, Capterra to name a few) have been successfully built in this exact premise and the user experience of searching for restaurants, real state or software on these sites is usually much better than searching directly on Google due to the availability of custom filters and the amount of domain…

StackOverflow and those other sites aren't search engines. They may have search engines in them but not many people use them (the only time I reach StackOverflow, booking.com etc is via search engine referral). They're user content hosting and curation sites.

Technically you are correct, in the sense that they do not crawl the web like Google or Bing do. But from a user perspective, they do provide a very useful service of aggregation, discovery and comparison of structured data that is way more effective than using Google search queries, if you know the type of information you are looking for.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#602

Post by iainmerrick »

GordonS wrote: post #210
ailideex wrote: post #168

Earlier quoted context omitted.

I agree, to me it seems like that really does not change anything except it mandates one more middle man.

Yep, it just makes what likely already operates as an "internal" customer an "external" one. Not really sure how it helps with competition?

That’s exactly the point, to make that internal customer external, and make any special access or APIs they have available to competitors on an equal footing.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#603

Post by mda »

nostrademons wrote: post #309

Ex-Google-Search engineer here, having also done some projects since leaving that involve data-mining publicly-available web documents. This proposal won't do very much. Indexing is the (relatively) easy part of building a search engine. CommonCrawl already indexes the top 3B+ pages on the web and makes it freely available on AWS. It costs about $50 to grep over it, $800 or so to run a moderately complex Hadoop job.…

I find it odd that you claim to be a former Google search engineer and in the end boil down the success of Google search to brand recognition / loyalty. You kinda glossed over the insane complexity of building and maintaining a high quality search engine, really weird comment to be honest.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#604

Post by collinmanderson »

nostrademons wrote: post #400
mrtksn wrote: post #388

Earlier quoted context omitted.

I think it is possible to make way, way better search engine because Google Search is no longer as good as it used to, at least for me. I can no longer find anything remotely good quality, I discover new and quality stuff from social media like Twitter and HN. The search results seem to be too general and too mainstream. Nothing new to discover, just a shortcut to the few websites like Reddit, StackOverflow for more…

The reason for that is because Google's building for a mainstream audience, because the mainstream (by definition) is much bigger than any niche. They increase aggregate happiness (though not your specific happiness) a lot more by doing so. It's probably possible to build a search engine for a specific vertical that's better than Google. However, you face a few really big problems that make this not worthwhile: 1) Sp…

> navigational query like [facebook login]

Definitely have seen malicious adds for "facebook login", though that was probably 2016 or 2017.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#605

Post by websitejanitor »

nostrademons wrote: post #587
adventured wrote: post #551

Earlier quoted context omitted.

It is a seemingly incorrect claim. Google has semi-recently, publicly said they still use PageRank as one of their signals. https://searchengineland.com/google-has-confirmed-they-are-r... https://twitter.com/methode/status/829755916895535104

They replaced it in 2006 with an algorithm that gives approximately-similar results but is significantly faster to compute. The replacement algorithm is the number that's been reported in the toolbar, and what Google claims as PageRank (it even has a similar name, and so Google's claim isn't technically incorrect). Both algorithms are O(N log N) but the replacement has a much smaller constant on the log N factor, bec…

So what's the name of the new algorithm?

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#606

Post by tyingq »

jibbed123 wrote: post #541
kyrra wrote: post #107

Earlier quoted context omitted.

It's not just ML, but the people that provide the labeling for the ML. Google pays some large number of people to do search and grade the various results they get to see if the answers are good, which then helps feed back ML. Heck, according to this article[0], google has been paying people to evaluate their search results since 2004. [0] https://searchengineland.com/interview-google-search-quality...

It doesn't feed back into the ML directly, according to Google. Instead they use it to evaluate changes to search algorithms. If they get an increase in thumbs up back from the Quality Raters then their changes were positive. If not, they figure out why.

The original 2012 FTC investigation of Google anti-trust activity showed how they might have abused this process. Interesting read, no matter which side you take: http://graphics.wsj.com/google-ftc-report/

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#607

Post by Strilanc »

sloppycee wrote: post #495
saalweachter wrote: post #461

Earlier quoted context omitted.

A Bloom filter is just way overkill. If you have a list of 20 trillion query strings, and each query string is on average < 100 bytes, you're looking at a three line MapReduce and < 1 PiB of disk to create a table which has the frequency of every query ever issued. Add a counter to your final reduce to count how often the # times seen is 1.

uh, is this sarcasm? A bloom filter is the most appropriate data structure for this use-case. How is it overkill when it uses less space and is faster to query?

Actually the bloom filter was just an approachable example. There are much more clever and space efficient solutions to this problem, such as HyperLogLog [1] (speculating purely based on the numbers in that article, it looks like a few megabytes of space would be far more than sufficient). See the Wikipedia page on the "Count-distinct problem" [2].

1: https://en.wikipedia.org/wiki/HyperLogLog 2: https://en.wikipedia.org/wiki/Count-distinct_problem

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#608

Post by bryan_w »

mrtksn wrote: post #432
nostrademons wrote: post #400

Earlier quoted context omitted.

The reason for that is because Google's building for a mainstream audience, because the mainstream (by definition) is much bigger than any niche. They increase aggregate happiness (though not your specific happiness) a lot more by doing so. It's probably possible to build a search engine for a specific vertical that's better than Google. However, you face a few really big problems that make this not worthwhile: 1) Sp…

Sure, they have a business reason to do exactly what they do but I think as people grow up they specialize and the general stuff that fits everybody becomes useless. Google tries to personalize search results but that so far yielded echo chambers, not personalized discoveries. I can't get better products by searching Google, I can get the best-spammed products or most promoted products only. The fact that I am gettin…

How would google's practices stop me from creating a search engine?

Keep in mind when Google started, Yahoooooo! Was the big player and Google overtook them by simply being better

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#609

Post by tracker1 »

brunoff wrote: post #545
tracker1 wrote: post #523

Earlier quoted context omitted.

I find the single hardest thing to search for these days is anything more than a few months old on YouTube... They hate older videos, it feels like. Beyond that, I keep seeing suggestions on new content from years ago... it's just weird. I know it's not google proper, but I'd guess a significant number of their searches are specific to youtube.

I believe they try to put newer content first in order to make a more fair distribution of views. If you order results by popularity on yt, you will see that it uses just an "order by view count desc" (no relationship with like/dislike ratio), which is bad because it keeps popular some not so good quality videos published on first yt years.

Worse still, imho is that it may not be a popular video I'm looking for. I really wish they'd factor in a "I have viewed" for results.
Post reply on HN