To Break Google’s Monopoly on Search, Make Its Index Public • Hacker News Board

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#561

Post by jammygit » Tue, Jul 16, 2019, 1:13 AM UTC

tills13 wrote: post #557

mrtksn wrote: post #388

Earlier quoted context omitted.

I think it is possible to make way, way better search engine because Google Search is no longer as good as it used to, at least for me. I can no longer find anything remotely good quality, I discover new and quality stuff from social media like Twitter and HN. The search results seem to be too general and too mainstream. Nothing new to discover, just a shortcut to the few websites like Reddit, StackOverflow for more…

I see comments like this all the time. Am I alone in that search results, for me, have gotten significantly _better_ since a couple years ago? I can't help but think it's partially due to people using tools _specifically designed_ to make Google's job harder (FF SandBoxes, uBlock, etc) and not understanding the implications of using them... and then blaming Google for returning "bad" results.

I get a lot more seo spam than I used to, but the results are still quite good. I think we should give google some credit for that at least.

Like, a lot more seo spam though.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#562

Post by devit » Tue, Jul 16, 2019, 1:14 AM UTC

nostrademons wrote: post #309

Ex-Google-Search engineer here, having also done some projects since leaving that involve data-mining publicly-available web documents. This proposal won't do very much. Indexing is the (relatively) easy part of building a search engine. CommonCrawl already indexes the top 3B+ pages on the web and makes it freely available on AWS. It costs about $50 to grep over it, $800 or so to run a moderately complex Hadoop job.…

IMHO a simpler and probably the only viable way to force competition is to legally force Google to not respond to any query on certain periodic time periods.

For instance, if you were to forbid Google from operating on every odd-numbered day, then 50% of the search engine market and revenues would immediately be distributed among competitors and furthermore users would be forced to test multiple engines and they could find a better one to use even when Google is allowed to operate.

Obviously this has a short-term economic cost if other search engines aren't good enough as well as imposing an arbitrary restriction on business, so it's debatable whether this would be a reasonable course of action

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#563

Post by tills13 » Tue, Jul 16, 2019, 1:20 AM UTC

jammygit wrote: post #561

tills13 wrote: post #557

Earlier quoted context omitted.

I see comments like this all the time. Am I alone in that search results, for me, have gotten significantly _better_ since a couple years ago? I can't help but think it's partially due to people using tools _specifically designed_ to make Google's job harder (FF SandBoxes, uBlock, etc) and not understanding the implications of using them... and then blaming Google for returning "bad" results.

I get a lot more seo spam than I used to, but the results are still quite good. I think we should give google some credit for that at least. Like, a lot more seo spam though.

People have gotten really good (i.e. it's their full-time job) at "gaming" Google. That's not to say Google is fallible - every search engine is game-able depending on its algorithm - it's just that these people are _very_ clever.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#564

Post by yy77 » Tue, Jul 16, 2019, 1:24 AM UTC

The key for goolge's monopoly is not only its index algorithm capability, but also the brand, the chrome browser, the android and all other google product integration.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#565

Post by marcescence » Tue, Jul 16, 2019, 1:26 AM UTC

tills13 wrote: post #557

mrtksn wrote: post #388

Earlier quoted context omitted.

I think it is possible to make way, way better search engine because Google Search is no longer as good as it used to, at least for me. I can no longer find anything remotely good quality, I discover new and quality stuff from social media like Twitter and HN. The search results seem to be too general and too mainstream. Nothing new to discover, just a shortcut to the few websites like Reddit, StackOverflow for more…

I see comments like this all the time. Am I alone in that search results, for me, have gotten significantly _better_ since a couple years ago? I can't help but think it's partially due to people using tools _specifically designed_ to make Google's job harder (FF SandBoxes, uBlock, etc) and not understanding the implications of using them... and then blaming Google for returning "bad" results.

> specifically designed to make Google's job harder "Better search" doesn't necessarily mean "more personalized search."

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#566

Post by beachy » Tue, Jul 16, 2019, 1:28 AM UTC

devit wrote: post #562

nostrademons wrote: post #309

Ex-Google-Search engineer here, having also done some projects since leaving that involve data-mining publicly-available web documents. This proposal won't do very much. Indexing is the (relatively) easy part of building a search engine. CommonCrawl already indexes the top 3B+ pages on the web and makes it freely available on AWS. It costs about $50 to grep over it, $800 or so to run a moderately complex Hadoop job.…

IMHO a simpler and probably the only viable way to force competition is to legally force Google to not respond to any query on certain periodic time periods. For instance, if you were to forbid Google from operating on every odd-numbered day, then 50% of the search engine market and revenues would immediately be distributed among competitors and furthermore users would be forced to test multiple engines and they coul…

Any search engine that was unavailable for 50% of the time would soon have 0% of the market, not 50%.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#567

Post by burtonator » Tue, Jul 16, 2019, 1:31 AM UTC

nostrademons wrote: post #309

Ex-Google-Search engineer here, having also done some projects since leaving that involve data-mining publicly-available web documents. This proposal won't do very much. Indexing is the (relatively) easy part of building a search engine. CommonCrawl already indexes the top 3B+ pages on the web and makes it freely available on AWS. It costs about $50 to grep over it, $800 or so to run a moderately complex Hadoop job.…

I can also comment here. I built and still run a petabyte-scale web crawler:

https://www.datastreamer.io/

Common Crawl and other sources do in fact have a ton of data that can be used which is very affordable.

The DATA itself stopped being a real competitive advantage probably 2008-2010.

Google's major advantage now is its algorithms and the fact that they've proven it works and is reliable.

Most importantly, its the brand. Google MEANS search in the US and that won't change anytime soon.

PS,... if you need tons of web and social data Datastreamer can hook you up too :)

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#568

Post by thanhhaimai » Tue, Jul 16, 2019, 1:50 AM UTC

devit wrote: post #562

nostrademons wrote: post #309

Ex-Google-Search engineer here, having also done some projects since leaving that involve data-mining publicly-available web documents. This proposal won't do very much. Indexing is the (relatively) easy part of building a search engine. CommonCrawl already indexes the top 3B+ pages on the web and makes it freely available on AWS. It costs about $50 to grep over it, $800 or so to run a moderately complex Hadoop job.…

IMHO a simpler and probably the only viable way to force competition is to legally force Google to not respond to any query on certain periodic time periods. For instance, if you were to forbid Google from operating on every odd-numbered day, then 50% of the search engine market and revenues would immediately be distributed among competitors and furthermore users would be forced to test multiple engines and they coul…

Banning anything is often not a good policy since it usually creates secondary markets.

Depends on how you count the date, this could create markets where people in different countries will sell Google search results to each other. New VPN providers pop up with the promise of 24h Google coverage. Software startups switch to a system where you bing work 16 hours straight, then get the next 32 hours break and repeat. "Breaking news" has a new Oxford definition, since newspapers change plan to publish news 5 minutes before Google opens for search. Electricity price increases for the first 2 hours of the odd-numbered day to combat the spike in demand. Comcast introduces a new fast lane at only $199 a month that has no slow down access to Google. University groups lobby for a new exemption in the law allowing unrestricted weekdays access. Political parties lobby for also blocking Google on the day of debates, regardless of whether it's an odd-numbered day. It's kinda fun to keep going.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#569

Post by briantakita » Tue, Jul 16, 2019, 2:01 AM UTC

appleshore wrote: post #493

nostrademons wrote: post #309

Ex-Google-Search engineer here, having also done some projects since leaving that involve data-mining publicly-available web documents. This proposal won't do very much. Indexing is the (relatively) easy part of building a search engine. CommonCrawl already indexes the top 3B+ pages on the web and makes it freely available on AWS. It costs about $50 to grep over it, $800 or so to run a moderately complex Hadoop job.…

If there were viable alternatives, people would shift over time. If I type in “ Pentagon” on Google, the first link is LinkedIn. DuckDuckGo doesn’t even list it at all. There’s countless examples where DuckDuckGo just can’t find basic information. DDG is just unreliable beyond it’s silly name.

I've been using DDG almost exclusively & find it's results to be better than Google's with the exception of local businesses & maps. Google still has an advantage there.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#570

Post by ChrisCinelli » Tue, Jul 16, 2019, 2:01 AM UTC

nostrademons wrote: post #309

Ex-Google-Search engineer here, having also done some projects since leaving that involve data-mining publicly-available web documents. This proposal won't do very much. Indexing is the (relatively) easy part of building a search engine. CommonCrawl already indexes the top 3B+ pages on the web and makes it freely available on AWS. It costs about $50 to grep over it, $800 or so to run a moderately complex Hadoop job.…

> when I was at Google nearly all research & new features were done on the top 4B pages, and the remaining 150B+ pages were only consulted if no results in the top 4B turned up

This may help to explain the poor quality of some the results on queries I run on Google lately that return content obviously written for ranking in SEO but that have very little value.

I have 2 questions:

- What make"the top 3B+" the top ones?

- How can I "force"a search on the other 150B+ pages?