This is dumb. You mean if I work two decades to develop tech that nobody else can copy, I have to open-source it because my competitors are dumbasses?
This is not how it's suppose to work.
To Break Google’s Monopoly on Search, Make Its Index Public
371–380 of 630 posts
Re: To Break Google’s Monopoly on Search, Make Its Index Public
#372Is Duck Search's Index Public?
Re: To Break Google’s Monopoly on Search, Make Its Index Public
#373Earlier quoted context omitted.
Storage and bandwidth are cheaper than ever before, people scrape a billion pages for much more mundane purposes these days, even for academic papers. Having a full text index on that is more involved but hardly impossible. You're completely right that it's not at all Google's secret sauce. Bing has clearly indexed much more than that, plus invested a ton in actually returning good results from their index. And still…
> It's just not easy to make a better Google It depends which sense of "better" you mean. It's nearly trivial to make an ethically superior search engine by just not building the spyware bits of Google. It's difficult to make a search engine that's "better" along the dimensions of speed, profitability, etc.
That exists, it's called duck duck go, and even less people care about it than Bing. For the most part, people don't actually care about Google collecting their entire search history and combining it with their other data on you. We may live to regret that in a hypothetical future where the government turns more authoritarian and requisitions that data for evil.
Re: To Break Google’s Monopoly on Search, Make Its Index Public
#374Earlier quoted context omitted.
> I don’t see any mention in this article of what seems like the most obvious way to split up Google, separating their search and ad businesses. What would be the revenue model for their search then?
The ad business would pay the search business for ad placement. The search business would also be free to auction ad space to other providers.
>The ad business would pay the search business for ad placement. The search business would also be free to auction ad space to other providers.
What you're describing is exactly what Google ads are today.
Re: To Break Google’s Monopoly on Search, Make Its Index Public
#375Earlier quoted context omitted.
> Yes, Google has a huge index, but most queries aren't in the long tail. I'm not quite sure about that. 15% of Google searches per day are unique, as in, Google has never seen them before. [1]. That's quite an insane number. [1] https://searchengineland.com/google-reaffirms-15-searches-ne...
Now I feel bad for putting gibberish like jsjsjdkktkwoapaoalf in my address bar and searching Google to test if my internet is working..
I just type "test", hopefully they do that too and it is ignored.
Re: To Break Google’s Monopoly on Search, Make Its Index Public
#376I am no expert on Google search, but it seems to me that "accessing the index" and "retrieving ranked results" must be one and the same. That is, the very structure of the index is designed around a particular ranking scheme in order to make accessing it computationally tractable. In light of that, how would one expose an API to offer access to the index without having something about Google's curation or ranking built in. A query API is clearly out because query responses include some results and leave out others, but even a dump runs into the problem that the data was collected, parsed, and structured with a particular scheme in mind.
Re: To Break Google’s Monopoly on Search, Make Its Index Public
#377Okay, this is a relatively serious proposal to require Google to allow API access to its search index, with the premise that it would democratize the search engine ecosystem. There are some issues with the regulations he proposes (you have to allow throttling to prevent DDoS attacks, and you can't let anyone with API access add content to prevent garbage results), but it's roughly feasible. The main problem is, I thi…
> Bing had the money and persistence to make that investment, but how many others will?
I hypothesized once with an ex Microsoft HIGH up that it probably took 10B to launch bing. He said I was almost exactly on the nose.
Also this is a ridiculous thing to ask for. How much money do you think Google pays for the bandwidth to crawl the web? How much do you think it costs to run the machines that create indexes out of that? How do you value the IP involved in the process?
Google should give away the fruits of that labor for free, plus invest in a reasonable API to download that index? Plus the bandwidth of sharing that index with third parties? It’s probably not even feasible aside from putting disks or tapes on multiple semis to send to clients. The index is 100 petabytes according to [0]. With dual fiber lines, and no latency for mind bending numbers of API calls, that would take 12.6 YEARS to download a single snapshot.
[0] https://www.google.com/search/howsearchworks/crawling-indexi...
Re: To Break Google’s Monopoly on Search, Make Its Index Public
#378Earlier quoted context omitted.
Why would you expect “brad Pitt email” to be something that auto completes? You would, on the other hand expect “Hillary Clinton email” to auto complete because there was a huge controversy about it. I’m not saying google is manipulating auto complete intentionally (though they might be), I’m just saying your counter examples are irrelevant. It would be like “Donald trump Russia” NOT auto completing then someone sayi…
The poster claimed Hillary Clinton was unique, meaning the only person that applied to. For me, she was not. Since she's not unique, then her being unique can't be used as evidence. Claiming that she's unusual , since you expect it to work for her based on stories written about her is a different claim.
You're misrepresenting my claim.
1. I said "only for Clinton" in the contect of Trump vs Clinton. Then I compared to other U.S. politicians, where the example still holds. The intended meaning is perfectly clear.
2. Obviously the fact that Clinton was the topic of a scandal involving email is the assumed context here. That's why I said "if anything, we would not expect "email" to be autocompleted for trump" (implied: but for Clinton, email is a more relevant search term based on published news, etc.).
Re: To Break Google’s Monopoly on Search, Make Its Index Public
#379So although everyone likes to believe google is a monopoly it’s far from it. You have choices- bing, biadu , yandex, DuckDuckGo... there is also nothing about googles search position that prevents you from building a competitor. What we do have is peter thiel backing an administration that’s anti google, Russia, China that are anti google. Why? it’s a source of truth that challenges their lies. We also have an emerge…
Google is a profit-seeking corporation that needs to expand into China to generate more money and bends the knee as much as it can to make inroads there. Far from being a source of truth they follow all Chinese censorship laws (why, because they would be blocked from operating there and unable to make money off of the Chinese). China likes everything about Google other than the fact that it is not a domestic corporat…
I saw Peter Thiel on one of the networks this morning pushing this message. The problem with his message (and yours) is that Google doesn't work with the Chinese Government. Dragonfly was shut down, and for the most part, Google is not even available in China.
Thiel's criticism was that because Google wasn't working with the U.S. Government, and was operating in China, that Google isn't patriotic.
Not only is this rhetoric a false equivocation, but it's dangerously wrong.
Re: To Break Google’s Monopoly on Search, Make Its Index Public
#380Ex-Google-Search engineer here, having also done some projects since leaving that involve data-mining publicly-available web documents. This proposal won't do very much. Indexing is the (relatively) easy part of building a search engine. CommonCrawl already indexes the top 3B+ pages on the web and makes it freely available on AWS. It costs about $50 to grep over it, $800 or so to run a moderately complex Hadoop job.…
>"Indexing is the (relatively) easy part of building a search engine. CommonCrawl already indexes the top 3B+ pages on the web and makes it freely available on AWS."
Interesting I would have thought that crawling at this scale and finishing in a reasonable amount of time would still be somewhat challenging. Might you have any suggested reading for how this is done in practice?
>"It costs about $50 to grep over it, $800 or so to run a moderately complex Hadoop job." Curious what type of Hadoop job you might referring to here. Would this be building smaller more specific indexes or simply sharding a master index?
>"Google hasn't used PageRank since 2006." Wow that's a long time now. What did they replace it with? Might you have any links regarding this?