To Break Google’s Monopoly on Search, Make Its Index Public • Hacker News Board

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#291

Post by jliptzin » Mon, Jul 15, 2019, 3:46 PM UTC

tryptophan wrote: post #103

noahl wrote: post #61

Okay, this is a relatively serious proposal to require Google to allow API access to its search index, with the premise that it would democratize the search engine ecosystem. There are some issues with the regulations he proposes (you have to allow throttling to prevent DDoS attacks, and you can't let anyone with API access add content to prevent garbage results), but it's roughly feasible. The main problem is, I thi…

>2) 20 years of experience fighting SEO spam. Tangential - but does anyone else feel that google results are useless a lot of the time? If you search for something, you will get 100% SEO optimized shitty ad-ridden blog/commercial pages giving surface level info about what you searched about. I find for programming/IT topics its pretty good, but for other topics it is horrible. Unless you are very specific with your s…

Yes, at least half the time I search about a particular topic, it seems the first few pages are written by some contractor in the Philippines probably getting paid $2 / hr who just spent the prior 30 minutes researching the topic.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#292

Post by eitland » Mon, Jul 15, 2019, 3:46 PM UTC

GordonS wrote: post #195

b_tterc_p wrote: post #134

Earlier quoted context omitted.

Do people do this? Or do you mean queries forwarded by home assistants trying to parse inputs?

Heh, yes, they do. Which is a reminder that devs are not "typical" users. As a developer, I search using keywords; for example, if I was looking for property for sale in Inverness, I might search for "property Inverness", whereas I've seen and heard "typical" users use something like "find me a 2 bedroom house with a garden for sale in the North of Inverness" - much more verbose, and containing stop words and phrases…

> As a developer, I search using keywords;

So did I.

Around the time I left Google behind I had started to search like my wife did, using full sentences. it sometimes worked better I think.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#293

Post by auslander » Mon, Jul 15, 2019, 3:47 PM UTC

It takes $Billions to get new search engine to same level as Google..

Candidates with $Bs to spare (!) are Apple, Facebook, Microsoft. First two never started it, Bing is quite stupid so far. My default SE in Bing. I wish it was Apple :)

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#294

Post by bodono » Mon, Jul 15, 2019, 3:49 PM UTC

scott_s wrote: post #289

bodono wrote: post #236

Earlier quoted context omitted.

Try "hillary clinton emai". From a fresh chrome session in NYC I get nothing, not a single autocomplete result. On the other hand "donald trump emai" gets: * donald trump email * donald trump email address * donald trump email list * donald trump email newsletter * donald trump email list signup And just to drive the point home I tried "root_axis emai" and got "root_axis email". Try anyone else and you get similar re…

I got curious and tried the names of a bunch of public figures. Some of " e" yielded "email" as the suggestion. But these did not: elizabeth holmes, tom jones, tom cruise, brad pitt, gwyneth paltrow, roger federer, will smith, jimmy carter. Since Hillary Clinton is not unique, then it's not proof that her results are treated differently.

They all work fine for me, e.g., 'tom jones emai' -> 'tom jones email', only Clinton didn't.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#295

Post by buildzr » Mon, Jul 15, 2019, 3:50 PM UTC

Angostura wrote: post #198

thaumasiotes wrote: post #191

Earlier quoted context omitted.

Those searches are unlikely to be unique.

Hmmm - Error: System.InvalidOperationException: The workflow with id=15f08b34-33f5-4063-8dea-d4ca6212c0d6 is no longer available. is not atypical.

Does that actually work? I must be old school, I always delete such IDs before searching, but then again I used Google back when it actually did what you told it instead of misinterpreting everything for you.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#296

Post by malvosenior » Mon, Jul 15, 2019, 3:50 PM UTC

scott_s wrote: post #289

bodono wrote: post #236

Earlier quoted context omitted.

Try "hillary clinton emai". From a fresh chrome session in NYC I get nothing, not a single autocomplete result. On the other hand "donald trump emai" gets: * donald trump email * donald trump email address * donald trump email list * donald trump email newsletter * donald trump email list signup And just to drive the point home I tried "root_axis emai" and got "root_axis email". Try anyone else and you get similar re…

I got curious and tried the names of a bunch of public figures. Some of " e" yielded "email" as the suggestion. But these did not: elizabeth holmes, tom jones, tom cruise, brad pitt, gwyneth paltrow, roger federer, will smith, jimmy carter. Since Hillary Clinton is not unique, then it's not proof that her results are treated differently.

None of those other people were involved in major stories with email in the headline. If you look at the actual results, you'll see that it doesn't make any sense to not get suggestions for "Hillary Clinton email".

Compare these results:

* https://www.google.com/search?hl=en&q=hillary%20clinton%20em...

* https://www.google.com/search?hl=en&q=will%20smith%20email

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#297

Post by Strilanc » Mon, Jul 15, 2019, 3:52 PM UTC

madez wrote: post #287

dkyc wrote: post #81

Earlier quoted context omitted.

> Yes, Google has a huge index, but most queries aren't in the long tail. I'm not quite sure about that. 15% of Google searches per day are unique, as in, Google has never seen them before. [1]. That's quite an insane number. [1] https://searchengineland.com/google-reaffirms-15-searches-ne...

> 15% of Google searches per day are unique, as in, Google has never seen them before. That is impossible, and therefore wrong (I'm wrong, please see below). To know if a search is unique, as in Google has never seen them before, Google must be able to decide if a query it receives was seen before or not. Even if we assume Google needed only one bit for each message it has ever seen, and assuming it only saw 15% of n…

How are you computing that number? It's definitely wrong.

Assume Google receives 1 trillion queries per year, and has been around for 20 years. Using a bloom filter you can achieve a 1% error rate with ~10 bits per item. So a 200 terabyte bloom filter would be more than sufficient to estimate the number of unique queries.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#298

Post by mattferderer » Mon, Jul 15, 2019, 3:52 PM UTC

exhaze wrote: post #216

mattferderer wrote: post #201

Is Google's Search quality that much better than Duck Duck Go or Bing? I get about the same quality of content when using any one of the three. All three are terrible at serving me up websites to buy crap when I typically want to learn about something. I would love a filter that says "Don't try to sell me anything"! I believe Google's biggest advantage is in their marketing & that the word "Google" now is a synonym f…

I don't know the answer to this - you could be right, but just outright questioning that Google's search quality is no better than DDG or Bing and basing it off of a single data point is pretty foolhardy. Your second statement reinforces this point - I've run a lot of Google ads, and they're quite effective. Google ads are very effective at selling things to people. In the future, when coming up with an opinion about…

I was looking for personal opinions to gather data from, hence the question. I was not trying to cite stats & persuade with anything more than my personal experience. It would have been very easy to link to articles such as this Yale study - https://www.networkworld.com/article/2225489/research-buries... if I wanted to do that. FYI, this argues the 2 are fairly close.

Also asking for personal opinions will give me much different results than looking up a blind test research as brand loyalty will play a big part in ones like Duck Duck Go.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#299

Post by r3bl » Mon, Jul 15, 2019, 3:52 PM UTC

madez wrote: post #287

dkyc wrote: post #81

Earlier quoted context omitted.

> Yes, Google has a huge index, but most queries aren't in the long tail. I'm not quite sure about that. 15% of Google searches per day are unique, as in, Google has never seen them before. [1]. That's quite an insane number. [1] https://searchengineland.com/google-reaffirms-15-searches-ne...

> 15% of Google searches per day are unique, as in, Google has never seen them before. That is impossible, and therefore wrong (I'm wrong, please see below). To know if a search is unique, as in Google has never seen them before, Google must be able to decide if a query it receives was seen before or not. Even if we assume Google needed only one bit for each message it has ever seen, and assuming it only saw 15% of n…

I believe that they're unique in a sense that nobody has typed in that exact query previously.

Of course, Google knows better but to treat every search query literally. Slight deviations and synonyms work for the majority of the people, even if us techies highly oppose them and look for alternative solutions (like DDG) that still treat our searches quite literally.

Re: To Break Google’s Monopoly on Search, Make Its Index Public

#300

Post by matheweis » Mon, Jul 15, 2019, 3:53 PM UTC

asark wrote: post #123

tryptophan wrote: post #103

Earlier quoted context omitted.

>2) 20 years of experience fighting SEO spam. Tangential - but does anyone else feel that google results are useless a lot of the time? If you search for something, you will get 100% SEO optimized shitty ad-ridden blog/commercial pages giving surface level info about what you searched about. I find for programming/IT topics its pretty good, but for other topics it is horrible. Unless you are very specific with your s…

Google signed an armistice in the Great Spamsite War some time around '08 or '09, to the effect that spam can have all the search results aside from those pointing at a few top, trusted sites, so long as they provide any content at all. Bad content is fine. Farmed content is fine. Content that was probably machine-generated is fine. Just content. Play the game, make sure your markov chain article generator or mechani…

There’s a reason for this; click through rate on ads is higher on pages that don’t achieve the user goal.

I suspect that the AI models powering the search results develop a sort of symbiotic relationship with the spam - if the user actually finds what they are looking for by clicking through an ad on an otherwise spammy page, everyone “wins”; the user found what they were looking for with minimum effort, google got their ad revenue, and the spammy page got a little cut for generating content that best approximating the local minimum that links the users keywords to actual intent...