Ask HN: Is there room for another search engine?

Here's what I want in a search engine:

Charge me $15-25 dollars per year

Let me decide what demographic information I wish to share- make it easy for me to control and help me protect my information. Because you are charging me money you can afford it and I trust you.

Give me two search options: one, I'm only seeking information. two, I'm looking to buy. Do this for me as an advertiser: help me qualify the clicks I'm paying for

Perhaps allow me to pay per 1000 impressions (CPM) instead of per click.

By the way, I would also subscribe to a facebook that did this.

If by "search engine", you mean something similar to Google/Bing then probably not.

However, if we expand the concept of "search" to something beyond text on webpages and "engine" to something beyond a linear algebra pagerank problem that weighs url links, there's room for many more competitors.

Let's say we want to search for "best restaurant":

Method #1 might be searching millions of web pages, twitter posts, newspaper archives, etc where ngram such as "best restaurant" is mentioned. That's what Google/Bing engines already do.

Method #2 might rank restaurants by collecting crowd-sourced opinions. That's what Yelp & Tripadvisor does. (Although Google also piggybacks on their data and lists yelp pages in SRP.)

Method #3 might be a company like Visa/Mastercard analyzing their billions of transactions[1] and based on actual spending amounts & frequency of a billion cardholders, they can also provide their own calculation of a "best restaurant". (I know that Visa/MC already offer limited marketing data to some entities but they don't surface that data to every day web surfers.)

The idea is that there's plenty of room for more imaginative scenarios of #2 & #3. The common theme is that Google doesn't have the data (e.g. credit-card transactions) and therefore, the new "search engines" can give fresh answers that Google algorithms can't provide. To try and boil it down to a simple question: "What interesting answers can a new engine provide that _can't_ be extracted from the text of webpages?"

Btw, I ran across some posts from a Microsoft employee (but not a Bing team member) stating his opinions on building competing search engines. https://news.ycombinator.com/item?id=7011472

[1] http://marketrealist.com/2016/10/why-visas-processing-and-in...

I'd argue that paywalled sites, Captcha-guarded pages, things like that are second-class content and should be treated as such. If somebody wants to dig deeper and see such results, that should be an option, but as you said, this is a tough problem to solve satisfactorily and the truth lies somewhere inbetween. An obstacle to this happening right now is that Google doesn't want to shake the advertising tree too hard. Their business model is spammy so they tolerate spamminess. That's the hardest problem of all, and one I don't think anybody's close to solving yet: How do you monetize web search without automatically wrapping results in ads? But some bright person will come up with an answer and hopefully defeat this huge conflict of interest.

Google's crawlers don't exactly perform 2FA before they crawl a website. If impersonating Google's crawlers doesn't suit your fancy, there are all manner of ways to anonymize a crawler. So I think blocking is one of the least interesting challenges. The other side of the coin is robots.txt is not inherently adversarial, and a crawler could waste quite some time and energy crawling truly meaningless content. That, in my mind, is the interesting challenge: Obsoleting robots.txt with an intelligent (and gentle) crawler.

There are already many "serious upstarts" that are doing just fine but will never reach the status of Google, like duckduckgo, bing, etc.

Some even say bing has better search quality than google. But that's not what matters. No matter how good the product is, if you can't get distribution it doesn't matter. Also, even if you do have distribution, it may not be the right context so it won't be effective.

If anyone is trying to really build a search engine to compete directly against Google, I can only say good luck, and prepare to be the most patient person on earth, because it won't happen fast, and probably won't even happen. Instead, while you're slowly making progress day by day with small traffic increase, you'll see some random new thing come out that had no intention of becoming a "search engine" (just like Youtube) and "disrupt" Google.

I have seen a lot of people talk about "disrupting something", and also have seen a lot of people who actually have disrupted their industry. The former are just wannabes, and the latter never said anything about "disrupting" something. They just saw an opportunity and went for it. And it ended up disrupting. Hope this makes sense.

p.s. Google doesn't operate on the original pagerank anymore, there are tons of other things going on underneath. Also they now own many innovative AND popular "search UI" both through acquisitions and their R&D. Google Maps owns a lot of the map space, Youtube owns the video search space, and so on. Look deeper and you'll find even more that you've been taking for granted.