We Need Viable Search Engine Competition • Hacker News Board

Re: We Need Viable Search Engine Competition

#71

Post by mtrimpe » Sat, Jan 04, 2014, 6:46 PM UTC

sigil wrote: post #44

Is PageRank really the indomitable tech of our generation? Nobody can do better algorithmically, or integrate some kind of crowd sourced feedback, or measure browsing time and habits, or simply hand tune some of the most competitive key phrases? I’m sure I’m oversimplifying, but I wonder if we haven’t all been hypnotised by the complexity, much of which is marketing hype... Actually, the tech is extremely complex, an…

When watching talks like http://www.youtube.com/watch?v=vShMxxqtDDs I'd be surprised if Google's search model couldn't be leap-frogged if a competitor were to have live access to Google's dataset.

That if is actually Google's biggest strategic advantage though: the resources to keep a near-live and nicely deduplicated copy of all relevant data on the internet. [PS1]

That's why by far the biggest strategic threat to Google is actually JavaScript UIs + RESTful APIs and the easily accessible data sources they form.

Google's demise will eventually come in the form of the adoption of protocols which allow you to efficiently maintain a live view of a service's public resources.

[PS1] And as per https://news.ycombinator.com/item?id=7011816 click-throughs of course, although I don't see a way to side-step that with Google having implemented [not provided]

Re: We Need Viable Search Engine Competition

#72

Post by mrfusion » Sat, Jan 04, 2014, 6:52 PM UTC

CombAce wrote: post #35

or: We need an inclusionist wikipedia - where every possible search term starts with an empty page/any possible search term would have an entry, even if it's just for redirecting a spelling mistake. Eventually every search term would grow its own little community of experts. Unfortunately, as pg said, "Deletionists rule Wikipedia."

That's a really smart idea. I wonder how you'd get a list of search queries to launch it.

Maybe you could require a tiny amount of bitcoin to edit a page to prevent spammers?

Re: We Need Viable Search Engine Competition

#73

Post by ghostdiver » Sat, Jan 04, 2014, 6:54 PM UTC

sigil wrote: post #44

Is PageRank really the indomitable tech of our generation? Nobody can do better algorithmically, or integrate some kind of crowd sourced feedback, or measure browsing time and habits, or simply hand tune some of the most competitive key phrases? I’m sure I’m oversimplifying, but I wonder if we haven’t all been hypnotised by the complexity, much of which is marketing hype... Actually, the tech is extremely complex, an…

Vertical search engine might be the answer, much easier to implement and much more useful from users point of view.

Re: We Need Viable Search Engine Competition

#74

Post by erikpukinskis » Sat, Jan 04, 2014, 6:56 PM UTC

sigil wrote: post #44

Is PageRank really the indomitable tech of our generation? Nobody can do better algorithmically, or integrate some kind of crowd sourced feedback, or measure browsing time and habits, or simply hand tune some of the most competitive key phrases? I’m sure I’m oversimplifying, but I wonder if we haven’t all been hypnotised by the complexity, much of which is marketing hype... Actually, the tech is extremely complex, an…

> Thought experiment: what could that disruptive niche be?

Someone needs to write/buy a search engine and then build a really rich API into the internals of it that lets 3rd parties write customized search engines. How about an auto parts search engine, or a search engine for spanish-speaking people living in southern california? What about a Christian search engine, or a meme search engine?

When someone can empower 3rd party developers to make the same kinds of decisions as Google does, but with different tradeoffs, and they really put the full institution behind supporting that, I think:

A) Google will have a hard time competing because they won't be able to give the personal attention to the needs of what is, effectively, a community of modmakers.

B) This new company will capture the long tail of search. Only a sliver of that is covered right now, with a scattering of niche search engines (Google Scholar, etc).

C) The number of users could be VERY large. It could be the cable to Google's broadcast television.

D) Getting started wouldn't require any massive technological achievements. Just find an underserved niche where even a really really stupid search engine would work better than Google. Write it, figure out how to make money off it it, grow. Start with something that requires only a small index. Slowly expand into additional niches according to what will help keep the company and the tech moving forward.

Google's bet is that any information you can glean from someone coming to a niche search engine can be reasonably approximated with contextual information in the query. That's proved true for many queries, but the key is to find the queries where it's really not.

Re: We Need Viable Search Engine Competition

#75

Post by throwawaykf03 » Sat, Jan 04, 2014, 6:56 PM UTC

jorgecastillo wrote: post #42

I don't think anything can compete with Google when it comes to search in English. I almost feel like Google can read my mind. With the adequate query you can find anything you've ever read on the internet if it's still online. Yesterday night I wanted to re-read a blog post about a guy that made a few bucks with QNX, but I didn't remember much I tried a few queries and I got the expected blog post with this: https:/…

Just tried it on Bing: http://www.bing.com/search?q=Netherlands+qnx+a+quick+buck

Worked for me.

Edit: Different query from yours, my bad. I mixed both your links up.

Re: We Need Viable Search Engine Competition

#76

Post by alan_cx » Sat, Jan 04, 2014, 7:01 PM UTC

nilved wrote: post #58

DuckDuckGo provides better results than Google. We have viable alternatives.

I am sorry to say this, but I have been trying to move to DDG for a while now. Its currently my default search where ever I can make it a default search, and frankly, for me, it's pretty much useless. Most results point to US sites, and in most of my search cases, thats no good. Unless my search is incredibly simple, I end up back on google.co.uk. OK, ethics wise, I don't like google one little bit, but in the end, google gives me useful results. DDG rarely does.

I simply do not know how people, and there is a lot of them, come to the conclusion that DDG results are in any way better than google, apart from the fact that DDG is not google. It would not totally surprise me if people were more making do with DDG because its not google to make a point, and as a result being over generous about its utility.

Maybe for Americans, DDG does give better results, I cant say, but I'd like to know on what criteria that is based. But for me, a non American, like I say, DDG is regrettably mostly useless.

Re: We Need Viable Search Engine Competition

#77

Post by Isofarro » Sat, Jan 04, 2014, 7:10 PM UTC

erikpukinskis wrote: post #74

sigil wrote: post #44

Is PageRank really the indomitable tech of our generation? Nobody can do better algorithmically, or integrate some kind of crowd sourced feedback, or measure browsing time and habits, or simply hand tune some of the most competitive key phrases? I’m sure I’m oversimplifying, but I wonder if we haven’t all been hypnotised by the complexity, much of which is marketing hype... Actually, the tech is extremely complex, an…

> Thought experiment: what could that disruptive niche be? Someone needs to write/buy a search engine and then build a really rich API into the internals of it that lets 3rd parties write customized search engines. How about an auto parts search engine, or a search engine for spanish-speaking people living in southern california? What about a Christian search engine, or a meme search engine? When someone can empower…

Sounds like Yahoo Boss: http://developer.yahoo.com/boss/ And that's been around for years already.

Essentially, you get the Yahoo results, and decorate, filter, reorganise, improve, combine as you wish. So you get the organic results, which you can then innovate upon.

And it has survived the transition from Yahoo-powered search results, to Bing powered search results: http://searchengineland.com/yahoo-boss-moves-from-yahoos-ind...

Which means you have API access to the second most-used search engine in the industry. So what better way to voice your discontent with Google by supporting a competitor.

Running a successful search engine is expensive, it needs continuous investment into R&D. That's why Yahoo took a step back and partnered with Bing instead. The level of investment needed just to hold status quo with the existing market runs into billions of dollars a year, something Yahoo baulked at. Microsoft, however, were still strongly inclined to invest that every year.

Re: We Need Viable Search Engine Competition

#78

Post by saurik » Sat, Jan 04, 2014, 7:15 PM UTC

erikpukinskis wrote: post #74

sigil wrote: post #44

Is PageRank really the indomitable tech of our generation? Nobody can do better algorithmically, or integrate some kind of crowd sourced feedback, or measure browsing time and habits, or simply hand tune some of the most competitive key phrases? I’m sure I’m oversimplifying, but I wonder if we haven’t all been hypnotised by the complexity, much of which is marketing hype... Actually, the tech is extremely complex, an…

> Thought experiment: what could that disruptive niche be? Someone needs to write/buy a search engine and then build a really rich API into the internals of it that lets 3rd parties write customized search engines. How about an auto parts search engine, or a search engine for spanish-speaking people living in southern california? What about a Christian search engine, or a meme search engine? When someone can empower…

http://aws.typepad.com/aws/2005/12/alexa_web_searc.html

Re: We Need Viable Search Engine Competition

#79

Post by nostrademons » Sat, Jan 04, 2014, 7:28 PM UTC

greglindahl wrote: post #43

Aqueous wrote: post #5

Part of the problem is that Google's PageRank algorithm is patented and the patent won't expire until 2017. Google insists that its current search mix is based on a lot more than just PageRank, but it seems that PageRank probably contributes the foundation of their business. I don't see a way of competing with Google's results unless we are allowed to use something like PageRank, which we can't do unless we pay royal…

The PageRank patent is owned by Stanford; anyone can buy a license. And BTW, it isn't necessary or even that useful these days to compute it.

I'd thought that Google had an exclusive license, but I just checked and the exclusive license expired in 2011:

http://www.seroundtable.com/pagerank-patent-12731.html

So yes, anyone can license PageRank now.

Re: We Need Viable Search Engine Competition

#80

Post by freshhawk » Sat, Jan 04, 2014, 7:33 PM UTC

rogerbinns wrote: post #57

sigil wrote: post #44

Is PageRank really the indomitable tech of our generation? Nobody can do better algorithmically, or integrate some kind of crowd sourced feedback, or measure browsing time and habits, or simply hand tune some of the most competitive key phrases? I’m sure I’m oversimplifying, but I wonder if we haven’t all been hypnotised by the complexity, much of which is marketing hype... Actually, the tech is extremely complex, an…

> Even then, is DDG viable competition ... An irritation for me with DDG is them formatting result URLs incorrectly, and then ignoring any feedback about it. Two examples are them adding spurious www. prefixes to Google code results, and leaving out slashes when creating Apple developer URLs. DDG could distinguish themselves by playing bazaar to Google's cathedral, but they don't appear to. A search engine that uses…

Crowd sourcing would be really tricky. It's inherently easier to manipulate, the more traction you got the more resources would be spent on gaming the crowd sourcing system. Look at what those "reputation management" firms are doing these days with sophisticated blackhat SEO and wikipedia astroturfing rings for example.

Some kind of bitcoin-esque system where you bought sponsored positions by doing search engine scoring related number crunching could be interesting though. Since, if you ever matter, you'll have people devoting resources to trying to game your search engine results you need some system to deal with that. Of course it's sooo "out of the box" to suggest a bitcoin inspired solution to things right now ...