Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We Need Viable Search Engine Competition

peebs.org

Post reply on HN
Print view View original

131–140 of 146 posts

Re: We Need Viable Search Engine Competition

#131

Post by bborud »

Aqueous wrote: post #5

Part of the problem is that Google's PageRank algorithm is patented and the patent won't expire until 2017. Google insists that its current search mix is based on a lot more than just PageRank, but it seems that PageRank probably contributes the foundation of their business. I don't see a way of competing with Google's results unless we are allowed to use something like PageRank, which we can't do unless we pay royal…

Two facts:

a) PageRank is just one of several hundred factors used for determining ranking. Search engine ranking is a lot more complicated than even most information retrieval-programmers tend to think.

b) There may have been a window of a few weeks or months in 99/00 where not every major search engine used some form of link-based ranking, but it was, as noted, a very brief period.

It always amazes me that people for almost 15 years now have believed in the myth of PageRank's uniqueness and power. I would like to challenge people to think about two things:

- how useful do you think a pure static ranking of web tens of billions of web pages is? Think about what it represents. What does it mean to assign one page higher rank than another?

- do you really believe that other search engines would not have implemented PageRank or something similar? Do you really think that search engine designers do not read papers and apply every trick in the book that they can manage to implement in a scalable manner?

There are lots of hard problems you need to solve if you want to build a web scale search engine. If ranking becomes your biggest problem: that would be a luxury. The biggest hurdle today is money to buy computing power and storage. The time when you could build a competitive web scale search engine for regular startup-money is over. It has been over for close to a decade.

It saddened me greatly when Yahoo threw in the towel because it meant that search in the western world was effectively a two-horse race. And once you get off that horse there is no getting back on it again without some seriously heavy lifting.

Re: We Need Viable Search Engine Competition

#132

Post by bborud »

nemesisj wrote: post #95
antics wrote: post #87

The comments here about building search engines that viably compete with Google in terms of quality are hilariously wrong. Hilariously wrong. The dominating cost is not hiring smart people to work on the problem, it's hiring enough smart people to work on the problem. Consider. Here at Microsoft, we employ a search relevance staff of -- what, 20% the size of Google's? And Google's engineers are rock solid. MS might b…

You know what? I really can't get behind the idea that MS is somehow the stirling example of engineering prowess and discipline. This is the organisation who fails their way through every other operating system. One of the poster children for missing the boat on the internet, tablets, operating systems, cloud services, and more. I also reject this idea that Google's got a lock on every smart person ever. That they do…

The difference between building an operating system and building a web scale search engine is that the former is mostly just work and there is a lot of material to learn from in order to inform that work. Sure, it requires talent, skill and perseverance, but there is a shitload of operating systems you can study. It also doesn't require tons of hardware, nuclear-reactor-scale power and enough bandwidth to rapidly copy, and keep updated, a significant portion of the web.

Sure, just making an OS doesn't mean it takes over the world. But guess what: you have the same problem with search engines.

Re: We Need Viable Search Engine Competition

#133

Post by wglb »

devcpp wrote: post #119
tptacek wrote: post #117

Earlier quoted context omitted.

You could point out all the lame exploits and bugs, but only if you want to spark a long argument that you will end up losing. There are a variety of things I don't like about Windows, and I avoid using it. But the idea that it's somehow less secure than other operating systems is for the most part a Linux advocacy myth. The fundamental security architecture of WinAPI is just not that different from that of Linux or…

Fair enough, you know much more than me in that domain. :) Are you going to argue about the famous general instability of Windows compared to its large competitors though? It seems like a good indicator of bad design in low level implementations.

What do you mean by famous general instability of Windows? Windows is perhaps my 3rd choice for OS, but almost all my clients use Windows and among the problems they deal with, instability is not one of them.

Re: We Need Viable Search Engine Competition

#134

Post by tlogan »

antics wrote: post #87

The comments here about building search engines that viably compete with Google in terms of quality are hilariously wrong. Hilariously wrong. The dominating cost is not hiring smart people to work on the problem, it's hiring enough smart people to work on the problem. Consider. Here at Microsoft, we employ a search relevance staff of -- what, 20% the size of Google's? And Google's engineers are rock solid. MS might b…

I disagree that this is the main reason why MS is behind Google on search.

I think the main problem of MS is that it is copying Google search and not providing anything substantially different or better. In other words, fighting against incumbent is not about spending more money than incumbent and doing the same. It is providing service for "niches" where incumbent is not willing or not able to provide service.

Here is an example:

Google shuts down Google Code search. I'm not happy. But, I was thinking... There is Bing... They will maybe jump in and provide state of art code search. But... nothing happen.

Re: We Need Viable Search Engine Competition

#136

Post by gesman »

lubujackson wrote: post #98
gesman wrote: post #67

One of the viable venues to compete with Google is not by developing "new, better search engine", but rather "better vertical platform". What's the best search engine where buyers go? Not google.com and never really been google, it's amazon.com and ebay.com. What's the best search engine to find answers on technical questions? Stackoverflow.com is approaching google very fast. So the answer would be for sharp minds t…

That's exactly what my startup is trying to address - how do you help users navigate 100, 1000 or even a million search engines? Ultimately, you need a way to browse search engines by topic and bundle them into custom lists for your specific needs. It's early going, but check Nuggety out if you're interested: http://nuggety.com/

Looks neat.

Search for "cms" does not return "sitecore" anywhere.

Re: We Need Viable Search Engine Competition

#137

Post by lubujackson »

namenotrequired wrote: post #125
lubujackson wrote: post #98

Earlier quoted context omitted.

That's exactly what my startup is trying to address - how do you help users navigate 100, 1000 or even a million search engines? Ultimately, you need a way to browse search engines by topic and bundle them into custom lists for your specific needs. It's early going, but check Nuggety out if you're interested: http://nuggety.com/

I'm interested, but I've checked the front page and am still confused what / who it's for, could you clarify?

Sure. I know the site needs some more clarity and that's what I'm working on now. The basic idea is a Pinterest-type community where you can build a list of sites on any topic and share lists with others. But each list lets you search each website directly from the same search box (results come up in a new tab, like Kayak).

So what is it good for? Any subject where you want depth or breadth over surface results. So collectors and researchers can build a portal for themselves. And there are some interesting possibilities for specific needs, like a search list for many places to find a cached URL: http://nuggety.com/u/nuggety/cached-webpages or a search list to search any of Google's 190 international websites: http://nuggety.com/u/nuggety/international-google

Re: We Need Viable Search Engine Competition

#138

Post by dreamfactory »

antics wrote: post #96
nemesisj wrote: post #95

Earlier quoted context omitted.

You know what? I really can't get behind the idea that MS is somehow the stirling example of engineering prowess and discipline. This is the organisation who fails their way through every other operating system. One of the poster children for missing the boat on the internet, tablets, operating systems, cloud services, and more. I also reject this idea that Google's got a lock on every smart person ever. That they do…

Hi OP. Your points, as I understand them, are the following. Correct me if I'm wrong. * MS is not a good example of a strong engineering org. Further, it is not a good example because Windows sucks. * Google doesn't employ every smart person ever. * Google can't actually compete. (ed note: in any field, or just against iOS and Fb?) * My argument is like the argument that no one would supplant Windows. * Given enough…

> For starters, we built the entire Bing stack from scratch. No OSS. No common platforms like the JVM. Nothing like that. We started from nothing, and invented the server infrastructure, the data pipeline, the runtime that would support the site, the ML tools, everything. The fact that the site runs at all is a small miracle, but the site does not "just" run: the most remarkable thing by far is that the quality of our tooling is quite incredible, generally an order of magnitude better than the OSS equivalents. For example, the largest deployment of an OSS NoSQL datastore seems to be a few thousand nodes. The small NoSQL cluster backing our MapReduce implementation is stably deployed on a cluster an order of magnitude larger than this. This is something you only really see at companies like Amazon, Google, or MS.

You lost me here. Not building on OSS seems like setting yourself up or failure from the start, particularly when you are fighting a manpower war, which is where OSS is beating every proprietary entity. OSS already powers Google and Amazon and OSS db's will scale to billions of nodes, not a few thousand.

Re: We Need Viable Search Engine Competition

#139

Post by Aqueous »

bborud wrote: post #131
Aqueous wrote: post #5

Part of the problem is that Google's PageRank algorithm is patented and the patent won't expire until 2017. Google insists that its current search mix is based on a lot more than just PageRank, but it seems that PageRank probably contributes the foundation of their business. I don't see a way of competing with Google's results unless we are allowed to use something like PageRank, which we can't do unless we pay royal…

Two facts: a) PageRank is just one of several hundred factors used for determining ranking. Search engine ranking is a lot more complicated than even most information retrieval-programmers tend to think. b) There may have been a window of a few weeks or months in 99/00 where not every major search engine used some form of link-based ranking, but it was, as noted, a very brief period. It always amazes me that people f…

It's not really a static ranking, though, is it? The PageRank is constantly recalculated when incoming links change.

I don't know what other search engines have implemented because none of them will let me see their code :-) I do believe if they could implement a link-based ranking system without fear of being sued by Stanford they would. I don't know how different a link-based ranking system would have to be from PageRank to avoid getting sued by Stanford, or whether Stanford litigates this when they suspect unlicensed use. I'm guessing they do sue to defend the patent, because Google's royalties for using the algorithm number in the hundreds of millions, a significant amount towards Stanford's endowment.

Re: We Need Viable Search Engine Competition

#140

Post by Aqueous »

greglindahl wrote: post #43
Aqueous wrote: post #5

Part of the problem is that Google's PageRank algorithm is patented and the patent won't expire until 2017. Google insists that its current search mix is based on a lot more than just PageRank, but it seems that PageRank probably contributes the foundation of their business. I don't see a way of competing with Google's results unless we are allowed to use something like PageRank, which we can't do unless we pay royal…

The PageRank patent is owned by Stanford; anyone can buy a license. And BTW, it isn't necessary or even that useful these days to compute it.

Right. But that is the prohibitive factor for many startups. They cannot afford nor are they willing to pay Stanford for PageRank.
Post reply on HN