Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What strategy would you take to build a search engine today?

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 45 posts

Re: Ask HN: What strategy would you take to build a search engine today?

#21

Post by jdrock »

matt1 wrote: post #19
moonchuck wrote: post #15

Unless you are ready to get your hands dirty with the semantic side of search, I would focus on creating a more enjoyable and emotionally appealing experience for a particular demographic of consumers. Maybe teens, students, mothers, whatever...focus on building a loyal, targeted user base. Otherwise you run the risk of just being a couple cool features that the big boys can use as inspiration for their own work.

PowerSet comes to mind. How hard was it to do what they did? (I know nothing about semantic search with the exception of what it is.)

It's hard. Natural language processing and semantic analysis is a fairly deep field of knowledge.

One evidence of its complexity is PowerSet itself. PowerSet launched with just being able to search Wikipedia. Wikipedia is a highly, highly structured body of text that is much, much easier for NLP and semantic technology to analyze. Taking the same technology to the garbled soup that is the web is a whole different ballgame.

Re: Ask HN: What strategy would you take to build a search engine today?

#22

Post by ajordan »

yannis wrote: post #9

Search technology - perhaps because of the cost in setting up - has stayed behind. I would like to see: (01)User ability to adjust the 'algo' for ranking results. I may want all the newer websites and news in my ield rather than the websites with the highest page rank. (02) Ability to distinguish an 'authority website', i.e I search for Topic X, I do not want the wikipedia. I want the website perhaps of a Ph.D. stude…

http://hounder.org/ does 01 and 03 quite well, and 02 through bayesian filters you can train to find and rank relevant sites.

:)

Re: Ask HN: What strategy would you take to build a search engine today?

#23

Post by jdrock »

jpeterson wrote: post #10
jdrock wrote: post #4

Use us. http://www.80legs.com . :)

Why are comments like this sometimes voted to the top, and other times buried with downvotes? Where's the line between good self-promotion and spam? Seriously asking.

For what it's worth, we actually are helping new search engines scale their capabilities. ..and not just on crawling (or indexing, as most people confuse it), but also on applying their search technology.

There are a ton of really cool search technologies (NLP, deep web, media analysis, etc.) out there that don't have the processing power to apply their stuff to billions of web pages. That's a big stumbling block for these guys. Google's #1 competitive advantage (IMHO) is its low operating cost (achieved through hundreds of millions of dollars in CapEx).

Re: Ask HN: What strategy would you take to build a search engine today?

#24

Post by jdrock »

jacquesm wrote: post #11

I would aggregate all the knowledge in linkdumps / fora where voting is possible, possibly seed the database by creating a giant repository of links such as a social bookmarking site. It would get built up slowly but it would be of a higher quality than you could probably reach with crawling. Effectively you'd be crowdsourcing the ratings system of your search engine. A karma system would keep the spammers out, or at…

This approach would be suitable for "mainstream" search results, but wouldn't be able to serve the long tail.

Or to put it more humorously, you wouldn't be able to find the earrings because only you (and no one else) were interested in them :)

Re: Ask HN: What strategy would you take to build a search engine today?

#25

Post by jacquesm »

jdrock wrote: post #24
jacquesm wrote: post #11

I would aggregate all the knowledge in linkdumps / fora where voting is possible, possibly seed the database by creating a giant repository of links such as a social bookmarking site. It would get built up slowly but it would be of a higher quality than you could probably reach with crawling. Effectively you'd be crowdsourcing the ratings system of your search engine. A karma system would keep the spammers out, or at…

This approach would be suitable for "mainstream" search results, but wouldn't be able to serve the long tail. Or to put it more humorously, you wouldn't be able to find the earrings because only you (and no one else) were interested in them :)

I'm not so sure about that.

The 'long tail' in search is - for google at least - everything beyond page 100 (or position 1,000 , which ever way you want to slice it). Those pages might just as well not exist for those keywords. But because there is plenty of other content nobody notices.

The situation with the 'long tail' for pagesets where there are less than 1,000 'results' can be handled the exact same way it is being done today (here the long tail probably refers to rarity of search keywords / combinations).

For really small sets (1 full page of results or less) the ranking is pretty much irrelevant.

Re: Ask HN: What strategy would you take to build a search engine today?

#26

Post by jdrock »

jacquesm wrote: post #25
jdrock wrote: post #24

Earlier quoted context omitted.

This approach would be suitable for "mainstream" search results, but wouldn't be able to serve the long tail. Or to put it more humorously, you wouldn't be able to find the earrings because only you (and no one else) were interested in them :)

I'm not so sure about that. The 'long tail' in search is - for google at least - everything beyond page 100 (or position 1,000 , which ever way you want to slice it). Those pages might just as well not exist for those keywords. But because there is plenty of other content nobody notices. The situation with the 'long tail' for pagesets where there are less than 1,000 'results' can be handled the exact same way it is b…

Let me clarify..

How would you know how to value votes for different queries? Certain pages are more relevant to certain queries more so than others, and likewise for votes. Your approach doesn't have a way of accounting for this. And implementing something that does is a very, very hard problem.

Re: Ask HN: What strategy would you take to build a search engine today?

#27

Post by jacquesm »

jdrock wrote: post #26
jacquesm wrote: post #25

Earlier quoted context omitted.

I'm not so sure about that. The 'long tail' in search is - for google at least - everything beyond page 100 (or position 1,000 , which ever way you want to slice it). Those pages might just as well not exist for those keywords. But because there is plenty of other content nobody notices. The situation with the 'long tail' for pagesets where there are less than 1,000 'results' can be handled the exact same way it is b…

Let me clarify.. How would you know how to value votes for different queries? Certain pages are more relevant to certain queries more so than others, and likewise for votes. Your approach doesn't have a way of accounting for this. And implementing something that does is a very, very hard problem.

That's true, you would not have the 'text in the link' to guide you.

But you might be able to get around some of that by allowing users to tag the urls.

I realize it's a hard problem, I assume that the OP does not expect to walk out of here with a bullet proof business plan for a new search engine. There are bound to be issues with almost any suggestion that you could make here.

But it might give some useful hint or starting point.

Re: Ask HN: What strategy would you take to build a search engine today?

#28

Post by jdrock »

So most of the ideas mentioned here don't really consider the costs involved with making a search engine. New technologies/concepts/ideas are great and all, but unless you're building a very niche/vertical search engine, you're going to require several million dollars in servers to build a search engine.

Any strategy on building a search engine needs to address the costs. (Raising VC money is not an answer.)

Re: Ask HN: What strategy would you take to build a search engine today?

#29

Post by charlesju »

Search is a function of 2 problems:

1. Mainstream Search - This is the search for information that a lot of people want to know. Britney Spears, How good is the new transformer's movie, 1 + 1 = ?, etc. I think I would just go through slowly and optimize each page to show results from the various information portals on the web, then make competing websites bid for positions. ie. list game reviews from gamespot, ign, etc. with rotten tomatoe's algorithm.

2. Long Tail Search - This is random information throughout the web. I don't think there is a better way to aggregate this data than what traditional search engines are doing. Perhaps look into more advanced spam filtering algorithms, but that's a tweak, not a feature revolution. - Probably just use something like Yahoo Boss to get started

Re: Ask HN: What strategy would you take to build a search engine today?

#30

Post by moonchuck »

matt1 wrote: post #19
moonchuck wrote: post #15

Unless you are ready to get your hands dirty with the semantic side of search, I would focus on creating a more enjoyable and emotionally appealing experience for a particular demographic of consumers. Maybe teens, students, mothers, whatever...focus on building a loyal, targeted user base. Otherwise you run the risk of just being a couple cool features that the big boys can use as inspiration for their own work.

PowerSet comes to mind. How hard was it to do what they did? (I know nothing about semantic search with the exception of what it is.)

If you know nothing about it and are looking to launch a product anytime soon, I would focus on targeting a particular demographic with better design, marketing and demographic-specific content.
Post reply on HN