31–40 of 45 posts
Re-inventing crawling, relevance clustering, etc isn't worth the trouble or the cost. Finding ways to enhance a specific market segment however would be a differentiater worth pursuing.
disclaimer: I work for Y!
The more narrowly focused you get, the less keyword rich the data is likely to become. This creates obvious problems. I've been working on a search engine where last.fm is a major source of data, and their data is comprehensive, but keyword poor. How to work around this? There are ways but they're far from trivial or resource friendly.
I would focus on anticipating the user's search needs even before he/she formulates them and types out a search query. I would use as much context that a user is willing to provide me - location, recent email messages, voice call transcripts, unread messages, web browsing history, etc - to try to anticipate what the user is likely to query for. For example, a sales engineer who receives a technical query in his email…
Mobile search today is an absolute fright.
(Google is about as good at mobile search as Alta Vista was at web search 10 years ago. Why not be this decade's Google and show them how it could be done?)
Index the data: Figure out a way to index the massive volume of data that's not a live query system (lucene starts to gag at queries > 50GB or so), so generate all possible results for all possible queries in a database and update those results occasionally as you pull in more data. When someone queries for a search string, pull up the results from a pre-generated query from a database, don't do a live search of all of your terrabytes of data, or the query will take days.
Generate interesting results: find a niche. Don't plan to take on Google, Bing & Yahoo on a personal scale. People put together good engines, but target main sites, and front pages, or a shallow-depth crawl. Don't plan on indexing every forum and every blog on the internet.
I've been impressed by this guy's search engine: http://gigablast.com/
In short, unless you've got a TON of money, machinery, people and time, don't try to compete with Google. Find a niche like shopping search or movie search, or be human-powered like Mahalo. Google's got a dedicated computer for every possible search query out there or close to it, plus a team of 500,000 Chinese people making sure that popular results are relevant (Google does human-validated results for many of the most popular queries, not like an error message query).
Yahoo treated the web like a phone book or directory. Altavista relied on self-categorization efforts in meta tags. Google treated links as votes.
You need to come up with a new (and hopefully better) way of thinking about what the web is. Come up with an inventive way of thinking about what linking means, what DOM structure means, how to think about non-standard types of content, and so on.
If you start with the same premises about the web that Google started with in 1997, you'll never surpass them much less carve about anything more than a toy niche.
I would focus on anticipating the user's search needs even before he/she formulates them and types out a search query. I would use as much context that a user is willing to provide me - location, recent email messages, voice call transcripts, unread messages, web browsing history, etc - to try to anticipate what the user is likely to query for. For example, a sales engineer who receives a technical query in his email…
If you can anticipate a user's search needs, why not take the next step and provide the answers before the user gets around to searching?
What you suggest would benefit from the performance/feedback from what the OP suggests.
This is basically what I am interested on working on currently , using "clues" gleaned from what the user is doing (a la RescueTime) to reorganize/reformulate a user's search query.
Earlier quoted context omitted.
If you can anticipate a user's search needs, why not take the next step and provide the answers before the user gets around to searching?
One step at a time :) What you suggest would benefit from the performance/feedback from what the OP suggests. This is basically what I am interested on working on currently , using "clues" gleaned from what the user is doing (a la RescueTime) to reorganize/reformulate a user's search query.