Ask HN: What strategy would you take to build a search engine today?

I would aggregate all the knowledge in linkdumps / fora where voting is possible, possibly seed the database by creating a giant repository of links such as a social bookmarking site.

It would get built up slowly but it would be of a higher quality than you could probably reach with crawling. Effectively you'd be crowdsourcing the ratings system of your search engine.

A karma system would keep the spammers out, or at least have them identified fast enough.

Of course there are plenty of things that would need to be fleshed out wrt to abuse potential and 'gaming the system' but I think it would stand a fighting chance.

Crawling the web is just going to get you tons of garbage, comparable with moving the contents of the local trash dump into your house because you know there must be a pair of earrings in there somewhere.

There will come a time when 'pagecount' is not the defining measure of search engine quality. Google seems to be on to that because they've dropped their 'indexing xxx pages' message from the homepage long ago.

I would also allow users to 'hide' certain domains from their future search results, and use that to help identifying sites that contain mostly (or exclusively) trash.

There's room for a search engine that can take more advanced boolean queries and can handle custom ranking. For example, a while ago I was trying to find a section on a blog that mention someone whose last name is "White". The blog mentions politics a lot, so nearly every article also had the phrase "white house". It would be nice to search for mentions on the blog of "white" not including "white house," but not excluding it, either.

The closest you can get on Google is to search for white and NOT "white house," then search for "white house," and search within the results for "white" to see if anything else pops up.

Unless you are ready to get your hands dirty with the semantic side of search, I would focus on creating a more enjoyable and emotionally appealing experience for a particular demographic of consumers. Maybe teens, students, mothers, whatever...focus on building a loyal, targeted user base.

Otherwise you run the risk of just being a couple cool features that the big boys can use as inspiration for their own work.

There is a lot of possibility to carve your niche, although this is true for most software products (a product can be considered a market leader in one vertical, yet almost unknown in another).

Whether you choose to seek out a niche or not, you still need a novel approach. I know one search company that in 2000/2001 started building a solution that relied heavily on in-memory indexes. At the time memory was expensive but it was a smart play b/c it became cheap very quickly which gave them a huge advantage in the depth and breadth of calculations they could do. They got a contract with Verizon to provide search for superpages.com and became quite popular in the YP space. They were acquired in 2007.

I mention the story because it's important to remember that there are many, many successful companies doing cool stuff that we've never heard about. That's the norm. The ones we do hear about are the outliers.

PowerSet comes to mind.

How hard was it to do what they did? (I know nothing about semantic search with the exception of what it is.)