Earlier quoted context omitted.
> What we really need is a new Google, built on open principles (decentralized / peer to peer, fully free software, backed by a nonprofit), We have them, they suck. > and focused on indexing the long tail of insightful content that is neglected by Google because it lacks SEO How would you even define that? SEO is changing all the time. And google is fighting it all the time. And how would you prevent SEO focusing on…
I think the goal is to exclude anything that was created with a popular CMS (to exclude some content marketing), or in the top 100k websites. Then we might go back to something sort of interesting.
I wonder whether doing a parallel search on google and filter out by their top-results from your own results would be a feasiable solution? Add a filter on the top 500 websites, and whether known ad-sources are used and you might get slowly there.
Maybe instead of a smart searchengine it would be better to focus on a dumb focus which gives access to all the metadata of a page too, and allows people to optimize for themself. Fulltext alone is not the only relevant content for good results. Google knows it and uses them, but has very limited acces to it for the enduser.