A search engine that favors text-heavy sites and punishes modern web design
search.marginalia.nu
A search engine that favors text-heavy sites and punishes modern web design
1–10 of 735 posts
Re: A search engine that favors text-heavy sites and punishes modern web design
#2Great idea, awful UI
Re: A search engine that favors text-heavy sites and punishes modern web design
#3Yeah so this is my project. It's very much a work in progress, but occasionally I think it works remarkably well for something I cobbled together alone out of consumer hardware and home-made code :-)
Re: A search engine that favors text-heavy sites and punishes modern web design
#4See also: https://wiby.me/
Re: A search engine that favors text-heavy sites and punishes modern web design
#5See also: https://wiby.me/
The "surprise me..." button is adequately labelled.
Re: A search engine that favors text-heavy sites and punishes modern web design
#6As everything in life flows in cycle, I predict the search engine that will de-throne Google will be like Google when it started - a simple variation of page rank.
No smarts, no bubble, no signals decided by over fitting to a biased engineer preference.
Re: A search engine that favors text-heavy sites and punishes modern web design
#7Where does the data come from? Do you index the whole web yourself? I see it totally impossible for a personal project. I'm very curious about that.
Re: A search engine that favors text-heavy sites and punishes modern web design
#8Where does the data come from? Do you index the whole web yourself? I see it totally impossible for a personal project. I'm very curious about that.
I do indeed index the web myself. Not the entire web, just a subset of it. The crawler quickly loses interest in javascript:y websites and only indexes at depth those websites that are simple. It also focuses on websites in English, Swedish and Latin and tries to identify and ignore the rest (best-effort).
You'd be surprised how much you can do with modern hardware if you are scrappy. The current index is about 17.7 million URLs. I've gone as far as 50 million and could probably double that if I really wanted to. The difficulty isn't having a small enough index, but rather having a relevant enough index, weeding out the link farms and stuff that just take space.
I only index N-grams of up to 4 words, carefully chosen to be useful. The search engine, right now, is backed by a 317 Gb reverse index and a 5.2 Gb dictionary.
Re: A search engine that favors text-heavy sites and punishes modern web design
#9(Old man yells at cloud.)
Re: A search engine that favors text-heavy sites and punishes modern web design
#10Great idea, awful UI
How so? It's intuitive and super fast. I whish there were more websites with such a simple UI.