Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A search engine that favors text-heavy sites and punishes modern web design

search.marginalia.nu

Post reply on HN
Print view View original

21–30 of 735 posts

Re: A search engine that favors text-heavy sites and punishes modern web design

#21

Post by c0wb0yc0d3r »

marginalia_nu wrote: post #8
samsaga2 wrote: post #7

Where does the data come from? Do you index the whole web yourself? I see it totally impossible for a personal project. I'm very curious about that.

I do indeed index the web myself. Not the entire web, just a subset of it. The crawler quickly loses interest in javascript:y websites and only indexes at depth those websites that are simple. It also focuses on websites in English, Swedish and Latin and tries to identify and ignore the rest (best-effort). You'd be surprised how much you can do with modern hardware if you are scrappy. The current index is about 17.7…

How did you go about seeding your web crawler with URLs to crawl?

Re: A search engine that favors text-heavy sites and punishes modern web design

#22

Post by dannyw »

marginalia_nu wrote: post #8
samsaga2 wrote: post #7

Where does the data come from? Do you index the whole web yourself? I see it totally impossible for a personal project. I'm very curious about that.

I do indeed index the web myself. Not the entire web, just a subset of it. The crawler quickly loses interest in javascript:y websites and only indexes at depth those websites that are simple. It also focuses on websites in English, Swedish and Latin and tries to identify and ignore the rest (best-effort). You'd be surprised how much you can do with modern hardware if you are scrappy. The current index is about 17.7…

This is unbelievably impressive on a technical and ambition level for a solo, self-hosted hardware project. Kudos.

Re: A search engine that favors text-heavy sites and punishes modern web design

#23

Post by popcube »

jerrre wrote: post #13
gtmb wrote: post #6

As everything in life flows in cycle, I predict the search engine that will de-throne Google will be like Google when it started - a simple variation of page rank. No smarts, no bubble, no signals decided by over fitting to a biased engineer preference.

I wouldn't say the existence of this page proves your prediction right (as it's not dethroning Google anytime soon). It's easy to forget that the goal of Google isn't to provide a useful search engine (at least not anymore), but the search engine is a by product of them wanting to show ads.

now Google try immitate a document system on your computer, usually I rely on Google know what I need:(

Re: A search engine that favors text-heavy sites and punishes modern web design

#25

Post by dannyw »

c0wb0yc0d3r wrote: post #21
marginalia_nu wrote: post #8

Earlier quoted context omitted.

I do indeed index the web myself. Not the entire web, just a subset of it. The crawler quickly loses interest in javascript:y websites and only indexes at depth those websites that are simple. It also focuses on websites in English, Swedish and Latin and tries to identify and ignore the rest (best-effort). You'd be surprised how much you can do with modern hardware if you are scrappy. The current index is about 17.7…

How did you go about seeding your web crawler with URLs to crawl?

Not OP, but if I was to do this, I'd start by downloading Wikipedia and all its external links and references, and crawling from there. You should eventually reach most of the publicly visible internet.

Re: A search engine that favors text-heavy sites and punishes modern web design

#26

Post by antupis »

gtmb wrote: post #6

As everything in life flows in cycle, I predict the search engine that will de-throne Google will be like Google when it started - a simple variation of page rank. No smarts, no bubble, no signals decided by over fitting to a biased engineer preference.

As dev I would love search engine which would only do search to stackoverflow github issues, documentation etc.

Re: A search engine that favors text-heavy sites and punishes modern web design

#27

Post by axelroze »

gtmb wrote: post #6

As everything in life flows in cycle, I predict the search engine that will de-throne Google will be like Google when it started - a simple variation of page rank. No smarts, no bubble, no signals decided by over fitting to a biased engineer preference.

Wouldn't the dethroner of Google be some new technology which is not a search engine like Google but better at solving the original task of finding information on how to solve problems?

Just like how iPad dethroned Windows PCs for average home user but not Mac because Windows had the monopoly and then an innovation destroyed MS in this space and not a competitor.

I don't think Google dethrones Yahoo and AltaVista scenario will occur again.

Re: A search engine that favors text-heavy sites and punishes modern web design

#30

Post by kebos »

This is really cool, it filters out all fluff.

It's not always taking me to totally relevant sites but the results contain my favourite type of content.

Full of writing and pure html - usually the hallmark of someone who knows what they are doing, wants to communicate but doesn't want to waste their time.

Post reply on HN