Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A search engine that favors text-heavy sites and punishes modern web design

search.marginalia.nu

Post reply on HN
Print view View original

551–560 of 735 posts

Re: A search engine that favors text-heavy sites and punishes modern web design

#551

Post by wizzwizz4 »

Add an OpenSearch file: https://developer.mozilla.org/en-US/docs/Web/OpenSearch

Maybe something like:

  
    Marginalia
    Marginalia Search - a search engine that favors text-heavy sites and punishes modern web design
    UTF-8
    https://search.marginalia.nu/favicon.ico
    
    https://search.marginalia.nu/
and then add:
to your page's .

Re: A search engine that favors text-heavy sites and punishes modern web design

#552

Post by marginalia_nu »

soheil wrote: post #481
marginalia_nu wrote: post #85

Earlier quoted context omitted.

It's pretty much all bespoke. I use external libraries for parsing HTML (JSoup) and robots.txt; but that's about it.

What was the starting site you fed to the crawler to follow the links from to build the index?

Just my (swedish) personal website. The first iteration of the search engine was probably mainly seeded by these links:

https://www.marginalia.nu/00-l%C3%A4nkar/

But I've since expanded my websites, so now I think these play a decent role in later iterations, although they are virtually all of them pages I've found eating my own dogfood:

https://memex.marginalia.nu/links/fragments-old-web.gmi

https://memex.marginalia.nu/links/bookmarks.gmi

Re: A search engine that favors text-heavy sites and punishes modern web design

#553

Post by coldtea »

rasz wrote: post #504
snakeboy wrote: post #114

Wow, that's awesome. Great work! For a simple test, I searched "fall of the roman empire". In your search engine, I got wikipedia, followed by academic talks, chapters of books, and long-form blogs. All extremely useful resources. When I search on google, I get wikipedia, followed by a listicle "8 Reasons Why Rome Fell", then the imdb page for a movie by the same name, and then two Amazon book links, which are totall…

>followed by a listicle "8 Reasons Why Rome Fell" but arent you curious about the 7th reason? it will surprise you!

You wont believe how Claudius looks today!

Re: A search engine that favors text-heavy sites and punishes modern web design

#556

Post by marginalia_nu »

kews wrote: post #236
marginalia_nu wrote: post #8

Earlier quoted context omitted.

I do indeed index the web myself. Not the entire web, just a subset of it. The crawler quickly loses interest in javascript:y websites and only indexes at depth those websites that are simple. It also focuses on websites in English, Swedish and Latin and tries to identify and ignore the rest (best-effort). You'd be surprised how much you can do with modern hardware if you are scrappy. The current index is about 17.7…

Do you know what proportion of the texty web instructs unknown crawlers to go away (or blocks them)?

It's hard to give numbers, it doesn't seem to be very many, but losing out on a few key sites does make a pretty big impact.

You see stuff like this sometimes, makes me a bit sad.

https://linux.die.net/robots.txt

Re: A search engine that favors text-heavy sites and punishes modern web design

#557

Post by NuNotNon »

I have an interest in logic and cs curriculum and i like Geneses in general(last days i've read intro in math phylosophy from Russell and some acm report of cs curriculum. I search for cs curriculum and this is the first link https://www.cs.rice.edu/~vardi/sigcse/ Feels so good to recive good answers so easy. Thanks.

Re: A search engine that favors text-heavy sites and punishes modern web design

#558

Post by AQXt »

I searched for "giraffe evolution" (without quotes) and received the following links on the first page:

- Evolutionist scientists say the theory is unscientific and worthless

- Seven Mysteries of Evolution

- OTHER EVIDENCE AGAINST EVOLUTION

- Evolution Falsified

Not a single result about the evolution of giraffes...

Re: A search engine that favors text-heavy sites and punishes modern web design

#559

Post by marginalia_nu »

hdjjhhvvhga wrote: post #132
snakeboy wrote: post #114

Wow, that's awesome. Great work! For a simple test, I searched "fall of the roman empire". In your search engine, I got wikipedia, followed by academic talks, chapters of books, and long-form blogs. All extremely useful resources. When I search on google, I get wikipedia, followed by a listicle "8 Reasons Why Rome Fell", then the imdb page for a movie by the same name, and then two Amazon book links, which are totall…

As long as few people use it, it will be great. Rest assured that the moment it becomes popular, the people who want to game it will appear.

Specialization mostly a problem in monocultures.

If you almost only plant wheat, you are going to end up with one hell of a pest problem.

If you almost only have Windows XP, you are going to have one hell of a virus problem.

If you almost only have SearchRank-style search engines (or just the one), you are going to have one hell of a content spam problem.

Even though they have some pretty dodgy incentives, I don't think google suffers quality problems because they are evil, I think ultimately they suffer because they're so dominant. Whatever they do, the spammers adapt almost instantly.

A diverse ecosystem on the other hand limits the viability of specialization by its very nature. If one actor is attacked, it shrinks and that reduces the opportunity for attacking it.

Re: A search engine that favors text-heavy sites and punishes modern web design

#560

Post by appel »

zachguo wrote: post #458
mumblemumble wrote: post #41

I like it. Coincidentally, the other day I was daydreaming about a search engine that favors sites that are updated less frequently. The thought being, the kinds of labors of love that characterized the 1990s Web that I still sometimes miss are still out there, it's just harder to find them amidst the flood of SEO dreck. So perhaps they could be made discoverable again with the help of a contrarian search engine that…

Try this https://wiby.me/ Clicking 'Surprise me' gave me an interesting article from 1994 http://milk.com/wall-o-shame/bucket.html

That was a great read, thanks for sharing.
Post reply on HN