Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A search engine that favors text-heavy sites and punishes modern web design

search.marginalia.nu

Post reply on HN
Print view View original

671–680 of 735 posts

Re: A search engine that favors text-heavy sites and punishes modern web design

#671

Post by marginalia_nu »

Siira wrote: post #598
snakeboy wrote: post #114

Wow, that's awesome. Great work! For a simple test, I searched "fall of the roman empire". In your search engine, I got wikipedia, followed by academic talks, chapters of books, and long-form blogs. All extremely useful resources. When I search on google, I get wikipedia, followed by a listicle "8 Reasons Why Rome Fell", then the imdb page for a movie by the same name, and then two Amazon book links, which are totall…

I tried some queries for Harry Potter fanfictions, and the results were pretty much completely unrelated. There weren’t that many results, either.

I'm curious what you searched for.

https://search.marginalia.nu/search?query=harry+potter+fanfi...

This seems to return a pretty decent number of sites relating to that (as well as some sites not relating to that).

The search engine isn't always great at knowing what a page is about, unfortunately.

This seemed to return mostly relevant results

https://search.marginalia.nu/search?query=%22harry+potter%22...

Re: A search engine that favors text-heavy sites and punishes modern web design

#672

Post by marginalia_nu »

drpixie wrote: post #645
marginalia_nu wrote: post #3

Yeah so this is my project. It's very much a work in progress, but occasionally I think it works remarkably well for something I cobbled together alone out of consumer hardware and home-made code :-)

Really like it, and love that you've done this yourself. I'd prefer if it does just one thing and does that really well. Don't waste your time on calculator and conversion functions, or pseudo-natural language queries. There are plenty of good calculator & converter tools and websites, but we all need a really good search engine. I think you'd be better looking at handling synonyms and plurals. Thanks.

I've mostly added these converters and frills because I'm trying to eat my own dogfood, and google is deeply engrained as my go-to source for unit conversions and back-of-envelope calculations.

Don't worry, this stuff is easy, and doesn't even remotely take away from the work on the harder problems.

Re: A search engine that favors text-heavy sites and punishes modern web design

#673

Post by marginalia_nu »

aoms wrote: post #652
marginalia_nu wrote: post #3

Yeah so this is my project. It's very much a work in progress, but occasionally I think it works remarkably well for something I cobbled together alone out of consumer hardware and home-made code :-)

What's the tech stack?

It's custom java code, for the most part. I'm using mariadb for some ancillary information, but the index and the crawler and everything is written from scratch.

It's hosted on my consumer-equipment server (Ryzen 3900x, 128Gb ram, Optane 900p+a few IronWolf drives), bare bones on Debian.

Re: A search engine that favors text-heavy sites and punishes modern web design

#674

Post by marginalia_nu »

arduinomancer wrote: post #413

Wow this is immediately useful If you figure out some sort of funding model (maybe even just Patreon) I could totally see this as a viable side project Already discovered this recipe site: https://based.cooking/ I love how adding recipes is through pull requests: https://github.com/LukeSmithxyz/based.cooking/pulls

I've added a patreon, and a few people have hopped onboard. I'm honestly a bit stunned at the reception this has gotten.

Re: A search engine that favors text-heavy sites and punishes modern web design

#675

Post by marginalia_nu »

sureklix wrote: post #668

So is this a filter on top of Google or is it search from scratch? Would love to understand more of the implementation.

It's built from scratch. I'm doing the crawling and indexing. Look through my comments and you'll find a few outlines of the stack and the index design.

Here are my blog entries relating to this:

https://memex.marginalia.nu/topic/astrolabe.gmi

Re: A search engine that favors text-heavy sites and punishes modern web design

#676

Post by marginalia_nu »

macksd wrote: post #450

This is a really cool idea. I tried a few technical queries I did on DDG today and didn't get amazing results - hence the warning in the About page about this engine giving you things you didn't know you were looking for, rather than specific facts. But the examples others have posted sound promising and refreshing. I would love to read about the algorithms behind this and how modern web design gets detected in order…

Yeah I think you typically have the most interesting results searching for older topics.

I've posted a few comments outlining the tech, check my profile. It's also outlined some details in my blog if you dig around a bit, there's no real comprehensive overview, just bits and pieces though.

https://memex.marginalia.nu/topic/astrolabe.gmi

Re: A search engine that favors text-heavy sites and punishes modern web design

#677

Post by marginalia_nu »

marcosdumay wrote: post #86

I've got some results where the same site is has than 70% of the links. It was a very on topic and high quality site, but still, all the results shouldn't point to the same place. I think some grouping by site (and capping to only the few most relevant links there) would improve the engine.

It shouldn't give you more than three per domain on paper, but I've got issues with deduplication so sometimes you get six :-(

Still a lot of work needs to be done.

Re: A search engine that favors text-heavy sites and punishes modern web design

#678

Post by redwall_hp »

foofoo4u wrote: post #218
snakeboy wrote: post #114

Wow, that's awesome. Great work! For a simple test, I searched "fall of the roman empire". In your search engine, I got wikipedia, followed by academic talks, chapters of books, and long-form blogs. All extremely useful resources. When I search on google, I get wikipedia, followed by a listicle "8 Reasons Why Rome Fell", then the imdb page for a movie by the same name, and then two Amazon book links, which are totall…

Good comparison. Reminds me of an analogy I like to make of today's web, which is it feels like browsing through a magazine store — full of top 10s, shallow wow-factoids, and baity material. I genuinely believe terrible results like this are making society dumber.

So did Tim Berners-Lee. He was vehemently opposed to people shoehorning images into the WWW, because he didn't want it to turn into the equivalent of magazines. Which, I believe, he shared the opinion of them making society dumber.

Appropriately enough, I couldn't find a good quote to verify that since Google is only giving me newspapers and magazines talking about Sir Tim in the context of current events. I do believe it's in his book "Weaving the Web" though.

Re: A search engine that favors text-heavy sites and punishes modern web design

#679

Post by marginalia_nu »

amelius wrote: post #626

Question: how do we benchmark search engines? Are there any groups attempting to provide (open) solutions in this space? (It seems to me that if you want to build a good search engine, this is the question you need to address first.)

The benchmark clearly must depend on the purpose of the search engine, though. If the purpose is to find a thing you know is out there, but haven't been arsed to bookmark, then my search engine is hot garbage.

If the purpose is to provide links you will find interesting when you click on them, my search engine really actually has its moments. I guess it's almost closer to old reddit, before it was overtaken by astroturfing and bots.

Re: A search engine that favors text-heavy sites and punishes modern web design

#680

Post by marginalia_nu »

palijer wrote: post #259

This has been needed in my life for a while. I am growing really apathetic about the internet lately, but I realize that is because my entry point is always a google search. I miss finding blog posts and scholarly articles in long form. I hate the SEO sites with unreadable UI because the information in them is often a lot lower quality as well.

I think the small web and gemini protocol might be up your alley too, if you feel this way about the Internet.
Post reply on HN