Live data from Hacker News

A search engine that favors text-heavy sites and punishes modern web design

search.marginalia.nu

421–430 of 735 posts

Re: A search engine that favors text-heavy sites and punishes modern web design

#421

Earlier quoted context omitted.

Likewise, I co-maintain the only "fan" site on one of my all-time favourite composers/performers, and gave the engine a shot with a unique string query. While my text-heavy WP-driven site didn't seem to make the cut, the results were highly relevant in that they were links to former band members and collaborators - a couple of which I didn't realize existed. That being said, there were a few sites (including my own)…

The search engine doesn't actually do full text search, so maybe your query was too... unique. But do first of all verify that you haven't been hacked. There's about quarter of a million domains I've flagged that, besides their wordpress content, also host a ton of link spam crap off in some hidden folder. This reflects on the quality rating extremely negatively to the point where you may have not been indexed at all…

Thanks for the advice; not hacked, but I have "resurrected" many WP sites that have been (including my wife's non-profit). Just running on an EC2 micro instance, but I tried adding "site:" and received "No such domain". Actually, I think it's because I haven't enabled "HTTPS" yet! That's on my to-do along with migrating off EC2-Classic to VPC...

Re: A search engine that favors text-heavy sites and punishes modern web design

#423
post #409

Earlier quoted context omitted.

Again it's a one man hobby project, I don't have a team of people to go through every formulation and every error message to ensure nobody can read them in a way that offends them. It's just me, writing code on an unfinished project that HN discovered. In this case, the code doesn't match the word regexp, like it may be a @TwitterHandle or a "comp.lang.c" with periods in it, or an unsupported Unicode range. It doesn'…

I must congratulate you on this achievement. That's certainly a useful take on search. Nonetheless, even when coding, one should also consider thoroughly the UX and how it would be addressing the others. Saying "unsupported word" is much more sympathetic than "needs to be a word" (where you define what a "word" is, and the general user is unaware of such definition).

Fair point, I refined the phrasing a bit.

> The term "𓀀" contains characters that are not currently supported

Re: A search engine that favors text-heavy sites and punishes modern web design

#425

Wow, that's awesome. Great work! For a simple test, I searched "fall of the roman empire". In your search engine, I got wikipedia, followed by academic talks, chapters of books, and long-form blogs. All extremely useful resources. When I search on google, I get wikipedia, followed by a listicle "8 Reasons Why Rome Fell", then the imdb page for a movie by the same name, and then two Amazon book links, which are totall…

Imagine if you were looking for the movie.

!imdb

Re: A search engine that favors text-heavy sites and punishes modern web design

#426
post #350

Earlier quoted context omitted.

This got me thinking that maybe one of the other big reasons for this is that the algorithms prioritize newer pages over older pages. This produces the problem where instead of covering a topic and refining it over time, the incentive is to repackage it over and over again. It reminds me of an annoyance I have with the Kindle store. If I wanted to find a book on, let's say, Psychology, there is no option to find all-…

> is that the algorithms prioritize newer pages over older pages. They do? That would explain a lot - but ironically, I can't find a good source on this. Do you have one at hand?

Your Google search results show the date on articles do they not? If people are more likely to click on "Celebrity Net Worth (2021)" than "Celebrity Net Worth (2012)", then the algo will update to favour those results, because people are clicking on them.

The only definitive source on this would be the gatekeeper itself. But Google never says anything explicitly, because they don't want people gaming search rankings. Even though it happens anyway.

Re: A search engine that favors text-heavy sites and punishes modern web design

#427
I use webcrawler.com, and IMO it's better than any other search engine for finding exactly what I'm looking for. Not what's "trending", or "popular", or what the sheeple are searching for. It finds the exact matching keywords that I'm looking for. No inference or other bullshit -- just the matches.

Such a relief to not wade through oceans of worthless crap any more.

Re: A search engine that favors text-heavy sites and punishes modern web design

#428

Search for playwright waitForSelector and you land in pretty useless page. I'm all in for text websites, but something like playwright.dev documentation is top notch - fuzzy search being key thing.

Yeah I wasn't really planning for this to blow up like it did today. It's currently sitting at about 35% of the index size I usually aim for, so besides the stuff I can't index because it's behind CDNs, there's a lot of pages it just hasn't gotten to yet. playwright.dev is pretty low on the priority list because it has a metric crap-ton of javascript on its front page. The crawler has visited it, looked at it, and put it very far down the priority queue.

Re: A search engine that favors text-heavy sites and punishes modern web design

#429

Yeah so this is my project. It's very much a work in progress, but occasionally I think it works remarkably well for something I cobbled together alone out of consumer hardware and home-made code :-)

I searched Warcraft and got a gold selling/ level boosting site. Some things never change :)

Re: A search engine that favors text-heavy sites and punishes modern web design

#430
post #352

Earlier quoted context omitted.

Part of what I wanted to show with this project is that there is no such thing as an objective search engine. Even seemingly irrelevant technological decisions drastically impact the narrative.

Well the focus on text content isn't the only technical difference here. Google is obviously weighing hundreds of signals in its search results that your engine is not accounting for. These omitted signals are also relevant.

> These omitted signals are also relevant.

Certainly. And sometimes they're relevant in a good way, sometimes in a bad user-hostile way. Every search engine rquires discrimination and intelligent usage by the person doing the search, just in different areas.

Post reply on HN