Live data from Hacker News

A search engine that favors text-heavy sites and punishes modern web design

search.marginalia.nu

391–400 of 735 posts

Re: A search engine that favors text-heavy sites and punishes modern web design

#391

Wow, that's awesome. Great work! For a simple test, I searched "fall of the roman empire". In your search engine, I got wikipedia, followed by academic talks, chapters of books, and long-form blogs. All extremely useful resources. When I search on google, I get wikipedia, followed by a listicle "8 Reasons Why Rome Fell", then the imdb page for a movie by the same name, and then two Amazon book links, which are totall…

Good comparison. Reminds me of an analogy I like to make of today's web, which is it feels like browsing through a magazine store — full of top 10s, shallow wow-factoids, and baity material. I genuinely believe terrible results like this are making society dumber.

what I really want is a true AI to search through all that and figure out the useful truth. I don't know how to do this (and of course whoever writes the AI needs to be unbiased...)

Re: A search engine that favors text-heavy sites and punishes modern web design

#392

Absolute textgasm. Wonder how 'text-only first' prioritization is being implemented, algorithmically speaking?

This post - https://news.ycombinator.com/item?id=28551183 - suggests it's a simple set of hueristics, looking for things like javascript, link/SEO spam, language, amount of text content, etc, filtering out unwanted results and only indexing wanted ones.

Thank you!

Re: A search engine that favors text-heavy sites and punishes modern web design

#393

Yeah so this is my project. It's very much a work in progress, but occasionally I think it works remarkably well for something I cobbled together alone out of consumer hardware and home-made code :-)

This is absolutely wonderful. I am LOVING the results I'm getting back from it: the sort of content-rich sites that have become nigh unreachable using traditional search engines. Thank you for building this!

Re: A search engine that favors text-heavy sites and punishes modern web design

#395
post #276
post #275

This is a search engine indexing the internet on a mariadb database hosted on consumer hardware maintained by a single person as a hobby and it does not suffer from HN hug of death

how on earth do you index so much on consumer hardware? my frontend developer mind is blown.

Consumer hardware today is simply what was cutting-edge and crazy expensive 5 years ago.

Re: A search engine that favors text-heavy sites and punishes modern web design

#396
Quoted from the linked site:

> Convenience functions have been added, and the search engine can now perform simple calculations and unit conversions. Try 1 pint in cubic centimeters, or 50+sqrt(pi). This functionality is still under development, be patient if it doesn't work.

Why would you make any ever so small effort to implement calculations? I don't get it.

If your search engine enabled me to find more useful search results to my queries than google or yacy or whatever, I wouldn't care one tiny bit about being able to do calculations with it.

Why not focus on the search functionality?

Re: A search engine that favors text-heavy sites and punishes modern web design

#397
post #378

effort is good, but needs some work, no results here : https://search.marginalia.nu/search?query=rxjava+2+api+docs https://www.google.com/search?q=rxjava+2+api+docs&oq=rxjava+...

It is very much a work in progress, still struggling with some areas. I only really got into the territory of "sometimes actually useful" like this weekend. Wasn't planning on blowing up on HN just yet.

Re: A search engine that favors text-heavy sites and punishes modern web design

#398
post #371

Too bad it rejects non-Latin words, as if the definition of "text" is a sequence of alphabetical letters originated from Latin. I thought that we've reached the time to embrace all cultures in the world, but this retrogressive engine proves that most modern tech designers are myopic about other civilizations in the globe.

Understand that this is something I built for myself, by myself, so it focuses on languages I understand. It hosted on a single consumer grade computer in my living room. I built it out of pocket and anyone is free to use it. Does this make me a villain? If I can do this, what's preventing some guy in Japan or India or Peru from doing the same, of course focusing on their languages?

Maybe a better suited choice for errors than an insulting message: when I provided a query in my native language it regurgitated the error "needs to be a word" instead of more acceptable "not a supported language".

When you claim that a word in some other culture is not "a word", just because it's not recognized by your machine, that's demeaning to say the least.

Re: A search engine that favors text-heavy sites and punishes modern web design

#399

Yeah so this is my project. It's very much a work in progress, but occasionally I think it works remarkably well for something I cobbled together alone out of consumer hardware and home-made code :-)

I love this idea, and admire the work you put into it. I'm a fan of long reads and historical non-fiction, and Google's results are truly garbage.

I have a criticism that I think may pertain to the ranking methodology. I searched for "discovery of Australia". Among the top results were:

* A site claiming that the biblical flood was caused by Earth colliding with a comet (with several other pages from that site also making the top search results with other wild claims, e.g. that the Egyptians discovered Arizona);

* Another site claiming the first inhabitants of Australia were a lost tribe of Israel;

* A third site claiming that Australia was discovered and founded by members of a secret society of Rosicrucians who had infiltrated the Dutch East India Company and planned to build an Australian utopia...

These were all pages heavy with HTML4 tags and virtually devoid of Javascript, the kinds of pages you'd frequently see in the late 1990s from people who had built their own static websites in a text editor, or exported HTML from MS Word. At that time, there were millions of those sites with people paying for their own unique domain names, and so the proportion of them that were home to wild-eyed conspiracy theories was relatively small. What I think has happened is that kooks continued to keep these sites up - to the point where it's almost a visual trope now to see a red tag in Times New Roman and think, uh oh, I've stumbled on an "ancient aliens" site. Whereas scholars and journals offering higher quality information have moved to more modern platforms that rely more heavily on modern browsers - with or without their own domain names. So as a result what seemed to surface here were the fragments of the old web that remain live - possibly because people living in cabins in Montana forget to cancel their web hosting, or because the nature of old-school conspiracy theorists is to just keep packing their old sites with walls of text surrounded by

tags.

Arguably, this seems to rank the way Google's engine used to, since it couldn't run JS and they wanted to punish sites that used code to change markup at render time. At least, when I used to have to do onsite SEO work, it was always about simple tag hierarchies.

I wonder whether there isn't some better metric of validity and information quality than what markup is used. Some of the sites that surfaced further down could be considered interesting and valuable resources. I think not punishing simple wall-of-text content is a good thing. But to punish more complicated layouts may have the perverse effect of downranking higher-quality sources of information - i.e. people and organizations who can afford to build a decent website, or who care to migrate to a modern blogging platform.

Re: A search engine that favors text-heavy sites and punishes modern web design

#400
Wow. I tested it on recipes which Google has destroyed and this was the first result, a simple clear recipe:

http://demont.myds.me/leerecipes/mainmeals/mainmeals1/chicke...

compared to Google's endless drivel of "This chicken stir fry recipe will become a staple in your home. It’s so quick to make and you can use whatever vegetables you have on hand. It tastes wonderful regardless of how you alter the ingredients. ... " JUST SHUT UP AND GIVE ME THE RECIPE!

https://natashaskitchen.com/chicken-stir-fry-recipe/

Post reply on HN