Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A search engine that favors text-heavy sites and punishes modern web design

search.marginalia.nu

Post reply on HN
Print view View original

501–510 of 735 posts

Re: A search engine that favors text-heavy sites and punishes modern web design

#501

Post by knuthsat »

ThalesX wrote: post #314
foxfluff wrote: post #283

Earlier quoted context omitted.

Wait till you learn that modern CPUs run billions of cycles per second. With multiple cores in parallel! And they can reach transfer rates of tens of gigabytes per second to RAM, or around a terabyte per second into L3.

And then you add a single HTTP request and everything tones down to the speed of the web. Or I/O. Or DB call.

You can still support millions of these requests per second if you just bake all of the dependencies directly in a small OS running on your fastest raspberry pi.

Re: A search engine that favors text-heavy sites and punishes modern web design

#502

Post by bencollier49 »

This is brilliant - I can actually surf the web for fun again. This engine's actually a nice complement to another mainstream engine, as the regular one is good for searches during the working day, whilst "marginalia" (?) is great for recreational reading, and actual learning.

Re: A search engine that favors text-heavy sites and punishes modern web design

#503

Post by raflemakt »

I tried two searches in Norwegian ("norsk ordbok" [norwegian dictionary] and "stortinget" [the parliament]), and they both returned many extreme or "alternative" websites. It was especially striking that the neo-nazi group Vigrid's website was the top hit for both searches. Maybe these sites just have less modern web design?

Re: A search engine that favors text-heavy sites and punishes modern web design

#504

Post by rasz »

snakeboy wrote: post #114

Wow, that's awesome. Great work! For a simple test, I searched "fall of the roman empire". In your search engine, I got wikipedia, followed by academic talks, chapters of books, and long-form blogs. All extremely useful resources. When I search on google, I get wikipedia, followed by a listicle "8 Reasons Why Rome Fell", then the imdb page for a movie by the same name, and then two Amazon book links, which are totall…

>followed by a listicle "8 Reasons Why Rome Fell"

but arent you curious about the 7th reason? it will surprise you!

Re: A search engine that favors text-heavy sites and punishes modern web design

#505

Post by marginalia_nu »

Aeolun wrote: post #448
marginalia_nu wrote: post #35

Earlier quoted context omitted.

I wrote it myself from scratch. I have some metadata in mariadb, but the index is bespoke. A design sketch of the index is that it uses one file with sorted URL IDs, one with IDs of N-grams (i.e. words and word-pairs) referring to ranges in the URL file; as well as a dictionary for relating words to word-IDs; that's a GNU Trove hash map I modified to use memory map data instead of direct allocated arrays. So when you…

I’m not sure how you go from word to url range? Range implies contiguous, but how can you make that happen for a bunch of words without keeping track of a list of urls for each word (or URL ids, the idea is the same)?

The trick is that the list of URLs for each word already is in the URLs file.

The URLs in a range are sorted. A sorted list (or list-range) forms an implicit set-like data structure, where you can do binary searches to test for existence.

Consider a words file with two words, "hello" and "world", corresponding to the ranges (0,3), (3,6). The URLs file contains URLs 1, 5, 7, 2, 5, 8.

The first range corresponds to the URLs 1, 5, 7; and the second 2, 5, 8.

If you search for hello world, it will first pick a range, the range for "hello", let's say (1,5,7); and then do binary searches in the second range -- the range corresponding to "world" -- (2,5,8) to find the overlap.

This seems like it would be very slow, but since you can trivially find the size of the ranges, it's possible to always do them in an order of increasing range-sizes. 10 x log(100000) is a lot smaller than 100000 x log(10)

Re: A search engine that favors text-heavy sites and punishes modern web design

#507

Post by FractalHQ »

lukas099 wrote: post #335
FractalHQ wrote: post #107

Ok this is great if all I want to do is read text, but often times that is very much not all I want to do. The web is much more than text and images these days. I can appreciate this as long as it’s branded as a search engine for blogs and articles specifically, as opposed to being touted as a drop-in replacement for the modern search engine.

Is this a criticism? It doesn't at all seem touted as a drop-in replacement for the modern search engine.

I do think this is a great tool, I didn’t intend to come off as being critical. I think I had some misdirected frustration from people on HN talking about how the modern web is bad and should be replaced with a text protocol. I aspire to create web apps that push the boundaries of what is possible today, and feel disappointed whenever I encounter people advocating for regression.

But I digress. In hindsight, my comment was obnoxious and under-appreciative of the tool being shared, and my rant was only tangentially related.

Re: A search engine that favors text-heavy sites and punishes modern web design

#508

Post by artificial »

fullsend wrote: post #376
SPBS wrote: post #274

Earlier quoted context omitted.

Cool, it appears that the trend towards JS may be causing self-selection -- if a page has a high amount of JS, it is highly unlikely to contain anything of value.

True. Unfortunately many large corporate websites through which you pay bills, order tickets, etc. are becoming infested with JS widgets and bulky, slow interfaces. These are hard to avoid.

Conversely no software to install. Browser as a platform. Don’t have to boot to Windows to pay your bills with activex for example

Re: A search engine that favors text-heavy sites and punishes modern web design

#509

Post by marginalia_nu »

raflemakt wrote: post #503

I tried two searches in Norwegian ("norsk ordbok" [norwegian dictionary] and "stortinget" [the parliament]), and they both returned many extreme or "alternative" websites. It was especially striking that the neo-nazi group Vigrid's website was the top hit for both searches. Maybe these sites just have less modern web design?

Yeah this is actually a bit of a concern of mine.

As very much a friend of Voltaire's, I don't think it's my place to police people's opinions no matter how disagreeable, but I also don't want my search engine to become branded as the search engine of choice for nazis because it's decent at cataloguing extremist sites.

Re: A search engine that favors text-heavy sites and punishes modern web design

#510

Post by Shingbogle »

marginalia_nu wrote: post #329
typon wrote: post #301

Earlier quoted context omitted.

The Google results tell you why Ivermectin is not a good replacement for vaccination against Covid, the Marginalia results tell you that Ivermectin is a miracle drug for treating Covid 19. Really shows how much technology has the power to change reality in today's world.

Part of what I wanted to show with this project is that there is no such thing as an objective search engine. Even seemingly irrelevant technological decisions drastically impact the narrative.

It's definitely not irrelevant. My first search time covid related because I knew the non-official, random person on the internet blog wouldn't have the money to create flashy sites.
Post reply on HN