Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A search engine that favors text-heavy sites and punishes modern web design

search.marginalia.nu

Post reply on HN
Print view View original

681–690 of 735 posts

Re: A search engine that favors text-heavy sites and punishes modern web design

#681

Post by marginalia_nu »

infogulch wrote: post #669

Not to overemphasize meta commentary, but damn 3200 points, 650 comments in 2 days -- this is one of the highest rated posts I can remember. Seems HN readers are very interested in alternatives to the current search hegemony and the kind of low-quality junk articles that litter it.

I'm still a bit stunned at the reception this has gotten.

I wasn't even planning to launch this, not like this, there's so much that needs to be fixed to get it working good, so much jank and so many weird limitatons. It's quietly been online for a few months. But it's not really worked all that well except in rare cases. Then I fixed a few issues and implemented some improvements, and it was really just last weekend that I was struck by the sense that it actually was coming together into something really viable, and then... this happened.

I've gotten so many positive comments about this project, a large number of emails and I haven't had the time to get back to half the people who wrote, even people donating money to support it.

I half thought I becoming the next TempleOS-guy, hacking away at some madcap scheme all by myself. I just had no idea this resonated with so many people.

It's incredibly encouraging and motivating. Thanks everyone!

Re: A search engine that favors text-heavy sites and punishes modern web design

#683

Post by niutech »

marginalia_nu wrote: post #3

Yeah so this is my project. It's very much a work in progress, but occasionally I think it works remarkably well for something I cobbled together alone out of consumer hardware and home-made code :-)

Nice project, but have you heard of FrogFind? it also presents lightweight search results.

Re: A search engine that favors text-heavy sites and punishes modern web design

#684

Post by freediver »

marginalia_nu wrote: post #196
q3k wrote: post #148

Earlier quoted context omitted.

Good stuff. I've also been toying with doing some homegrown search engine indexing (as an exercise in scalable systems), and this is a fantastic result and great inspiration. Definitely want to see more people doing that kind of low-level work instead of falling back to either 'use elasticsearch' or 'you can't, you're not google'.

Well just crunching the numbers should indicate what is possible and what isn't. For the moment I have just south of 20 million URLs indexed. 1 x 20 million bytes = 20 Mb. 10 x 20 million bytes = 200 Mb. 100 x 20 million bytes = 2 Gb. 1,000 x 20 million bytes = 20 Gb. 10,000 x 20 million bytes = 200 Gb. 100,000 x 20 million bytes = 2 Tb. 1,000,000 x 20 million bytes = 20 Tb. This is still within what consumer hardwar…

What crawler are you using and what kind of crawling speeds are you achieving?

How do you rank the results (is it based on content only) or you have external factors too?

What is your personal preferred search option of the 7 and why?

Thanks for making something unique and sorry that despite all the hype this got, you got only $39/month on Patreon. It is telling in a way.

Re: A search engine that favors text-heavy sites and punishes modern web design

#685

Post by 1vuio0pswjnm7 »

snakeboy wrote: post #114

Wow, that's awesome. Great work! For a simple test, I searched "fall of the roman empire". In your search engine, I got wikipedia, followed by academic talks, chapters of books, and long-form blogs. All extremely useful resources. When I search on google, I get wikipedia, followed by a listicle "8 Reasons Why Rome Fell", then the imdb page for a movie by the same name, and then two Amazon book links, which are totall…

No one mentioned the "bonus" audio in the page source: https://www.youtube.com/watch?v=7fCifJR6LAY

Re: A search engine that favors text-heavy sites and punishes modern web design

#686

Post by xhrpost »

> This search engine isn't particularly well equipped to answering queries posed like questions, instead try to imagine some text that might appear in the website you are looking for, and search for that.

Heh, I guess I'm getting old but I remember when this was the only way to search the web

Re: A search engine that favors text-heavy sites and punishes modern web design

#687

Post by mlang23 »

This is absolutely great. Seeing this for the first time somehow reminds me when I learnt about Google. many years ago. The salespitch for google back then was "They have all the linux related docs and infos indexed". The promise of this engine seems even more promising. A search engine specialized in text, look 'ma! I hope this grows and gets the user attention it deserves. Google has becomes so ad-infested in the last 3 years, its time something replaces it.

Re: A search engine that favors text-heavy sites and punishes modern web design

#688

Post by crucialfelix »

anonuser123456 wrote: post #595
foofoo4u wrote: post #218

Earlier quoted context omitted.

Good comparison. Reminds me of an analogy I like to make of today's web, which is it feels like browsing through a magazine store — full of top 10s, shallow wow-factoids, and baity material. I genuinely believe terrible results like this are making society dumber.

> I genuinely believe terrible results like this are making society dumber. You have to e causality reversed. Google results reflect the fact that society is dumb.

Or the distribution of people now online better reflects stupidity in the general population.

Re: A search engine that favors text-heavy sites and punishes modern web design

#689

Post by bullen »

marginalia_nu wrote: post #35
santamex wrote: post #14

Earlier quoted context omitted.

Which software do you use to index the sites?

I wrote it myself from scratch. I have some metadata in mariadb, but the index is bespoke. A design sketch of the index is that it uses one file with sorted URL IDs, one with IDs of N-grams (i.e. words and word-pairs) referring to ranges in the URL file; as well as a dictionary for relating words to word-IDs; that's a GNU Trove hash map I modified to use memory map data instead of direct allocated arrays. So when you…

Interesting, in my database (http://root.rupy.se) I have one file per word that contains the ids (long) of the nodes (URLs), so to search many words together I have to go through the first file and one by one see if I find matches in the second.

How does the range binary search work, does it just prune out the overlaps, how efficient is it and how much data do you have in there for say "hello" and "world" f.ex?

Re: A search engine that favors text-heavy sites and punishes modern web design

#690

Post by bullen »

marginalia_nu wrote: post #505
Aeolun wrote: post #448

Earlier quoted context omitted.

I’m not sure how you go from word to url range? Range implies contiguous, but how can you make that happen for a bunch of words without keeping track of a list of urls for each word (or URL ids, the idea is the same)?

The trick is that the list of URLs for each word already is in the URLs file. The URLs in a range are sorted. A sorted list (or list-range) forms an implicit set-like data structure, where you can do binary searches to test for existence. Consider a words file with two words, "hello" and "world", corresponding to the ranges (0,3), (3,6). The URLs file contains URLs 1, 5, 7, 2, 5, 8. The first range corresponds to the…

Hm, ok I understand more but how do you perform the "binary search", just loop over the URL ids?

Funny I also selected "hello" and "world" above! Xo

My system is also written in Java btw!

Here are example results of my word search:

http://root.rupy.se/node/data/word/four

http://root.rupy.se/node/data/word/only

etc.

Post reply on HN