Yes please! More of this!
A search engine that favors text-heavy sites and punishes modern web design
191–200 of 735 posts
Re: A search engine that favors text-heavy sites and punishes modern web design
#192Earlier quoted context omitted.
Hobby project leads angry person to interesting and unexpected material; angry person remains angry. Details at six.
The project explicitly bills itself as a "search engine", not an "interesting and unexpected material surfacer". Moreover, projecting emotions like "angry" onto a comment in order to discredit the content of the comment (hey! is that an ad-hominem?) is just about exactly the opposite of the discussions that the HN mods are trying to curate, and the discussions that I like to see here.
In the early days of google, I found what I was looking for on page 5+. On the way, I’d discover many interesting things I didn’t even know I was looking for, often completely unrelated to what I was searching for.
Re: A search engine that favors text-heavy sites and punishes modern web design
#193Wow, that's awesome. Great work! For a simple test, I searched "fall of the roman empire". In your search engine, I got wikipedia, followed by academic talks, chapters of books, and long-form blogs. All extremely useful resources. When I search on google, I get wikipedia, followed by a listicle "8 Reasons Why Rome Fell", then the imdb page for a movie by the same name, and then two Amazon book links, which are totall…
Imagine if you were looking for the movie.
Re: A search engine that favors text-heavy sites and punishes modern web design
#194Earlier quoted context omitted.
Likewise, I co-maintain the only "fan" site on one of my all-time favourite composers/performers, and gave the engine a shot with a unique string query. While my text-heavy WP-driven site didn't seem to make the cut, the results were highly relevant in that they were links to former band members and collaborators - a couple of which I didn't realize existed. That being said, there were a few sites (including my own)…
The search engine doesn't actually do full text search, so maybe your query was too... unique. But do first of all verify that you haven't been hacked. There's about quarter of a million domains I've flagged that, besides their wordpress content, also host a ton of link spam crap off in some hidden folder. This reflects on the quality rating extremely negatively to the point where you may have not been indexed at all…
It’d be nice if you had a page to get the current index status for a domain.
Re: A search engine that favors text-heavy sites and punishes modern web design
#195Wow, that's awesome. Great work! For a simple test, I searched "fall of the roman empire". In your search engine, I got wikipedia, followed by academic talks, chapters of books, and long-form blogs. All extremely useful resources. When I search on google, I get wikipedia, followed by a listicle "8 Reasons Why Rome Fell", then the imdb page for a movie by the same name, and then two Amazon book links, which are totall…
Imagine if you were looking for the movie.
The you'd use a different search engine. Why does everything have to be a Swiss Army knife?
Re: A search engine that favors text-heavy sites and punishes modern web design
#196Earlier quoted context omitted.
I wrote it myself from scratch. I have some metadata in mariadb, but the index is bespoke. A design sketch of the index is that it uses one file with sorted URL IDs, one with IDs of N-grams (i.e. words and word-pairs) referring to ranges in the URL file; as well as a dictionary for relating words to word-IDs; that's a GNU Trove hash map I modified to use memory map data instead of direct allocated arrays. So when you…
Good stuff. I've also been toying with doing some homegrown search engine indexing (as an exercise in scalable systems), and this is a fantastic result and great inspiration. Definitely want to see more people doing that kind of low-level work instead of falling back to either 'use elasticsearch' or 'you can't, you're not google'.
Well just crunching the numbers should indicate what is possible and what isn't.
For the moment I have just south of 20 million URLs indexed.
1 x 20 million bytes = 20 Mb.
10 x 20 million bytes = 200 Mb.
100 x 20 million bytes = 2 Gb.
1,000 x 20 million bytes = 20 Gb.
10,000 x 20 million bytes = 200 Gb.
100,000 x 20 million bytes = 2 Tb.
1,000,000 x 20 million bytes = 20 Tb.
This is still within what consumer hardware can deal with. It's getting expensive, but you don't need a datacenter to store 20 Tb worth of data.
How many bytes do you need, per document, for an index? Do you need 1 Mb of data to store index information about a page that, in terms of text alone, is perhaps 10 Kb?
Re: A search engine that favors text-heavy sites and punishes modern web design
#197Yeah so this is my project. It's very much a work in progress, but occasionally I think it works remarkably well for something I cobbled together alone out of consumer hardware and home-made code :-)
This is awesome. I've been looking for a long time for a search engine that basically takes everything Google does and does the opposite. Thank you for doing this, I will definitely be bookmarking it.
Is there a way to suggest or add sites? I went looking for woodgears.ca and only got one result. I also think my personal blog would be a good candidate for being indexed here but I couldn't find any results for it.
Re: A search engine that favors text-heavy sites and punishes modern web design
#198how about a search engine that bans all pinterest content.
I hate pinterest with a passion. I may need to get a "his" laptop separate from my wife, since she needs that darn pinterest extension for pinning photos.
Re: A search engine that favors text-heavy sites and punishes modern web design
#199Absolute textgasm.
Wonder how 'text-only first' prioritization is being implemented, algorithmically speaking?
Re: A search engine that favors text-heavy sites and punishes modern web design
#200Wow, that's awesome. Great work! For a simple test, I searched "fall of the roman empire". In your search engine, I got wikipedia, followed by academic talks, chapters of books, and long-form blogs. All extremely useful resources. When I search on google, I get wikipedia, followed by a listicle "8 Reasons Why Rome Fell", then the imdb page for a movie by the same name, and then two Amazon book links, which are totall…
Imagine if you were looking for the movie.
Imagine including the search term "movie".