Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A search engine that favors text-heavy sites and punishes modern web design

search.marginalia.nu

Post reply on HN
Print view View original

711–720 of 735 posts

Re: A search engine that favors text-heavy sites and punishes modern web design

#711

Post by marginalia_nu »

1vuio0pswjnm7 wrote: post #685
snakeboy wrote: post #114

Wow, that's awesome. Great work! For a simple test, I searched "fall of the roman empire". In your search engine, I got wikipedia, followed by academic talks, chapters of books, and long-form blogs. All extremely useful resources. When I search on google, I get wikipedia, followed by a listicle "8 Reasons Why Rome Fell", then the imdb page for a movie by the same name, and then two Amazon book links, which are totall…

No one mentioned the "bonus" audio in the page source: https://www.youtube.com/watch?v=7fCifJR6LAY

;-)

Re: A search engine that favors text-heavy sites and punishes modern web design

#712

Post by marginalia_nu »

ad404b8a372f2b9 wrote: post #44
marginalia_nu wrote: post #3

Yeah so this is my project. It's very much a work in progress, but occasionally I think it works remarkably well for something I cobbled together alone out of consumer hardware and home-made code :-)

Very cool project! How many websites do you have in your index? And how did you go about building it? I've been working on an engine for personal websites, currently trying to build a classifier to extract them from commoncrawl, if you have any general tips on that kind of project they'd be very welcome.

About 21 million, and I'm crawling myself.

Classification is really hard. I'm struggling with it myself, as a lot of like privacy policies and change logs turns out to share the shape of a page of text.

I'm thinking of experimenting with ML classifiers, as I do have reasonably good ways of extracting custom datasets. Finding change logs and privacy policies is easy, excluding them is hard.

Re: A search engine that favors text-heavy sites and punishes modern web design

#713

Post by Siira »

marginalia_nu wrote: post #671
Siira wrote: post #598

Earlier quoted context omitted.

I tried some queries for Harry Potter fanfictions, and the results were pretty much completely unrelated. There weren’t that many results, either.

I'm curious what you searched for. https://search.marginalia.nu/search?query=harry+potter+fanfi... This seems to return a pretty decent number of sites relating to that (as well as some sites not relating to that). The search engine isn't always great at knowing what a page is about, unfortunately. This seemed to return mostly relevant results https://search.marginalia.nu/search?query=%22harry+potter%22...

Yes, shorter queries return more relevant results. I think this was the first query that came to my mind:

https://search.marginalia.nu/search?query=Best+%22harry+pott...

Re: A search engine that favors text-heavy sites and punishes modern web design

#714

Post by eitland »

LeftHandPath wrote: post #620
quaintdev wrote: post #575

Earlier quoted context omitted.

I searched like this all my life and always got expected results. But just a week ago I found out that these "how", "what" questions give better and faster results on Google.

That switch happened some years ago. I've been unlearning and relearning how to use google for what feels like at least three or four years now. The main pain-point, though, is that a lot of long-tail searches you could've used to find different results in years past, now seem to funnel you to the same set of results based on your apparent intent. At least, it has felt that way -- I'm not entirely sure how the modern…

I realized this a few years ago when I observed my wife find things faster on Google than me.

I appreciate that it is easier for newcomers but I still hate it personally after years and especially that they cannot even avoid meddling with my queries even when I try to accept the new system and use the verbatim option.

Re: A search engine that favors text-heavy sites and punishes modern web design

#715

Post by eitland »

AQXt wrote: post #562
marginalia_nu wrote: post #437

Earlier quoted context omitted.

I think you need to approach it more like grep than google. It's a forgotten art, dealing with this type of dumb search engine. Like if you search for "How do I make a steak", you aren't going to get very good results. But a better query is "Steak Recipe", as that is at least a conceivable H1-tag.

So, you are re-implementing Altavista, Lycos and other old search engines. They used the naive approach: you searched for "steak", and they would bring the pages which included the word "steak". The problem is that people could fool these engines by adding a long sequence like "steak, steak, steak, steak, steak, steak" to their site -- to pretend that they were the most authoritative page about steaks. Google's big i…

> Google's big innovation was to count the referrers -- how many pages used the word "steak" to link to that particular page.

Then people fooled Google into showing the White House as top result when searching for "a miserable failure".

At the moment marginalia's approach of sorting pages into quality buckets based on lack of JS seems to be working extremely well, but of course it will be gamed if it gets popular.

However, I'd rather want SEO-crafting to consider itself with minimizing JS, rather than spamming links into every comment field on every blog across the globe ;-)

Re: A search engine that favors text-heavy sites and punishes modern web design

#716

Post by marginalia_nu »

Siira wrote: post #713
marginalia_nu wrote: post #671

Earlier quoted context omitted.

I'm curious what you searched for. https://search.marginalia.nu/search?query=harry+potter+fanfi... This seems to return a pretty decent number of sites relating to that (as well as some sites not relating to that). The search engine isn't always great at knowing what a page is about, unfortunately. This seemed to return mostly relevant results https://search.marginalia.nu/search?query=%22harry+potter%22...

Yes, shorter queries return more relevant results. I think this was the first query that came to my mind: https://search.marginalia.nu/search?query=Best+%22harry+pott...

Yeah, that's just not a type of query my search engine is particularly good at. It's pretty dumb, and just tries to match as much of the webpage against the query as it can.

This used to be how all search engines worked, but I guess people have been taught by google that they should ask questions now, instead of search for terms.

I wonder how I can guide people to make more suitable queries. Maybe I should just make it look less like google.

Re: A search engine that favors text-heavy sites and punishes modern web design

#717

Post by Aissen »

marginalia_nu wrote: post #709
marto1 wrote: post #295

super! How far would you say are you in indexing the blogosphere ? I tried the engine a few times, but I mostly get academic papers and I know most (good) blogs are in fact text-heavy.

It's not indexed particularly deeply. Blogs typically have a decent amount of javascripts.

I have a static website (granted, it's not well linked), with no JS, and it does not seem to be in the index. But I saw in a sibling post that you had index limits, so it makes sense IMO.

Re: A search engine that favors text-heavy sites and punishes modern web design

#718

Post by interleave »

Wow, I'm floored by the quality of the search results.

For example, I'm a huge fan of obscure Brian Eno stories and interviews and articles.

Using the default blend for "Brian Eno", I found http://www.moredarkthanshark.org/eno_interviews.html which is truly a labor of love and the most comprehensive list of articles (with links to them, too) I've ever seen.

Not in a hundred years would I have found this using Google.

Thank you for building this!

Re: A search engine that favors text-heavy sites and punishes modern web design

#719

Post by inanutshellus »

hdjjhhvvhga wrote: post #651
inanutshellus wrote: post #255

Earlier quoted context omitted.

I see your point, but argue you've misidentified the two audiences. One audience matches your description and is the invested reader. They want that blogger's story telling. they might make the recipe, but they're a dedicated reader. The other audience is not the recipe-searcher, but instead Google. Food bloggers know that recipe-searchers are there to drop in, get an ingredient list, and move on. They won't even rem…

> Food bloggers know that recipe-searchers are there to drop in, get an ingredient list, and move on. This is not entirely true, though. If a randomly found recipe turns out particularly good, I'll bookmark the site and try out other dishes. It's a very practical method to find particularly good* recipe collections. *) In this case "good" means what you need - not just subjectively "tasty", but e.g. low cost, quick t…

I'm aware of zero human-behavior-truisms that are "entirely true".

Re: A search engine that favors text-heavy sites and punishes modern web design

#720

Post by marginalia_nu »

Aissen wrote: post #717
marginalia_nu wrote: post #709

Earlier quoted context omitted.

It's not indexed particularly deeply. Blogs typically have a decent amount of javascripts.

I have a static website (granted, it's not well linked), with no JS, and it does not seem to be in the index. But I saw in a sibling post that you had index limits, so it makes sense IMO.

Probably just hasn't been discovered then. The Internet is big. I wish more small websites would be better at linking to other small websites :-(
Post reply on HN