Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A search engine that favors text-heavy sites and punishes modern web design

search.marginalia.nu

Post reply on HN
Print view View original

531–540 of 735 posts

Re: A search engine that favors text-heavy sites and punishes modern web design

#531

Post by jevgeni »

tentacleuno wrote: post #513
xipho wrote: post #50

Fascinating. I studied an "obscure" group of insects. My go-to search term to test an engine is their family name as it is a rarely used word and I know most (all?) of the major data sources that have accumulated data on it. When Wolfram Alpha added species names, I checked with the name, boring, Duck Duck, boring, Google (well we know Google isn't for search anymore, it's absolutely horrible) boring, Bing, boring...…

> well we know Google isn't for search anymore, If you're talking about the ads, I would bear in mind Google's whole business model is basically online advertising. Search is just the vehicle to deliver those ads; I'd say Google is pretty good at throwing things back.

But what’s their UVP? I’d say quick and relevant search results. And that seems to be constantly degrading.

Re: A search engine that favors text-heavy sites and punishes modern web design

#532

Post by Nition »

Ajef wrote: post #526
Nition wrote: post #368

Earlier quoted context omitted.

The Wikipedia link at the top is always given. It would maybe be good to make it a little clearer that it's not one of the true results.

I think this is just because of terms you have searched. In my test-searches Wikipedia has not come up once in first position (i think the highest was 3rd in the list). Here's what I've tried with a few variations: golang generics proposal, machine learning transformer, covid hospitalization germany [edit] formatting

I think maybe it's a special insert at the top, but only if a Wikipedia page is found that matches you search term? I'm not sure now though.

Re: A search engine that favors text-heavy sites and punishes modern web design

#533

Post by hilbert42 »

It's excellent, I looked up some physics topics and got some excellent results - real meaty stuff full of text, eqations and applicable diagrams, etc.

I've not only bookmarked it but also I've an icon linked to it on the taskbar. Will watch its progress with interest.

Re: A search engine that favors text-heavy sites and punishes modern web design

#534

Post by Dah00n »

rchaud wrote: post #322
foofoo4u wrote: post #218

Earlier quoted context omitted.

Good comparison. Reminds me of an analogy I like to make of today's web, which is it feels like browsing through a magazine store — full of top 10s, shallow wow-factoids, and baity material. I genuinely believe terrible results like this are making society dumber.

The context matters. I'd happily read "Top 10" lists on a website if the site itself was dedicated to that one thing. "Top 10 Prog Rock albums", while a lazy, SEO-bait title, would at least be credible if it were on a music-oriented website. But no, these stories all come from cookie-cutter "new media" blog sites, written by an anonymous content writer who's repackaged Wikipedia/Discogs info into Buzzfeed-style copy…

I'm not sure I agree with your example. It seems to me it is the exact same as a "Top ten drinks to drink on a rainy day" list. There's simply too many good albums and opinions differ, so a top ten would -just like the drinks- end up being a list of the most popular ones with maybe one the author picks to stir some controversy or discussion. In my opinion the world would be a smarter place if Google ranked all such sites low. Then we might at least get fluff like "Top ten prog rock albums if you love X, hate Y and listen to Z when no one is around" instead.

Re: A search engine that favors text-heavy sites and punishes modern web design

#535

Post by resynth1943 »

snakeboy wrote: post #114

Wow, that's awesome. Great work! For a simple test, I searched "fall of the roman empire". In your search engine, I got wikipedia, followed by academic talks, chapters of books, and long-form blogs. All extremely useful resources. When I search on google, I get wikipedia, followed by a listicle "8 Reasons Why Rome Fell", then the imdb page for a movie by the same name, and then two Amazon book links, which are totall…

Yeah, Google tends to send a lot of junk back.

Re: A search engine that favors text-heavy sites and punishes modern web design

#536

Post by marginalia_nu »

androceium wrote: post #363
marginalia_nu wrote: post #184

Earlier quoted context omitted.

Hmm, which OS? I only have a single Android phone so I've only fixed the CSS for that.

I was seeing it on Android w/ Firefox. Seems like it's fixed now though. :)

Curious, I haven't touched the stylesheets.

Re: A search engine that favors text-heavy sites and punishes modern web design

#537

Post by Dah00n »

hnick wrote: post #466
bluGill wrote: post #391

Earlier quoted context omitted.

what I really want is a true AI to search through all that and figure out the useful truth. I don't know how to do this (and of course whoever writes the AI needs to be unbiased...)

I think that is the goal, it's just what we currently have is an AI that's like a naive child who is easily tricked and distracted by clickbait.

>an AI that's easily tricked and distracted by clickbait.

So, AIs are actually on par with most adults now? (Sorry)

Re: A search engine that favors text-heavy sites and punishes modern web design

#538

Post by tentacleuno »

jevgeni wrote: post #531
tentacleuno wrote: post #513

Earlier quoted context omitted.

> well we know Google isn't for search anymore, If you're talking about the ads, I would bear in mind Google's whole business model is basically online advertising. Search is just the vehicle to deliver those ads; I'd say Google is pretty good at throwing things back.

But what’s their UVP? I’d say quick and relevant search results. And that seems to be constantly degrading.

Well, the unique value proposition is their gigantic index, really fast search and a bunch of other things.

I'm not sure about the quality of the results, I just use DuckDuckGo these days, but IMO the unique technical advancements are pretty unique to Google.

Re: A search engine that favors text-heavy sites and punishes modern web design

#539

Post by Dah00n »

SPBS wrote: post #274
snakeboy wrote: post #114

Wow, that's awesome. Great work! For a simple test, I searched "fall of the roman empire". In your search engine, I got wikipedia, followed by academic talks, chapters of books, and long-form blogs. All extremely useful resources. When I search on google, I get wikipedia, followed by a listicle "8 Reasons Why Rome Fell", then the imdb page for a movie by the same name, and then two Amazon book links, which are totall…

Cool, it appears that the trend towards JS may be causing self-selection -- if a page has a high amount of JS, it is highly unlikely to contain anything of value.

Huh. A weighted algorithm, somewhere between Google and the one linked, where you could subtract from sites by amount of JavaScript might be interesting.

Re: A search engine that favors text-heavy sites and punishes modern web design

#540

Post by marginalia_nu »

soheil wrote: post #439
marginalia_nu wrote: post #3

Yeah so this is my project. It's very much a work in progress, but occasionally I think it works remarkably well for something I cobbled together alone out of consumer hardware and home-made code :-)

Awesome project! How are you able to keep the site running after HN kiss of death? What is your stack, elastic search or something simper? How did you crawl so many websites for a project this size? Did you use any APIs like duck duck go or data from other search engines? Are you still incorporating something like PageRank to ensure good results are prioritized or is it just the text-based-ness factor?

> How are you able to keep the site running after HN kiss of death?

I originally targeted a Raspberry Pi4-cluster. It was only able to deal with about 200k pages at that stage, but it did shape the design in a way that makes very thrifty use of the available hardware.

My day job is also developing this sort of highly performance java applications, I guess it helps.

> What is your stack, elastic search or something simper?

It's a custom index engine I built for this. I do use mariadb for some ancillary data and to support the crawler, but it's only doing trivial queries.

> How did you crawl so many websites for a project this size?

It's not that hard. Like it seems like it would be, and there certainly is an insane number of edge cases, but if you just keep tinkering you can easily crawl dozens of pages per second even on modest hardware (of course distributed across different domains).

> Did you use any APIs like duck duck go or data from other search engines?

Nope, it's all me.

> Are you still incorporating something like PageRank to ensure good results are prioritized or is it just the text-based-ness factor?

I'm using a somewhat convoluted algorithm that takes into consideration the text-based-ness of the page, but also how many incoming links the domain has, but it's a weighted value that factors in the text-based-ness of the origin domains.

It would be interesting to try a page rank-style approach, but my thinking is that because it's the algorithm, it's also the algorithm everyone is trying to game.

Post reply on HN