A search engine that favors text-heavy sites and punishes modern web design • Hacker News Board

Re: A search engine that favors text-heavy sites and punishes modern web design

#61

Post by marginalia_nu » Thu, Sep 16, 2021, 1:42 PM UTC

asah wrote: post #46

marginalia_nu wrote: post #3

Yeah so this is my project. It's very much a work in progress, but occasionally I think it works remarkably well for something I cobbled together alone out of consumer hardware and home-made code :-)

Love it, kudos! This is great for developers and others who Just Need Answers and not shopping or entertainment. If you're looking for feedback, both from a UI design and utility standpoint, you might consider "inlining" results from selected sites, e.g. Wikipedia, stacked change, etc. Having worked on search for a long time, inlining (onebox etc) is a big reason users choose Google, and that channelers fail to get t…

If Google starts showing interesting text-heavy links instead of vapid listicles and storefronts, I have accomplished everything I ever could dream of.

Re: A search engine that favors text-heavy sites and punishes modern web design

#62

Post by ricardo81 » Thu, Sep 16, 2021, 1:44 PM UTC

ape4 wrote: post #32

Fast and doesn't crash when on the front page of Hacker News!

That crossed my mind too, considering vanilla webpages sometimes struggle with a top page HN thread, never mind a search engine backend.

Re: A search engine that favors text-heavy sites and punishes modern web design

#63

Post by marginalia_nu » Thu, Sep 16, 2021, 1:47 PM UTC

BugsJustFindMe wrote: post #43

marginalia_nu wrote: post #3

Yeah so this is my project. It's very much a work in progress, but occasionally I think it works remarkably well for something I cobbled together alone out of consumer hardware and home-made code :-)

I love this, and I love (many of) the results so far! What I can't find on the site is detail about what "too many modern web design features" means. Is it just penalizing sites with tons of JavaScript?

Javascript tags are penalized the hardest, but it also takes into consideration density of text per HTML. There's also some adjustments based on text length, which words occur in the page, etc.

Re: A search engine that favors text-heavy sites and punishes modern web design

#64

Post by monkeybutton » Thu, Sep 16, 2021, 1:49 PM UTC

platz wrote: post #47

modern design = low information density?

Definitely low signal to noise. Looking at you recipe websites and cooking blogs.

Re: A search engine that favors text-heavy sites and punishes modern web design

#65

Post by streamofdigits » Thu, Sep 16, 2021, 1:50 PM UTC

Let a thousand search engines bloom.

btw, interesting how many http (as opposed to https) sites show up...

Re: A search engine that favors text-heavy sites and punishes modern web design

#66

Post by c0wb0yc0d3r » Thu, Sep 16, 2021, 1:50 PM UTC

dannyw wrote: post #25

c0wb0yc0d3r wrote: post #21

Earlier quoted context omitted.

How did you go about seeding your web crawler with URLs to crawl?

Not OP, but if I was to do this, I'd start by downloading Wikipedia and all its external links and references, and crawling from there. You should eventually reach most of the publicly visible internet.

I feel a little embarrassed that I didn't think of something like that.

When I did some crawler experimenting in my younger years, I thought I was pretty clever using sites that would let you perform a random Google searches. I would just crawl all the pages from the results returned.

Your method would undoubtedly be more interesting I think. It would certainly lead to interesting performance problems quicker, I bet.

Re: A search engine that favors text-heavy sites and punishes modern web design

#67

Post by michaelgrafl » Thu, Sep 16, 2021, 1:51 PM UTC

I just looked up my last name and found a World class heavyweight weightlifter named Josef Grafl born in 1872 who has an awesome portrait of him on Wikipedia. Never before have I read about that man.

I love this.

Re: A search engine that favors text-heavy sites and punishes modern web design

#68

Post by xattt » Thu, Sep 16, 2021, 1:53 PM UTC

duckmysick wrote: post #60

xipho wrote: post #50

Fascinating. I studied an "obscure" group of insects. My go-to search term to test an engine is their family name as it is a rarely used word and I know most (all?) of the major data sources that have accumulated data on it. When Wolfram Alpha added species names, I checked with the name, boring, Duck Duck, boring, Google (well we know Google isn't for search anymore, it's absolutely horrible) boring, Bing, boring...…

I'm intrigued by this experiment but I can't visualize it. What do you mean by boring results? Would combing through a library (the one with paper books) also produce boring results? What's your ideal results?

There’s certain grey literature that’s not captured in university library federated searches nor easily found with mainstream search engines.

Re: A search engine that favors text-heavy sites and punishes modern web design

#69

Post by IggleSniggle » Thu, Sep 16, 2021, 1:53 PM UTC

camillomiller wrote: post #2

Great idea, awful UI

Huh. I think it's a great UI. What did you not like about it?

Re: A search engine that favors text-heavy sites and punishes modern web design

#70

Post by trutannus » Thu, Sep 16, 2021, 1:53 PM UTC

xipho wrote: post #50

Fascinating. I studied an "obscure" group of insects. My go-to search term to test an engine is their family name as it is a rarely used word and I know most (all?) of the major data sources that have accumulated data on it. When Wolfram Alpha added species names, I checked with the name, boring, Duck Duck, boring, Google (well we know Google isn't for search anymore, it's absolutely horrible) boring, Bing, boring...…

> well we know Google isn't for search anymore

Do you suggest anything better? As far as I can tell, all the other search engines are either repackaged Bing (ie: DuckDuck), or are just as bad.