Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A search engine that favors text-heavy sites and punishes modern web design

search.marginalia.nu

Post reply on HN
Print view View original

301–310 of 735 posts

Re: A search engine that favors text-heavy sites and punishes modern web design

#301

Post by typon »

kodeninja wrote: post #238

Ivermectin (marginalia): https://search.marginalia.nu/search?query=ivermectin+ Ivermectin (Google): https://www.google.com/search?q=ivermectin The difference in the overall _thrust_ of the results is remarkable. Very interesting! Thanks for building it.

The Google results tell you why Ivermectin is not a good replacement for vaccination against Covid, the Marginalia results tell you that Ivermectin is a miracle drug for treating Covid 19. Really shows how much technology has the power to change reality in today's world.

Re: A search engine that favors text-heavy sites and punishes modern web design

#302

Post by dzink »

One use case to always test: "online wishlist" or "make a wishlist". If you start seeing tools like https://www.DreamList.com or others, you are on the right path. If you start seeing random web pages linking to individual wish lists, then people are likely not able to find tools on your search engine.

Re: A search engine that favors text-heavy sites and punishes modern web design

#303

Post by typon »

mmmpop wrote: post #230
spandrew wrote: post #223

All of my searches are turning up unrelated results ("college life after the pandemic", "post-pandemic teaching in higher education", "football news NFL" etc.) NFL one had 'some' decently related results, but the websites were all strangely disreputable.

> the websites were all strangely disreputable Interesting you'd feel that way when sites without "modern design" are encountered. Is this your own bias perhaps creating a judgment or are they sites that you already know have a bad reputation?

Or perhaps the websites being returned are garbage? I have the same experience trying a few searches and following the top 5 links. Besides wikipedia, I haven't found a single useful website.

Re: A search engine that favors text-heavy sites and punishes modern web design

#305

Post by judge2020 »

habibur wrote: post #289
marginalia_nu wrote: post #3

Yeah so this is my project. It's very much a work in progress, but occasionally I think it works remarkably well for something I cobbled together alone out of consumer hardware and home-made code :-)

How are you doing the crawling without getting blocking? -- the hardest part.

Not OP but crawling is easy if you don't try scanning 5+ pages a second - almost all rate limiting/heuristic based 'keep server costs low' engines, including Cloudflare, don't care if you request every page, but will take action if you do something like burst every page and take up just as many server resources as a hundred concurrent users.

Now, that is assuming you aren't on some VPS provider. If you're going to crawl, you'll have the best chance when you use your own IPs on your own ASN, with DNS and reverse DNS set up correctly. This makes it so the IP reputation systems can detect you as a crawler but not one that hammers every site it visits.

Also, I imagine that, for a search engine like this, it doesn't expect content to change much anyways - so it can take its time crawling every site only once every month or two, instead of the multiple times a week (or day) search engines like Google have to for the constantly-updated content being churned out.

Re: A search engine that favors text-heavy sites and punishes modern web design

#306

Post by artificial »

Torwald wrote: post #235
BenoitP wrote: post #221

Earlier quoted context omitted.

Definitely; We could create a meta search engine that queries them all, in desktop application format. Let's name it after a famous old scientist, and maybe add the year to prove it's modern: Galileo 2021.

I need that with a simpler interface, so I call it after a famous dedective: Sherlock.

a magic pop sound is faintly audible as a new side project is appended to several lists Excellent, thank you!

Re: A search engine that favors text-heavy sites and punishes modern web design

#307

Post by kwertyoowiyop »

tomrod wrote: post #263
WorldMaker wrote: post #250

Earlier quoted context omitted.

Personally, I think that has a lot more to do with the fact that Google killed the Recipe Databases. There did used to be a few startups that tried to be Recipe Aggregators with advertising based business models, that would show recipes and then link to source blogs and/or cookbooks, and in the brief period where they existed Google scraped them entirely and showed entire recipes on search results and ate their ad re…

That is a really bad thing by Google. Their core business is not recipes.

Their core business is making money from other people’s content, no matter what it is.

Re: A search engine that favors text-heavy sites and punishes modern web design

#308

Post by blondin »

marginalia_nu wrote: post #3

Yeah so this is my project. It's very much a work in progress, but occasionally I think it works remarkably well for something I cobbled together alone out of consumer hardware and home-made code :-)

fantastic project, thank you!

Re: A search engine that favors text-heavy sites and punishes modern web design

#310

Post by kwertyoowiyop »

withinboredom wrote: post #192
fouric wrote: post #168

Earlier quoted context omitted.

The project explicitly bills itself as a "search engine", not an "interesting and unexpected material surfacer". Moreover, projecting emotions like "angry" onto a comment in order to discredit the content of the comment (hey! is that an ad-hominem?) is just about exactly the opposite of the discussions that the HN mods are trying to curate, and the discussions that I like to see here.

In the early days of google, I found what I was looking for on page 5+. On the way, I’d discover many interesting things I didn’t even know I was looking for, often completely unrelated to what I was searching for.

And now Google hides that more than one page even exists, as they populate their first page with buttons to ask similar questions and go to the first page of THOSE results.
Post reply on HN