Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A search engine that favors text-heavy sites and punishes modern web design

search.marginalia.nu

Post reply on HN
Print view View original

651–660 of 735 posts

Re: A search engine that favors text-heavy sites and punishes modern web design

#651

Post by hdjjhhvvhga »

inanutshellus wrote: post #255
WorldMaker wrote: post #210

Earlier quoted context omitted.

That sort of recipe blog hasn't happened just for SEO. It's also a bit of a "two audiences" problem: if you are coming to that food blogger from a search you certainly would prefer the recipe first and then maybe any commentary on it below if the recipe looks good. If you are a regular reader of that food blogger you are probably invested in the stories up top and that parasocial connection and the recipes themselves…

I see your point, but argue you've misidentified the two audiences. One audience matches your description and is the invested reader. They want that blogger's story telling. they might make the recipe, but they're a dedicated reader. The other audience is not the recipe-searcher, but instead Google. Food bloggers know that recipe-searchers are there to drop in, get an ingredient list, and move on. They won't even rem…

> Food bloggers know that recipe-searchers are there to drop in, get an ingredient list, and move on.

This is not entirely true, though. If a randomly found recipe turns out particularly good, I'll bookmark the site and try out other dishes. It's a very practical method to find particularly good* recipe collections.

*) In this case "good" means what you need - not just subjectively "tasty", but e.g. low cost, quick to prepare, low calorie or in line with a particular diet and so on.

Re: A search engine that favors text-heavy sites and punishes modern web design

#652

Post by aoms »

marginalia_nu wrote: post #3

Yeah so this is my project. It's very much a work in progress, but occasionally I think it works remarkably well for something I cobbled together alone out of consumer hardware and home-made code :-)

What's the tech stack?

Re: A search engine that favors text-heavy sites and punishes modern web design

#653

Post by dotancohen »

acchow wrote: post #619
dotancohen wrote: post #611

Earlier quoted context omitted.

Mission accomplished, then.

If the goal was to remove modern web design, ok sure mission accomplished. If your goal was to create a search engine that ignored listicles and other fluff and instead got you meatier results like "academic talks" and such, then no.

When a measure becomes a target, it ceases to be a good measure.

https://en.wikipedia.org/wiki/Goodhart%27s_law

Re: A search engine that favors text-heavy sites and punishes modern web design

#654

Post by slim »

marginalia_nu wrote: post #647
slim wrote: post #275

This is a search engine indexing the internet on a mariadb database hosted on consumer hardware maintained by a single person as a hobby and it does not suffer from HN hug of death

Huh. I've gone through the logs, and at most I got about 2 search queries per second as a sustained load over several hours. It's calmed down to about 0.5-1 QPS now. This is ignoring other page loads. Not bad, if I do say so myself.

It's a statement to how much other websites are bad

Re: A search engine that favors text-heavy sites and punishes modern web design

#656

Post by marginalia_nu »

slim wrote: post #654
marginalia_nu wrote: post #647

Earlier quoted context omitted.

Huh. I've gone through the logs, and at most I got about 2 search queries per second as a sustained load over several hours. It's calmed down to about 0.5-1 QPS now. This is ignoring other page loads. Not bad, if I do say so myself.

It's a statement to how much other websites are bad

Do I wonder how much cookies play into it.

Sure I have small page loads and reasonably optimized code, but I also don't use any cookies. I'd imagine having to keep track of upwards of a million session objects would be a struggle for almost any web server.

Re: A search engine that favors text-heavy sites and punishes modern web design

#657

Post by Aachen »

marginalia_nu wrote: post #54
timdaub wrote: post #24

I'm developing a text-heavy site and philosophically I'm trying to view documents as just that... documents [1]. But I don't get good results for "rug pull". - 1 https://rugpullindex.com

Yeah it's hosted by cloudflare. I'm currently IP-blocking them, as because they keep prompting my crawler with a captcha, presumably because it's made millions of requests from their CDN. Some rigmarole getting recognized as a good bot by the CDNs. I've submitted a request fairly recently, but haven't heard back from them yet. Like I would like to be on good terms with them, and other websites that block small indepe…

I wonder if google had to plead the same way or if already-big players are given a free advantage.

Re: A search engine that favors text-heavy sites and punishes modern web design

#658

Post by GaylordTuring »

noduerme wrote: post #399
marginalia_nu wrote: post #3

Yeah so this is my project. It's very much a work in progress, but occasionally I think it works remarkably well for something I cobbled together alone out of consumer hardware and home-made code :-)

I love this idea, and admire the work you put into it. I'm a fan of long reads and historical non-fiction, and Google's results are truly garbage. I have a criticism that I think may pertain to the ranking methodology. I searched for "discovery of Australia". Among the top results were: * A site claiming that the biblical flood was caused by Earth colliding with a comet (with several other pages from that site also m…

I don’t want my search engine to somehow try to judge the believability of the results. I’d like to be the judge of that myself.

Re: A search engine that favors text-heavy sites and punishes modern web design

#659

Post by TeMPOraL »

freddref wrote: post #419
pomian wrote: post #288

Earlier quoted context omitted.

PS. I added Marginalia as a search option (even the default for now) in Firefox Nightly (on Android). In case others want to, under settings for search, you can add other, then name, and then: https://search.marginalia.nu/search?query=%s

After a good amount of searching it doesn't seem possible to add Marginalia as default search in firefox (84.0b8) on Debian. I did not expect this to not be available.

Open a search, right-click on the address bar. The author added the magic metadata needed, so Firefox should be able to pick it up and offer "add search engine" option in the context menu.

I know. It's an idiotic UX regression over being able to configure search engines yourself in the Settings screen. Takes control away from the user.

Re: A search engine that favors text-heavy sites and punishes modern web design

#660

Post by TeMPOraL »

jillesvangurp wrote: post #252
marginalia_nu wrote: post #8

Earlier quoted context omitted.

I do indeed index the web myself. Not the entire web, just a subset of it. The crawler quickly loses interest in javascript:y websites and only indexes at depth those websites that are simple. It also focuses on websites in English, Swedish and Latin and tries to identify and ignore the rest (best-effort). You'd be surprised how much you can do with modern hardware if you are scrappy. The current index is about 17.7…

Cool, I've been thinking on this topic a bit lately. Crawling is indeed not that hard of a problem. Google could do it 23 years ago. The web is a bit bigger now of course but it's not that bad. Those numbers are well within the range of a very modest search cluster (pick your favorite technology; it shouldn't be challenging for any of them). 10x or 1000x would not matter a lot for this. Although it would raise your c…

> The hard part is keeping the system fair and balanced.

It is, which is why I think the author should stay away from anything requiring users to vote on things.

The problem with deriving reputability from votes over time is in distinguishing legitimate votes from malicious votes. Voting is something that doesn't just get gamed, it gets gamed as a service. You'll have companies selling votes, and handling all the busywork necessary to game the bad vote detector.

Search engines and social media companies don't ignore this topic - on the contrary, they live by it. The problem of reputation vote quality is isomorphic to the problem of ad click quality. The "vote" is a click event on an ad, and the profitability for both the advertiser and the ad network depend on being able to tell legitimate clicks and fake clicks apart. Ludicrous amounts of money went into solving this problem, and the end result is... surveillance state. All this deep tracking on the web, it doesn't exist just - or even primarily - to target ads. It exists to determine whether a real would-be customer is looking at an ad, or if it's a bot farm (including protein bot farm, aka. people employed to click on ads en masse).

We need something better. Something that isn't as easy to game, and where mitigations don't come with such a high price for the society.

Post reply on HN