Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A search engine that favors text-heavy sites and punishes modern web design

search.marginalia.nu

Post reply on HN
Print view View original

131–140 of 735 posts

Re: A search engine that favors text-heavy sites and punishes modern web design

#131

Post by 0xbadcafebee »

marginalia_nu wrote: post #89
marginalia_nu wrote: post #37

Earlier quoted context omitted.

Well,... yet. Load average is at 1.2, not that bad. But the services are getting a solid workout.

The real test is in now, index server is reconstructing its index. It does this every 6 hours if there is new pages. Takes half an hour or so usually. It's supposed to be able to handle searches at the same time, but jeepers, it's gonna have to chew through nearly 400 Gb of data while dealing with over 1 request per second.

Is your site/code on GitHub? I would be happy to give performance tips/tweaks. Also Fyi, https://marginalia.nu/ gives a certificate error (I know that's not the search site)

Re: A search engine that favors text-heavy sites and punishes modern web design

#132

Post by hdjjhhvvhga »

snakeboy wrote: post #114

Wow, that's awesome. Great work! For a simple test, I searched "fall of the roman empire". In your search engine, I got wikipedia, followed by academic talks, chapters of books, and long-form blogs. All extremely useful resources. When I search on google, I get wikipedia, followed by a listicle "8 Reasons Why Rome Fell", then the imdb page for a movie by the same name, and then two Amazon book links, which are totall…

As long as few people use it, it will be great. Rest assured that the moment it becomes popular, the people who want to game it will appear.

Re: A search engine that favors text-heavy sites and punishes modern web design

#133

Post by skyfaller »

This is a fantastic search engine. It delivers on its promise of "serendipity". I found pages featuring my name that I'm not sure I've ever seen before, after many years of searching myself to test out search engines.

Perhaps more importantly, it delivers the most correct result when searching for my username: the first result is not any of my social media accounts, or even my own blog, but the text of the obscure science fiction story that I took my username from! Well done.

I've immediately added this as a search keyword in Firefox, and I'll be using it more in the future.

Could meta search engines like DuckDuckGo include this as a source? Should they?

Re: A search engine that favors text-heavy sites and punishes modern web design

#134

Post by rchaud »

marcos100 wrote: post #40
jerrre wrote: post #13

Earlier quoted context omitted.

I wouldn't say the existence of this page proves your prediction right (as it's not dethroning Google anytime soon). It's easy to forget that the goal of Google isn't to provide a useful search engine (at least not anymore), but the search engine is a by product of them wanting to show ads.

If Google isn't useful then nobody will use it. The search engine and the ads are tightly coupled. A better search engine means it can predict with more accuracy what you are looking for and can serve you an even more targeted ad that increases the chance you'll click.

> If Google isn't useful then nobody will use it.

Or they'll continue to use it out of sheer inertia. Google is paying Apple $15 billion to keep its place as iOS default search engine.

IE6 didn't die overnight when Firefox arrived.

Re: A search engine that favors text-heavy sites and punishes modern web design

#135

Post by krapp »

xipho wrote: post #100
mda wrote: post #81

Earlier quoted context omitted.

IMHO, that is a trendy claim in HN with little evidence.

You want evidence? Search for a plumber/tradesperson in your area THEN try to find rational discourse about your options. There are literally 100s of results of websites remixing a small set of data, presenting it to you, and asking you to buy something to see more, when you know there is nothing behind the scenes. This type of engine would punish these sites, in theory, and may turn up a discussion in some forum, ne…

> Search for a plumber/tradesperson in your area THEN try to find rational discourse about your options.

I searched "plumber Austin TX" in Google and got a map and list of company websites near me. There are a lot of "top x y in z" list sites, but the top results were still the most relevant. I don't know what "rational discourse" I'm expected to find, though, or why I should assume the discourse I would find through Google is less rational than discourse I would find elsewhere.

I searched the same thing in OP and found nothing even remotely significant. Not even anything related to plumbing.

OP's project isn't optimized for relevance, it's optimized for nostalgia - providing a filter that keeps the modern web away and dropping quirky, interesting breadcrumbs to distract you and remind you of what it was like to wander around the web of the 90's.

Which is all well and good if that's what you want, and judging from the comments it is what a lot of people here want, but Google giving me a list of company names, numbers, websites and a map showing their location by distance is more useful, even if it uses "modern web design" and javascript.

Re: A search engine that favors text-heavy sites and punishes modern web design

#136

Post by rfrey »

This is stunning. I searched "winemaking" because it's my latest obsession, and turned up dozens of links to high-quality pages I'd never seen despite spending an hour a day for three months cruising Google on the topic.

Please do announce it here if you ever decide to solicit help or contributors. My stab at this problem was to have a search index of only ad-free pages, on the hypothesis it would turn up self-hosted blogs, university personal pages, that sort of thing. But the results were too thin, your approach is much better.

Re: A search engine that favors text-heavy sites and punishes modern web design

#139

Post by marginalia_nu »

mrpf1ster wrote: post #124

I searched "c strtok" and got one result saying '"strtok" could be spelled "stroke", "stork", "sarto", "strop"'. Cool concept though!

The spelling suggestions are presented whenever there isn't any results, but sometimes they can be pretty misleading.

What happens is that C, as a word, isn't indexed because it's deemed too short, and the bigram "c strtok" can't be found anywhere.

Try 'strtok' instead.

Re: A search engine that favors text-heavy sites and punishes modern web design

#140

Post by petercooper »

marginalia_nu wrote: post #54
timdaub wrote: post #24

I'm developing a text-heavy site and philosophically I'm trying to view documents as just that... documents [1]. But I don't get good results for "rug pull". - 1 https://rugpullindex.com

Yeah it's hosted by cloudflare. I'm currently IP-blocking them, as because they keep prompting my crawler with a captcha, presumably because it's made millions of requests from their CDN. Some rigmarole getting recognized as a good bot by the CDNs. I've submitted a request fairly recently, but haven't heard back from them yet. Like I would like to be on good terms with them, and other websites that block small indepe…

Aha, I was going to ask how you were coping with CDNs like Cloudflare blocking bots. It's sad we've got to this point where basically only the established search engines are grandfathered in to be able to crawl sites.
Post reply on HN