MangaDex infrastructure overview • Hacker News Board

Re: MangaDex infrastructure overview

#201

Post by majora2007 » Tue, Sep 07, 2021, 4:39 PM UTC

creshal wrote: post #121

germanjoey wrote: post #93

Many manga fans have a love/hate relationship with mangadex. On one hand, it's provided hosting for countless hours of entertainment over the years. Their "v3" version of the site was basically perfect from a usability point of view, to the point that the entire community chose to unite itself under its flag. On the other hand, directly because of the above, their hasty self-inflicted take down earlier this year near…

Yeah. This whole mess pushed me to moving everything I had (or could remember, anyway) to Tachiyomi¹, so I can hop between hosting websites freely without losing progress or access to old chapters (as long as I don't run out of local storage). And while it works fine for reading, it kills any interaction with the hosting sites. No chance for monetization, socialization or anything else that can help sites survive lon…

Totally a self plug, but if you're looking to take it a step further, Kavita is a great program to host your own, Plex-like manga server.

https://kavitareader.com

Re: MangaDex infrastructure overview

#202

Post by hsn915 » Tue, Sep 07, 2021, 4:40 PM UTC

ctvo wrote: post #172

manigandham wrote: post #164

Earlier quoted context omitted.

There are plenty of people who build here on HN (more than most other sites) and the requirements are pretty clearly described in the article. While it's not as simple as a Go program on a VPS, there is certainly a lot of unnecessary overhead here. I think you underestimate just how much poor and wasteful engineering there is out there.

> While it's not as simple as a Go program on a VPS, there is certainly a lot of unnecessary overhead here. I think you underestimate just how much poor and wasteful engineering there is out there. I don't under estimate poor and wasteful engineering at all, but that's not what I saw in the article. Serving traffic is a single element of their design. They also designed for security, redundancy, and observability. Al…

If you have a lot of extra money to throw I'd be happy to oblige.

Re: MangaDex infrastructure overview

#203

Post by level3 » Tue, Sep 07, 2021, 4:42 PM UTC

nrabulinski wrote: post #160

hnlmorg wrote: post #146

Earlier quoted context omitted.

Privacy reasons? It's all static content that is publicly accessible. I don't understand what the privacy reasons could be under this context. Are they worried about CDNs logging the images their visitors access? Seems like an absurd edge case to worry about in my opinion. > however, those options are unlikely to be free I wasn't even talking about free CDNs :)

They’re basically hosting illegal content, or at least a good chunk of it is copyright-infringing so they cannot use cloudflare or any of the other off the shelf offerings

Hm? There are lots of copyright-infringing sites using Cloudflare, and Cloudflare seems pretty content to generally ignore infringement notices.

Re: MangaDex infrastructure overview

#204

Post by hsn915 » Tue, Sep 07, 2021, 4:44 PM UTC

thoughtFrame wrote: post #182

vymague wrote: post #171

Earlier quoted context omitted.

> If they went old-school they'd get another order of magnitude savings. Many veterans know of systems doing 10x that in 10x less cost. Remember C10K was in 1999. How to learn more about the old-school way without getting a job related to it? Like, topic or book recommendations.

I'm also curious about this, since in many areas of computing (not only webdev) the old-school guys take some stuff as so obvious that they don't even bother writing about it or explaining it beyond "This is obviously possible, dude". They know how to achieve this level of performance, but for everyone else, we have to cobble together fragmented insights. So if anyone out there reads this and thinks like the GP, plea…

What topics specifically are you interested in? And where do all the people like you hang out?

I'm not an old-school guy by any means .. but I might have something to contribute.

Re: MangaDex infrastructure overview

#205

Post by hsn915 » Tue, Sep 07, 2021, 4:46 PM UTC

kragen wrote: post #48

jiggawatts wrote: post #40

What kills me is that this was a rather pedestrian outcome on a much cheaper 2-core virtual machine back in 2007 or so. I easily got 3K requests / sec out of my laptop at the same time, and it was not a trivial app! People's expectations have shifted so much it's absurd. If you look at the TechEmpower benchmarks, ordinary VMs can easily push 100K requests per second, no sweat, even with managed languages. Trivial stu…

I think even a distributed cache in front of a database shouldn't have any trouble handling 2000 requests per second. The issue is not really the number of requests per second, probably, but the number of bytes, which they don't talk about at all in the article; reading manga with no ads is a pretty static kind of application, which could be satisfied amply with a web browser or even a much simpler program loading im…

> distributed cache in front of a database

Already an over kill.

Think smaller. Think simpler.

A single machine serving files directly from the file system (yes, from the SSD attached to the machine) will be able to handle a LOT more.

Re: MangaDex infrastructure overview

#206

Post by IncRnd » Tue, Sep 07, 2021, 4:46 PM UTC

Hamuko wrote: post #76

IncRnd wrote: post #45

Earlier quoted context omitted.

Okay, but isn't most of their content stolen? Why would you want to contribute to that?

Depends on what you consider "stolen". In most cases, the manga that is available is translated and edited by fans to make it accessible to English-speakers when the IP owners do not see a reason to do it themselves. The amount of manga that actually get official English releases is very tiny and western licensing companies do not have many incentives to start picking up obscure manga that no one without the ability…

It's what most of the world considers stolen.

Scanlations are often viewed by fans as the only way to read comics that have not been licensed for release in their area. However, according to international copyright law, such as the Berne Convention, scanlations are illegal. [1]

This is a snippet about the Berne Convention:

The Berne Convention for the Protection of Literary and Artistic Works, usually known as the Berne Convention, is an international agreement governing copyright, which was first accepted in Berne, Switzerland, in 1886. The Berne Convention has 179 contracting parties, most of which are parties to the Paris Act of 1971.

The Berne Convention formally mandated several aspects of modern copyright law; it introduced the concept that a copyright exists the moment a work is "fixed", rather than requiring registration. It also enforces a requirement that countries recognize copyrights held by the citizens of all other parties to the convention. [2]

  [1] https://en.wikipedia.org/wiki/Scanlation#Legal_action
  [2] https://en.wikipedia.org/wiki/Berne_Convention

Re: MangaDex infrastructure overview

#207

Post by maxk42 » Tue, Sep 07, 2021, 4:54 PM UTC

Cipater wrote: post #157

Folcon wrote: post #139

Earlier quoted context omitted.

To architecting a high traffic site =)...

He posted a reply to his own comment. https://news.ycombinator.com/item?id=28443113

Actually, my reply was to Folcon. HN simply doesn't allow you to reply to comments beyond a certain depth sometimes.

Perhaps mods have the ability to extend this for active discussions and that's why I can reply now?

Re: MangaDex infrastructure overview

#208

Post by detaro » Tue, Sep 07, 2021, 4:55 PM UTC

maxk42 wrote: post #207

Cipater wrote: post #157

Earlier quoted context omitted.

He posted a reply to his own comment. https://news.ycombinator.com/item?id=28443113

Actually, my reply was to Folcon. HN simply doesn't allow you to reply to comments beyond a certain depth sometimes. Perhaps mods have the ability to extend this for active discussions and that's why I can reply now?

it's timing based. you can always reply by going to the permalink of the comment you want to reply to.

Re: MangaDex infrastructure overview

#209

Post by maxk42 » Tue, Sep 07, 2021, 4:58 PM UTC

detaro wrote: post #208

maxk42 wrote: post #207

Earlier quoted context omitted.

Actually, my reply was to Folcon. HN simply doesn't allow you to reply to comments beyond a certain depth sometimes. Perhaps mods have the ability to extend this for active discussions and that's why I can reply now?

it's timing based. you can always reply by going to the permalink of the comment you want to reply to.

Couldn't reply to this comment - but sure enough, the permalink gives me the option. Thank you for the info!

Re: MangaDex infrastructure overview

#210

Post by kmeisthax » Tue, Sep 07, 2021, 4:58 PM UTC

vymague wrote: post #154

germanjoey wrote: post #93

Many manga fans have a love/hate relationship with mangadex. On one hand, it's provided hosting for countless hours of entertainment over the years. Their "v3" version of the site was basically perfect from a usability point of view, to the point that the entire community chose to unite itself under its flag. On the other hand, directly because of the above, their hasty self-inflicted take down earlier this year near…

> their hasty self-inflicted take down earlier this year nearly killed the entire hobby It won't kill the hobby. Because these scanlators are making mad money from ads, patreon, crypto mining. I'll never get why they don't get more aggressive take down notices from Chinese/Japanese/Korean publishers.

Copyright enforcement is actually quite expensive, both for the litigant and the defendant. The only way for it to be actually profitable to sue someone who is stealing your work is if they immediately settle, which is how copyright trolls operate. Everything else is a massive money pit for everyone involved, even the lawyers. Since this is an international enforcement action, the costs go up more, because now you need multiple legal teams on the bar in each jurisdiction, translators qualified for interpreting laws in foreign languages, knowledge of local copyright quirks, and a lot more coordination than just asking your local counsel to send a takedown notice locally.

(Just as an example of a local copyright quirk that will probably confuse a lot of people in the audience from Europe: copyright registration. America really, really wants you to register your copyright, even though they signed onto Berne/WTO/TRIPS which was supposed to abolish that regime entirely. As a result, America did the bare minimum of compliance. You don't lose your copyright if you don't register, but you can't sue until you do, and if you register after your work was infringed, you don't get statutory damages... which means your costs go way up.)

Furthermore, every enforcement action you take risks PR backlash. The whole fandom surrounding import Japanese comic books basically grew out of a piracy scene. Originally, there were no English translations, and the scene was basically reusing what we'd now call "orphan works". There used to be an unspoken rule among most fansubbers of not translating material that was licensed in the US. All that's changed; most everything gets licensed and many fan translators absolutely are stepping on the toes of licensees. However, every time a licensee or licensor actually takes an enforcement action, they get huge amounts of blowback from their own fans.