Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MangaDex infrastructure overview

mangadex.dev

Post reply on HN
Print view View original

191–200 of 241 posts

Re: MangaDex infrastructure overview

#191

Post by tpetry »

jiggawatts wrote: post #122
z3t4 wrote: post #55

Earlier quoted context omitted.

2k sockets on a test bed vs 2k real user request in production is very different. I doubt you ran a top 1000 Alexa site on your laptop. Today we need to deal with SSL which eats from the performance budget.

> SSL which eats from the performance budget. That was a short-lived thing, and has now become a myth perpetuated by companies like Citrix and F5 that sell "SSL offload" appliances for $$$. Have you benchmarked the overhead of TLS? In my experience, a single CPU core can easily put out multiple gigabytes of AES-256 (tens of gigabits). This benchmark shows 3 GB/s (24 Gbps) for recent AMD CPUs, and nearly 40 Gbps per c…

The problem with these benchmarks is they measure the bandwidth you can push through an established tls connection. Try to build 2000 new tls connections a seconds (yes many are still active and dont need to be restarted) that is what is the really slow part. Not sending the data over already established channels.

Re: MangaDex infrastructure overview

#192

Post by maxk42 »

probotect0r wrote: post #186
maxk42 wrote: post #145

Earlier quoted context omitted.

(1) Simple beats complex. (2) You can spend weeks building complex infrastructure or caching systems only to find out that some fixed C in your equation was larger than your overhead savings. In other words: Measure everything. In other other words: Premature optimization is the root of all evil. (3) Fewer moving parts equals less overhead. (Again: Simple beats complex.) It also makes things simpler to reason about.…

Can you share how you do logging/monitoring/alerting for your site?

Bash scripts and cron. Automatic alerts go out to devs via OpsGenie when resource availability drops so we can get out ahead of it. 0 seconds of downtime in the past 12 months.

Re: MangaDex infrastructure overview

#193

Post by Jcowell »

germanjoey wrote: post #93

Many manga fans have a love/hate relationship with mangadex. On one hand, it's provided hosting for countless hours of entertainment over the years. Their "v3" version of the site was basically perfect from a usability point of view, to the point that the entire community chose to unite itself under its flag. On the other hand, directly because of the above, their hasty self-inflicted take down earlier this year near…

I never got the sense that the manga community hated Mangadex and I’ve been following their Dev of V5 and the rise of other sites to use in their absence.

It’s seems weird to attribute Mangadex taking their site down for valid security concerns to the end of scandalization of certain series. That seems like entirely a Scan team problem if they decide not to upload via Cubari like other teams have done. And it doesn’t even matter since a series can get sniped at anytime.

It’s makes entire sense that if you’re going to rewrite the backend and API from scratch , you might as well do the front end too since it was a Goal from the beginning.

Re: MangaDex infrastructure overview

#194

Post by level3 »

vymague wrote: post #154
germanjoey wrote: post #93

Many manga fans have a love/hate relationship with mangadex. On one hand, it's provided hosting for countless hours of entertainment over the years. Their "v3" version of the site was basically perfect from a usability point of view, to the point that the entire community chose to unite itself under its flag. On the other hand, directly because of the above, their hasty self-inflicted take down earlier this year near…

> their hasty self-inflicted take down earlier this year nearly killed the entire hobby It won't kill the hobby. Because these scanlators are making mad money from ads, patreon, crypto mining. I'll never get why they don't get more aggressive take down notices from Chinese/Japanese/Korean publishers.

They get plenty of takedown notices, but they mostly get to hide behind services like Cloudflare who won't take action regarding these notices anyway. From the publishers/creators side, there is simply no effective way to take scanlators down.

Re: MangaDex infrastructure overview

#195

Post by maxk42 »

tyingq wrote: post #152
maxk42 wrote: post #97

I run an Alexa top-2000 website. (Mangadex is presently at about 6000.) I spend less than $250 a month. I have loads and loads of thoughts about what they could be doing differently to reduce their costs but I'll just say that the number one thing Mangadex could be doing right now from a cursory glance is to reduce the number of requests. A fresh load of the home page generates over 100 requests. (Mostly images, then…

>A fresh load of the home page generates over 100 requests. I see 17 requests, all over either h2 or h3. 4 of them JS, and 2 images.

Then you're not doing a fresh load of the page. There are over 30 images visible on the front page, so your measure doesn't pass the smell test, does it?

Re: MangaDex infrastructure overview

#196

Post by Jcowell »

hnlmorg wrote: post #163
nrabulinski wrote: post #160

Earlier quoted context omitted.

They’re basically hosting illegal content, or at least a good chunk of it is copyright-infringing so they cannot use cloudflare or any of the other off the shelf offerings

I see. That does complicate things somewhat. I wonder if there's merit in them approaching studios with a proper business plan?

Now with the way the manga, Manghwa, and WEBTOON industry is tied up. But I believe that is their end goal eventually.

Re: MangaDex infrastructure overview

#197

Post by hsn915 »

ctvo wrote: post #159
hsn915 wrote: post #89

I don't understand. Why is 2k requests/sec supposed to be massive? Try this yourself: write a simple web server in Go, host it on a cheap VPS provider, let's say at the option that costs $20/mo. Your website will be able to handle more than 1k/s requests with hardly any resource usage. ok, let's assume you're doing some complicated things. So what? You can scale vertically, upgrade to the $120/mo server. Your website…

> Try this yourself: write a simple web server in Go, host it on a cheap VPS provider, let's say at the option that costs $20/mo. Your website will be able to handle more than 1k/s requests with hardly any resource usage. These people have never heard of Go, obviously. The likely scenario is not that you haven't fully understood their constraints or requirements, it's that you're just smarter than they are. > So what…

> The likely scenario is not that you haven't fully understood their constraints or requirements, it's that you're just smarter than they are.

I never claimed to be smarter. I just understand some things that I noticed a lot of people in the industry don't understand.

My understanding is not even that great.

But still, this is just one example that I keep running into over and over and over:

People opting for a complicated infrastructure setup because that's what they think you should do.

No one showed them how to make a stable reliable website that just runs on a single machine and handle thousands of concurrent connections.

It's not hard. It's just that they've never seen it and assume it's basically impossible.

There are areas about computing that I feel the same way about. For example, before Casey Muratori demoed his refterm implementation, I had no idea that it was possible to render a terminal at thousands of frames per second. I just assumed such a feat was technically impossible. Partly because no one has done it. But then he did it, and I was blown away.

> Take that order of magnitude cheaper, single VPS server solution you're proposing and build something with it. Sounds like you'd make a lot of money.

Building something and making money out of it are not the same thing. But thanks for the advice. I'm in the process of trying. I know for sure I can build the thing, but I don't know if it will make any money. We will see.

> People who don't do / build explaining to the people who do how easy, simple, better their solutions would be.

I do and have done.

This kind of advice is exactly the kind of thing I know how to do because I have done it in the past using my trivial setup of a single process running on a cheap VPS. And I have also seen other teams struggle to get some feature nearly half-working on a complicated infrastructure setup with AWS and all the other buzzwords: Kibana, Elastic Search, Dynamo DB, Ansible, Terraform, Kubernetes ... what else? I can't even keep track of all this stuff that everyone keeps talking about even though hardly anyone needs at all.

I've seen 4 or 5 companies try to build their service using this kind of setup, with the proposed advantange of "horizontal" and "auto" scaling. And you know what? They ALL struggled with poor performance, ALL THE TIME. It's really sad.

Re: MangaDex infrastructure overview

#198

Post by ricktdotorg »

nrabulinski wrote: post #160
hnlmorg wrote: post #146

Earlier quoted context omitted.

Privacy reasons? It's all static content that is publicly accessible. I don't understand what the privacy reasons could be under this context. Are they worried about CDNs logging the images their visitors access? Seems like an absurd edge case to worry about in my opinion. > however, those options are unlikely to be free I wasn't even talking about free CDNs :)

They’re basically hosting illegal content, or at least a good chunk of it is copyright-infringing so they cannot use cloudflare or any of the other off the shelf offerings

Not disputing your statement, just made me laugh a bit because almost every single site I visit recently that offers links to copyrighted content [stored in file lockers] sits behind Cloudflare

Re: MangaDex infrastructure overview

#199

Post by level3 »

chaorace wrote: post #188
vymague wrote: post #154

Earlier quoted context omitted.

> their hasty self-inflicted take down earlier this year nearly killed the entire hobby It won't kill the hobby. Because these scanlators are making mad money from ads, patreon, crypto mining. I'll never get why they don't get more aggressive take down notices from Chinese/Japanese/Korean publishers.

I suspect it's because the international market for print manga (the primary cash cow) is rather anemic, particularly compared to anime. Publishers see the loss as minimal and creators see piracy as free advertising to drum up enthusiasm for anime adaptations, which actually do drum up decent profits internationally ( the committee keeps the streaming licensing fees, not the animation studio ).

Publishers definitely don't see it that way; that's mostly an extension of a myth in order to justify the piracy.

Most manga publishers will see relatively little revenue from international anime releases. Even for domestic anime releases of the vast majority of titles, the manga publisher is only a small part of the anime production committee, and the hope is mostly that popularity of the anime can lead to increased sales of the manga, merchandise, or other events. So when the anime is released internationally, they get an even smaller cut of that because the international licensee also has to take their profit.

But other than mega-hit titles where an international anime release may also lead to significant international manga sales, the popularity of an anime adaptation overseas is practically irrelevant to the original manga publisher.

Post reply on HN