Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MangaDex infrastructure overview

mangadex.dev

Post reply on HN
Print view View original

21–30 of 241 posts

Re: MangaDex infrastructure overview

#21

Post by m45t3r »

mdoms wrote: post #7

I loaded the front page of Mangadex and it made 114 web requests including 10 first-party XHR requests, 30(!!!!) Javascript resource requests and somehow 4 font requests, without me interacting with the page. Clicking one of the titles on the front page resulted in nearly 40 additional requests. Perhaps if you are limited by requests per second you could consider how many requests a single user is making per interact…

They're probably more limited by bandwidth than requests per second, but anyway you look the number of requests are still impressive considered the budget.

BTW, the site is not just fast: they serve images on high quality (same as the original [1], that can be multiple MBs per page [2]) at an pretty impressive speed too.

[1]: before someone asks why they don't optimize the images, this is by design since they want to serve high quality images. There is an optional toggle to reduce the image size, but this is disabled by default.

[2]: for those not familiar, the average number of pages on a manga is something like ~20, and this can be read in ~5 minutes depending on the density of the text. So you can easily consume 50MB+ per chapter.

Re: MangaDex infrastructure overview

#22

Post by 29athrowaway »

latch wrote: post #19

I've done things at scale (5-10K req/s) on a budget ($1000 USD) and I've done things at much smaller scales that required a much larger budget. _How_ you hit scale on a budget is one part of the equation. The other part is: what you're doing. Off the top of my head, the "how" will often involve the following (just to list a few): 1 - Baremetal 2 - Cache 3 - Denormalize 4 - Append-only 5 - Shard 6 - Performance focuse…

Try to convert as much content as you can into static content, and serve it via CDN. Then, use your servers only for dynamic stuff.

Also, put the browser to work for you, caching via Cache-Control, ETag, etc. Only then, optimize your server...

Re: MangaDex infrastructure overview

#23

Post by novok »

whateveracct wrote: post #4
snypher wrote: post #2

>more than 10 million unique monthly visitors >our ~$1500/month budget I understand not wanting to show ads, but is there no way for the users to contribute to hosting costs?

A $5/mo premium plan would break even so quickly

Or even a patreon style 'you get nothing but a supporter badge' with that kind of traffic levels.

Re: MangaDex infrastructure overview

#24

Post by robertwt7 »

I had nothing but respect for the whole team. Dedicating their time to build everything from scratch, not to mention that they maintain everything for free.. It's a cool project, not sure if there's a way for anyone to contribute.

I"ll join the discord afterwork to see if they need any extra hand.

Gee, how do these people find other people online to work on all of the cool projects. I would love to join rather than playing games after WFH on the same pc over and over again lol

Re: MangaDex infrastructure overview

#25

Post by bawolff »

urlgrey wrote: post #6

I'm amazed that their architecture doesn't include a CDN. These days I expect nearly all high traffic websites to make use of a CDN for all kinds of content, even content that's not cached. They cited Cloudflare not being used due to privacy concerns. It'd be interesting to hear more about that, as well as why other CDNs weren't worth evaluating too.

What's the benefit of a cdn if nothing is cacheable? Slightly lower latency on the tcp/tls handshake? That seems pretty insignificant.

Re: MangaDex infrastructure overview

#26

Post by ev1 »

bawolff wrote: post #25
urlgrey wrote: post #6

I'm amazed that their architecture doesn't include a CDN. These days I expect nearly all high traffic websites to make use of a CDN for all kinds of content, even content that's not cached. They cited Cloudflare not being used due to privacy concerns. It'd be interesting to hear more about that, as well as why other CDNs weren't worth evaluating too.

What's the benefit of a cdn if nothing is cacheable? Slightly lower latency on the tcp/tls handshake? That seems pretty insignificant.

The CDN part is kind of pointless because they can't really have nodes in large parts of the western world since.. it's a warez site. The CDN providers will get takedowns, requests to reveal the backing origin, etc. You can't use a commodity CDN provider for this.

Re: MangaDex infrastructure overview

#27

Post by gaudat »

urlgrey wrote: post #6

I'm amazed that their architecture doesn't include a CDN. These days I expect nearly all high traffic websites to make use of a CDN for all kinds of content, even content that's not cached. They cited Cloudflare not being used due to privacy concerns. It'd be interesting to hear more about that, as well as why other CDNs weren't worth evaluating too.

They do have a crowdsourced CDN called Mangadex@Home. I participated in it from last year until the site was hacked. The aggregate egress speed was around 10 Gbps.

The NSFW counterpart of MD also has a CDN appropriately named Hentai@Home run by volunteers.

These 2 sites are the only ones rolling their own CDN for free that I know.

Re: MangaDex infrastructure overview

#28

Post by TekMol »

My cheap $20/month VPS serves tens of thousands a user per day without breaking much of a sweat. Using a good old LAMP stack (Linux, Apache, MariaDB, PHP).

I don't know how many requests per second it can handle.

Trying a guess via curl:

time curl --insecure --header 'Host: www.mysite.com' https://127.0.0.1 > test

This gives me 0.03s

So it could handle about 30 requests per second? Or 30x the number of CPUs? What do you guys think?

Re: MangaDex infrastructure overview

#29

Post by dharmab »

TekMol wrote: post #28

My cheap $20/month VPS serves tens of thousands a user per day without breaking much of a sweat. Using a good old LAMP stack (Linux, Apache, MariaDB, PHP). I don't know how many requests per second it can handle. Trying a guess via curl: time curl --insecure --header 'Host: www.mysite.com' https://127.0.0.1 > test This gives me 0.03s So it could handle about 30 requests per second? Or 30x the number of CPUs? What do…

Does it serve 20-40 hi resolution images and uploads per user?

Re: MangaDex infrastructure overview

#30

Post by watermelon0 »

mdoms wrote: post #7

I loaded the front page of Mangadex and it made 114 web requests including 10 first-party XHR requests, 30(!!!!) Javascript resource requests and somehow 4 font requests, without me interacting with the page. Clicking one of the titles on the front page resulted in nearly 40 additional requests. Perhaps if you are limited by requests per second you could consider how many requests a single user is making per interact…

Frontend framework they use (NUXT) uses code splitting [1], which means that:

- first request is fast, because you only need to download chunks required for a single page/controller (and you prefetch others in the background)

- changing some parts of codebase requires to re-download only affected chunks, instead of the whole bundle

[1] https://www.telerik.com/blogs/what-you-should-know-code-spli...

Post reply on HN