Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MangaDex infrastructure overview

mangadex.dev

Post reply on HN
Print view View original

91–100 of 241 posts

Re: MangaDex infrastructure overview

#91

Post by rk06 »

adreamingsoul wrote: post #56
jiggawatts wrote: post #40

What kills me is that this was a rather pedestrian outcome on a much cheaper 2-core virtual machine back in 2007 or so. I easily got 3K requests / sec out of my laptop at the same time, and it was not a trivial app! People's expectations have shifted so much it's absurd. If you look at the TechEmpower benchmarks, ordinary VMs can easily push 100K requests per second, no sweat, even with managed languages. Trivial stu…

No. I think you have a healthy perspective and we should all be questioning if current trends are beneficial/sustainable. I haven’t read the article, but the headline alone to me seems alarming, $1,500 a month is a lot of money for only 2k rps.

There is more to it that http request response. Mangadex also need to store a lot of images and distribute them.

Re: MangaDex infrastructure overview

#92

Post by colesantiago »

hsn915 wrote: post #89

I don't understand. Why is 2k requests/sec supposed to be massive? Try this yourself: write a simple web server in Go, host it on a cheap VPS provider, let's say at the option that costs $20/mo. Your website will be able to handle more than 1k/s requests with hardly any resource usage. ok, let's assume you're doing some complicated things. So what? You can scale vertically, upgrade to the $120/mo server. Your website…

I agree, just had to read the article again, and took it as a fancy way of wasting money really.

Re: MangaDex infrastructure overview

#93

Post by germanjoey »

Many manga fans have a love/hate relationship with mangadex. On one hand, it's provided hosting for countless hours of entertainment over the years. Their "v3" version of the site was basically perfect from a usability point of view, to the point that the entire community chose to unite itself under its flag.

On the other hand, directly because of the above, their hasty self-inflicted take down earlier this year nearly killed the entire hobby. Many series essentially stopped updating for the ~5 months the site was down, and many more are likely never coming back again.

The decision to suddenly take the site down for a full site rewrite feels completely inexplicable from the outside. (A writeup the above or the previous one[1], both of which read like they were written by a Google Product Manager, especially don't help as they conspicuously avoid any comment to the one question on everyone's mind: "leaving aside the supposed security issues with the backend, why on earth also rewrite and redesign the entire front end from scratch at the same time?")

[1] https://mangadex.dev/why-rebuild/

Re: MangaDex infrastructure overview

#94

Post by AviKav »

m45t3r wrote: post #12
snypher wrote: post #2

>more than 10 million unique monthly visitors >our ~$1500/month budget I understand not wanting to show ads, but is there no way for the users to contribute to hosting costs?

There is the MangaDex@Home, where users can serve part of disk space/bandwidth to help serve (mainly old) manga chapters. It does need to be something that is running 24/7 (e.g.: not a PC that is shutdown frequently), so something like a VPS or a service is recommended.

Virtually every chapter is served via MD@H now. Client doesn't really need much availability, as long as it can do a graceful shutdown. Even in the event of a sudden shutdown, the trust penalties are much lower than H@H and in practice go away after a trickle of traffic to raise your score

Re: MangaDex infrastructure overview

#95

Post by AviKav »

hsn915 wrote: post #89

I don't understand. Why is 2k requests/sec supposed to be massive? Try this yourself: write a simple web server in Go, host it on a cheap VPS provider, let's say at the option that costs $20/mo. Your website will be able to handle more than 1k/s requests with hardly any resource usage. ok, let's assume you're doing some complicated things. So what? You can scale vertically, upgrade to the $120/mo server. Your website…

The only releases that link to external websites are the ones from sites such as MangaPlus and BiliBili (And delayed releases if you count those)

Re: MangaDex infrastructure overview

#96

Post by cyborgx7 »

KingOfCoders wrote: post #83
IncRnd wrote: post #45

Earlier quoted context omitted.

Okay, but isn't most of their content stolen? Why would you want to contribute to that?

Yes of course it is stolen. And people claiming otherwise are the same people who come here and ask "What can I do, some Chinese company ripped of my website?!?!?!"

>And people claiming otherwise are the same people who come here and ask "What can I do, some Chinese company ripped of my website?!?!?!"

Something you made up in your head with literally not a single shred of evidence.

Re: MangaDex infrastructure overview

#97

Post by maxk42 »

I run an Alexa top-2000 website. (Mangadex is presently at about 6000.) I spend less than $250 a month.

I have loads and loads of thoughts about what they could be doing differently to reduce their costs but I'll just say that the number one thing Mangadex could be doing right now from a cursory glance is to reduce the number of requests. A fresh load of the home page generates over 100 requests. (Mostly images, then javascript fragments.) Mangadex apparently gets over 100 million requests per day. My site - despite ostensibly having more traffic - gets fewer than half that many in a month. (And yes, it's image-heavy.)

A couple easy wins would be to reduce the number of images loaded on the front page. (Does the "Seasonal" slider really need 30 images, or would 5 and a link to the "seasonal" page be enough? Same thing with "Recently Added" and the numbers of images on pages in general.) The biggest win would probably be reducing the number of javascript requests. Somehow people seem to think there's some merit to loading javascript which subsequently loads additional javascript. This adds a tremendous amount of latency to your page load and generates needless network traffic. Each request has a tremendous amount of overhead - particularly for dynamically-generated javascript. It's much better to load all of the javascript you need in a single request or a small handful of requests. Unfortunately, this is probably a huge lift for a site already designed in this way, but the improved loading time would be a big UX win.

Anyway - best of luck to MangaDex! They've clearly put a lot of thought into this.

Re: MangaDex infrastructure overview

#98

Post by buzer »

kragen wrote: post #48
jiggawatts wrote: post #40

What kills me is that this was a rather pedestrian outcome on a much cheaper 2-core virtual machine back in 2007 or so. I easily got 3K requests / sec out of my laptop at the same time, and it was not a trivial app! People's expectations have shifted so much it's absurd. If you look at the TechEmpower benchmarks, ordinary VMs can easily push 100K requests per second, no sweat, even with managed languages. Trivial stu…

I think even a distributed cache in front of a database shouldn't have any trouble handling 2000 requests per second. The issue is not really the number of requests per second, probably, but the number of bytes, which they don't talk about at all in the article; reading manga with no ads is a pretty static kind of application, which could be satisfied amply with a web browser or even a much simpler program loading im…

> The issue is not really the number of requests per second, probably, but the number of bytes, which they don't talk about at all in the article; reading manga with no ads is a pretty static kind of application, which could be satisfied amply with a web browser or even a much simpler program loading images from a filesystem directory.

I assume they are talking about their more dynamic content serving in this post (for things like search, tracking which chapters are read, new chapter listing based on what user follows etc.).

They have a custom CDN that is hosted by volunteers to serve the images for the manga pages. They provide some metrics for that at https://mangadex.network, there are also some older screenshots where they hit 3.2GB/s.

Re: MangaDex infrastructure overview

#99

Post by nine_k »

bob1029 wrote: post #86
colesantiago wrote: post #65

Earlier quoted context omitted.

Not just you, but if it works for them, that's completely fine. But there are many ways to achieve 20K RPS without this type architecture and especially without k8s, for less than $1,500.

>20k RPS. If this metric is what you are chasing, there are ways to reliably break 1 million RPS using a single box if you don't play the shiny BS tech game. The moment you involve multiple computers and containers, you are typically removed from this level of performance. Going from 2,000 to 2,000,000 RPS (serialized throughput) requires many ideological sacrifices. Mechanical sympathy (ring buffers, batching, minim…

I frankly don't see where containers could lower the performance.

Basically a container is a glorified chroot. It has the same networking unless you asked for isolation, then packets have to follow a local (inside the host) route. It has exactly no CPU or kernel interface penalty.

Maybe you wanted to say about container orchestration like k8s, with its custom network fabric, etc.

Re: MangaDex infrastructure overview

#100

Post by Tenoke »

hsn915 wrote: post #89

I don't understand. Why is 2k requests/sec supposed to be massive? Try this yourself: write a simple web server in Go, host it on a cheap VPS provider, let's say at the option that costs $20/mo. Your website will be able to handle more than 1k/s requests with hardly any resource usage. ok, let's assume you're doing some complicated things. So what? You can scale vertically, upgrade to the $120/mo server. Your website…

>Looking at the website itself, mangadex.org, it doesn't even host the manga itself.

They do seem to. Clicking on a random manga on there the images are hosted on their server[0]. Also I guess some of those are much bigger images which is less trivial to serve at that rate than a 10kb static page.

0. blob:https://mangadex.org/e78bd61a-e761-4a73-a27c-5f58394e7ea4

Post reply on HN