Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MangaDex infrastructure overview

mangadex.dev

Post reply on HN
Print view View original

151–160 of 241 posts

Re: MangaDex infrastructure overview

#151

Post by tristan9 »

jiggawatts wrote: post #147
tristan9 wrote: post #141

Earlier quoted context omitted.

> The real problem is that generating that much JSON is very "heavy" on servers. Lots and lots of small object allocations, which gives the garbage collector a ton of work to do. It's also expensive to decode on the browser for similar reasons. For what it's worth, this isn't generated live but a mix of existing entity documents Most of it is page filenames which indeed could be made optional and fetched only by the…

You're basically dumping down a database to the web browser, including all of the internal metadata that's likely irrelevant to rendering the HTML. For example, user role memberships: { "id": "c80b68c5-09ae-4a50-a447-df7c5a4a6d01", "type": "user", "attributes": { "username": "kinshiki", "roles": [ "ROLE_MEMBER", "ROLE_GROUP_MEMBER", "ROLE_POWER_UPLOADER" ], "version": 1 } } Also record timestamp dates like created/ch…

As I said, it's not so much that we ask that data to be fetched -- it is there in the first place, and pulled from Elasticsearch, not a SQL database

Because of this model, we also make sure that Elasticsearch merely works a search cache, not as an authoritative content database (hence everything we add in there is considered public, on purpose, and what isn't meant to be public is just not indexed in ES)

However the gzip efficiency improvements would be really neat for sure

Fwiw I also don't work on the backend and there might be good reasons to not expressly filter out data (yet anyway, perhaps it will end up as a separate entity and be a include parameter)

Re: MangaDex infrastructure overview

#152

Post by tyingq »

maxk42 wrote: post #97

I run an Alexa top-2000 website. (Mangadex is presently at about 6000.) I spend less than $250 a month. I have loads and loads of thoughts about what they could be doing differently to reduce their costs but I'll just say that the number one thing Mangadex could be doing right now from a cursory glance is to reduce the number of requests. A fresh load of the home page generates over 100 requests. (Mostly images, then…

>A fresh load of the home page generates over 100 requests.

I see 17 requests, all over either h2 or h3. 4 of them JS, and 2 images.

Re: MangaDex infrastructure overview

#153

Post by vymague »

maxk42 wrote: post #145
maxk42 wrote: post #132

Earlier quoted context omitted.

My approach to what?

(1) Simple beats complex. (2) You can spend weeks building complex infrastructure or caching systems only to find out that some fixed C in your equation was larger than your overhead savings. In other words: Measure everything. In other other words: Premature optimization is the root of all evil. (3) Fewer moving parts equals less overhead. (Again: Simple beats complex.) It also makes things simpler to reason about.…

> But for the frameworks that aren't doing those things, the benchmark is solid.

Any example of such frameworks?

Re: MangaDex infrastructure overview

#154

Post by vymague »

germanjoey wrote: post #93

Many manga fans have a love/hate relationship with mangadex. On one hand, it's provided hosting for countless hours of entertainment over the years. Their "v3" version of the site was basically perfect from a usability point of view, to the point that the entire community chose to unite itself under its flag. On the other hand, directly because of the above, their hasty self-inflicted take down earlier this year near…

> their hasty self-inflicted take down earlier this year nearly killed the entire hobby

It won't kill the hobby. Because these scanlators are making mad money from ads, patreon, crypto mining. I'll never get why they don't get more aggressive take down notices from Chinese/Japanese/Korean publishers.

Re: MangaDex infrastructure overview

#155

Post by vymague »

hnlmorg wrote: post #146
rk06 wrote: post #137

Earlier quoted context omitted.

Mangadex can't use cloudflare because of privacy reasons. They may be facing similar issues with other popular CDNs. I am sure they must be using some kind of CDN for sure, however, those options are unlikely to be free

Privacy reasons? It's all static content that is publicly accessible. I don't understand what the privacy reasons could be under this context. Are they worried about CDNs logging the images their visitors access? Seems like an absurd edge case to worry about in my opinion. > however, those options are unlikely to be free I wasn't even talking about free CDNs :)

I think privacy as in the mangadex team don't want to get sued. So they avoid popular services who are more willingly share their identity.

Re: MangaDex infrastructure overview

#156

Post by true_religion »

hsn915 wrote: post #89

I don't understand. Why is 2k requests/sec supposed to be massive? Try this yourself: write a simple web server in Go, host it on a cheap VPS provider, let's say at the option that costs $20/mo. Your website will be able to handle more than 1k/s requests with hardly any resource usage. ok, let's assume you're doing some complicated things. So what? You can scale vertically, upgrade to the $120/mo server. Your website…

They also host the manga. It’s not just an link farm. Because they host… that’s why they use ceph.

Their goal is for scanlators to have a place to post their new translated manga, rather than always linking it off from some Wordpress instance.

Re: MangaDex infrastructure overview

#158

Post by baybal2 »

jiggawatts wrote: post #129
tristan9 wrote: post #113

Earlier quoted context omitted.

Hi, we're trying to lower the requests:pageview ratio in general, but for what it's worth this article essentially: - ignores the vast majority of "image serving" (most is handled by DDG and our custom CDN) - the JS fragments thankfully should load only on first visit and then get aggressively cached by DDG/your browser One of the pain points is that there are a lot of settings for users to decide what they should or…

Hi, I'm a performance tuning expert, and this thread piqued my interest. The first thing that I noticed is that even with caching enabled, you're loading "too much data". After loading the main page and then clicking one of the tiles, there are several JSON API calls. Here's an example, 195 kB transferred (528 kB size): https://api.mangadex.org/manga/bbaa17c4-0f36-4bbb-9861-34fc8... Oof. Half a megabyte of JSON! Igno…

> This is one of the core reasons most sites have difficulty scaling, because for every kilobyte of content output to the screen, they're powering through megabytes or even gigabytes of data behind the scenes.

> Can this API query be cut down to match what's displayed on the screen? Can it be cached for all users? Can it be cached precompressed?

This is why you want to bypass the JS realm, (or whatever language does the serdes) and send clients JSON or XML directly from the database, so the client is only getting the data at rest.

Re: MangaDex infrastructure overview

#159

Post by ctvo »

hsn915 wrote: post #89

I don't understand. Why is 2k requests/sec supposed to be massive? Try this yourself: write a simple web server in Go, host it on a cheap VPS provider, let's say at the option that costs $20/mo. Your website will be able to handle more than 1k/s requests with hardly any resource usage. ok, let's assume you're doing some complicated things. So what? You can scale vertically, upgrade to the $120/mo server. Your website…

> Try this yourself: write a simple web server in Go, host it on a cheap VPS provider, let's say at the option that costs $20/mo. Your website will be able to handle more than 1k/s requests with hardly any resource usage.

These people have never heard of Go, obviously. The likely scenario is not that you haven't fully understood their constraints or requirements, it's that you're just smarter than they are.

> So what? You can scale vertically, upgrade to the $120/mo server. Your website now should be able to comfortably handle 5k req/s

> Looking at the website itself, mangadex.org, it doesn't even host the manga itself. The whole website is just an index that links to manga on external websites. All you are doing is storing metadata and displaying it as a webpage. The easiest problem on the web.

Take that order of magnitude cheaper, single VPS server solution you're proposing and build something with it. Sounds like you'd make a lot of money. There has to be a business idea around "storing metadata and displaying it as a webpage" somewhere? Easiest problem on the web.

The peanut gallery at HN is out of control. People who don't do / build explaining to the people who do how easy, simple, better their solutions would be.

Re: MangaDex infrastructure overview

#160

Post by nrabulinski »

hnlmorg wrote: post #146
rk06 wrote: post #137

Earlier quoted context omitted.

Mangadex can't use cloudflare because of privacy reasons. They may be facing similar issues with other popular CDNs. I am sure they must be using some kind of CDN for sure, however, those options are unlikely to be free

Privacy reasons? It's all static content that is publicly accessible. I don't understand what the privacy reasons could be under this context. Are they worried about CDNs logging the images their visitors access? Seems like an absurd edge case to worry about in my opinion. > however, those options are unlikely to be free I wasn't even talking about free CDNs :)

They’re basically hosting illegal content, or at least a good chunk of it is copyright-infringing so they cannot use cloudflare or any of the other off the shelf offerings
Post reply on HN