Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MangaDex infrastructure overview

mangadex.dev

Post reply on HN
Print view View original

131–140 of 241 posts

Re: MangaDex infrastructure overview

#131

Post by pahae »

sofixa wrote: post #88
jprupp wrote: post #73

This is complexity for complexity's sake. Pay no attention to the disclaimer at the start of the article. They threw every buzzword-heavy bit of tech they could find at it, creating a Frankenstein monster.

Completely disagree. How would you do it in a simpler way, while keeping the features like redundancy ( including storage), logs, metrics, etc?

Looking at their diagrams it seems that the k8s cluster exists solely to handle their monitoring and logging needs which would be extreme overkill, especially since 18k metrics/samples and 7k logs per second are nothing. Plus you now suddenly need a S3-compatible storage backend for all your logs and metrics. Good thing Ceph comes 'free' with Proxmox, I guess.

Deploying an instance of Prometheus with *every host is also unusual, to say the least and I don't quite understand their comment to that. If you don't like a pull-based architecture (which is a valid point) why use one at all!? There are many more push-based setups out there that are simpler to set up and less complex.

Re: MangaDex infrastructure overview

#132

Post by maxk42 »

Folcon wrote: post #130
maxk42 wrote: post #112

Earlier quoted context omitted.

lol > In general the less moving parts you have in a system the more reliable, secure, efficient and cheaper the system becomes. 100% agreed. This is not my first high-traffic site, nor even the highest. (I built the analytics system for a an Alexa top-10 site in 2010, reaching some 30 billion writes / day off of a mere 14 small ec2 instances.) I've never seen a k8s implementation in production that was necessary. I…

Would you mind outlining your approach? Really interested to see how you think about this sort of thing =)...

My approach to what?

Re: MangaDex infrastructure overview

#133

Post by hwers »

IncRnd wrote: post #45
robertwt7 wrote: post #24

I had nothing but respect for the whole team. Dedicating their time to build everything from scratch, not to mention that they maintain everything for free.. It's a cool project, not sure if there's a way for anyone to contribute. I"ll join the discord afterwork to see if they need any extra hand. Gee, how do these people find other people online to work on all of the cool projects. I would love to join rather than p…

Okay, but isn't most of their content stolen? Why would you want to contribute to that?

Just curious if anyone reading this knows the answer: Would it be illegal to contribute man-hours on e.g. implementing features or fixing bugs on a project like this, or does that only apply to whoever actually hosts the content?

Re: MangaDex infrastructure overview

#134

Post by htns »

snypher wrote: post #2

>more than 10 million unique monthly visitors >our ~$1500/month budget I understand not wanting to show ads, but is there no way for the users to contribute to hosting costs?

They had a bit under $80k in crypto in their list of BTC and ETH addresses "leaked" along with the source code when the site was hacked earlier this year.

Re: MangaDex infrastructure overview

#135

Post by Hamuko »

hwers wrote: post #133
IncRnd wrote: post #45

Earlier quoted context omitted.

Okay, but isn't most of their content stolen? Why would you want to contribute to that?

Just curious if anyone reading this knows the answer: Would it be illegal to contribute man-hours on e.g. implementing features or fixing bugs on a project like this, or does that only apply to whoever actually hosts the content?

MPA tried to get the source code for Nyaa.si removed from GitHub because the "Repository hosts and offers for download the Project, which, when downloaded, provides the downloader everything necessary to launch and host a “clone” infringing website identical to Nyaa.si (and, thus, engage in massive infringement of copyrighted motion pictures and television shows)".

It was a completely retarded play on MPA's part and they only managed to get the repo down for days until GitHub restored it even without hearing from the repo owners. So really they only brought about some minor nuisance alongside a bunch of headlines to advertise Nyaa.si for the rest of the world.

https://torrentfreak.com/mpa-takes-down-nyaa-github-reposito...

https://torrentfreak.com/github-restores-nyaa-repository-as-...

Re: MangaDex infrastructure overview

#136

Post by [deleted] »

tristan9 wrote: post #113
maxk42 wrote: post #97

I run an Alexa top-2000 website. (Mangadex is presently at about 6000.) I spend less than $250 a month. I have loads and loads of thoughts about what they could be doing differently to reduce their costs but I'll just say that the number one thing Mangadex could be doing right now from a cursory glance is to reduce the number of requests. A fresh load of the home page generates over 100 requests. (Mostly images, then…

Hi, we're trying to lower the requests:pageview ratio in general, but for what it's worth this article essentially: - ignores the vast majority of "image serving" (most is handled by DDG and our custom CDN) - the JS fragments thankfully should load only on first visit and then get aggressively cached by DDG/your browser One of the pain points is that there are a lot of settings for users to decide what they should or…

[deleted]

Re: MangaDex infrastructure overview

#137

Post by rk06 »

hnlmorg wrote: post #109
rk06 wrote: post #91

Earlier quoted context omitted.

There is more to it that http request response. Mangadex also need to store a lot of images and distribute them.

CDNs have already solved this problem and are much cheaper than $1500/month. I've ran far more complex sites with much higher traffic for less.

Mangadex can't use cloudflare because of privacy reasons. They may be facing similar issues with other popular CDNs.

I am sure they must be using some kind of CDN for sure, however, those options are unlikely to be free

Re: MangaDex infrastructure overview

#138

Post by the8472 »

tristan9 wrote: post #113
maxk42 wrote: post #97

I run an Alexa top-2000 website. (Mangadex is presently at about 6000.) I spend less than $250 a month. I have loads and loads of thoughts about what they could be doing differently to reduce their costs but I'll just say that the number one thing Mangadex could be doing right now from a cursory glance is to reduce the number of requests. A fresh load of the home page generates over 100 requests. (Mostly images, then…

Hi, we're trying to lower the requests:pageview ratio in general, but for what it's worth this article essentially: - ignores the vast majority of "image serving" (most is handled by DDG and our custom CDN) - the JS fragments thankfully should load only on first visit and then get aggressively cached by DDG/your browser One of the pain points is that there are a lot of settings for users to decide what they should or…

One issue I see is that flipping back and forth between chapters reloads images from different URLs which means they're uncachable. I guess that's somehow related to the mangadex@home thing, but if the URLs were generated in a more deterministic manner (keyed on some client ID + the chapter being loaded) then the browser could avoid redundant traffic.

Re: MangaDex infrastructure overview

#140

Post by maxk42 »

tristan9 wrote: post #113
maxk42 wrote: post #97

I run an Alexa top-2000 website. (Mangadex is presently at about 6000.) I spend less than $250 a month. I have loads and loads of thoughts about what they could be doing differently to reduce their costs but I'll just say that the number one thing Mangadex could be doing right now from a cursory glance is to reduce the number of requests. A fresh load of the home page generates over 100 requests. (Mostly images, then…

Hi, we're trying to lower the requests:pageview ratio in general, but for what it's worth this article essentially: - ignores the vast majority of "image serving" (most is handled by DDG and our custom CDN) - the JS fragments thankfully should load only on first visit and then get aggressively cached by DDG/your browser One of the pain points is that there are a lot of settings for users to decide what they should or…

> the JS fragments thankfully should load only on first visit and then get aggressively cached by DDG/your browser

According to Alexa you have a 46.4% bounce rate. [1]

When 46% of your users aren't coming back, how does 31 round-trips to your server for 100% of first-page visitors save anyone time or bandwidth? Your pageviews per visitor is 6.8, meaning the 53.6% that stick around view an average of 11.8 pages each. Even if there are zero subsequent js requests on other pages (clicking a random page I see 8) you would be generating 31 requests up-front to save 10.8 subsequent requests for about half of your users. (And again - in any scenario where the number of js fragments transferred on subsequent requests >= 1 even this benefit goes out the window.) How does that save you or your users bandwidth, server load, or other overhead?

The scale is not quite linear, but generally speaking, if you get your number of requests down from > 100 to Would that have a material effect on your costs?

[1] https://www.alexa.com/siteinfo/mangadex.org

Post reply on HN