Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MangaDex infrastructure overview

mangadex.dev

Post reply on HN
Print view View original

101–110 of 241 posts

Re: MangaDex infrastructure overview

#101

Post by kragen »

buzer wrote: post #98
kragen wrote: post #48

Earlier quoted context omitted.

I think even a distributed cache in front of a database shouldn't have any trouble handling 2000 requests per second. The issue is not really the number of requests per second, probably, but the number of bytes, which they don't talk about at all in the article; reading manga with no ads is a pretty static kind of application, which could be satisfied amply with a web browser or even a much simpler program loading im…

> The issue is not really the number of requests per second, probably, but the number of bytes, which they don't talk about at all in the article; reading manga with no ads is a pretty static kind of application, which could be satisfied amply with a web browser or even a much simpler program loading images from a filesystem directory. I assume they are talking about their more dynamic content serving in this post (f…

Interesting! Thanks! It still doesn't sound like the kind of thing that would require load balancing, but maybe it was easier to write it in Python or PHP or something, and that made it so heavy that it did.

Re: MangaDex infrastructure overview

#102

Post by lifthrasiir »

neonbones wrote: post #90
IncRnd wrote: post #45

Earlier quoted context omitted.

Okay, but isn't most of their content stolen? Why would you want to contribute to that?

The world of scanlations is always on edge. Usually, when publishers announce official translations of manga titles, fans drop translations of this title. It's not rare that publishers hire fans who were translating this title before for free as an official team. To be more precise, the real reason why such sites are alive is that they delete titles that got licenses in Europe and the USA. Still, publishers can measu…

Heck no. As per the Berne Convention they are 100% illegal even in the western world and can only survive due to the neglect or lack of legal resources---I have seen multiple cases where artists were well aware of scanlations but couldn't fight against them because of that. A legal way to do scanlation would be always welcomed (and there have been varying degree of successes in other areas), but it is just wrong to claim that they are somehow legitimate at all.

Re: MangaDex infrastructure overview

#103

Post by radicalbyte »

maxk42 wrote: post #97

I run an Alexa top-2000 website. (Mangadex is presently at about 6000.) I spend less than $250 a month. I have loads and loads of thoughts about what they could be doing differently to reduce their costs but I'll just say that the number one thing Mangadex could be doing right now from a cursory glance is to reduce the number of requests. A fresh load of the home page generates over 100 requests. (Mostly images, then…

Do you manage to get as many buzz-words and OSS products into your system as they do? :)

In general the less moving parts you have in a system the more reliable, secure, efficient and cheaper the system becomes.

In their case they run a site that is probably under constant attack by the "hired goons", so they're going to need to have more moving parts than others. Plus they will want to optimise for minimal development time (it's a hobby) so just adding another tried and trusted system into the stack to do something you need makes sense.

Re: MangaDex infrastructure overview

#104

Post by ignoramous »

chairmanwow1 wrote: post #60
trampi wrote: post #39

Earlier quoted context omitted.

Reads like a small excerpt out of "Designing Data-Intensive Applications" :)

This is an amazing book that improved my effectiveness as an engineer by an undefinable amount. Instead of just randomly picking components for a cloud application, I learned that I could pick the right tools for the job. This book does a really good job communicating the trade-offs between different designs and tools.

I have always wondered "what next" after having read data-intensive. Some suggested looking at research publications by Google, Facebook, and Microsoft. What do others interested in the field read?

Re: MangaDex infrastructure overview

#105

Post by bebna »

deathanatos wrote: post #78
bytearray64 wrote: post #14

Earlier quoted context omitted.

You probably have Verizon - they've started null routeing traffic to sites "like this". https://old.reddit.com/r/mangadex/comments/nvj7qf/is_verizon...

Huh, right you are, on both accounts, it seems. That's disappointing. If only I had some choice to ISPs, then I could express my disappointment by voting with my wallet…

Just call their service often enough, and tell them internet isnt working.

Re: MangaDex infrastructure overview

#106

Post by Vosporos »

adreamingsoul wrote: post #56
jiggawatts wrote: post #40

What kills me is that this was a rather pedestrian outcome on a much cheaper 2-core virtual machine back in 2007 or so. I easily got 3K requests / sec out of my laptop at the same time, and it was not a trivial app! People's expectations have shifted so much it's absurd. If you look at the TechEmpower benchmarks, ordinary VMs can easily push 100K requests per second, no sweat, even with managed languages. Trivial stu…

No. I think you have a healthy perspective and we should all be questioning if current trends are beneficial/sustainable. I haven’t read the article, but the headline alone to me seems alarming, $1,500 a month is a lot of money for only 2k rps.

Then you should read the article ;)

Re: MangaDex infrastructure overview

#107

Post by bigiain »

toast0 wrote: post #80
kalev wrote: post #41

Earlier quoted context omitted.

The 1-7 list you mention definitely deserves it’s own blogpost and how to implement these. I’m currently not using any of these except 1, and probably don’t need the rest for a while but I do want to know what I should do when I need it. For example: what and how should things be cached? When and how to denormalize, why is it needed? Why append-only and how? Never ‘sharded’ before, no idea how that works. Heard some…

> Never ‘sharded’ before, no idea how that works. Sharding sucks, but if your database can't fit on a single machine anymore, you do what you've got to do. The basic idea is instead of everything in one database on one machine (or well redundant group of machines anyway), you have some method to decide for a given key what database machine will have the data. Managing the split of data across different machines is, o…

> you have some method to decide for a given key what database machine will have the data

Super simple example, splitting there phone book into two volumes, A-K and L-Z. (Hmmmm, is a "phonebook" a thing that typical HN readers remember?)

> you can fit a lot of database in 8 TB of ram, and if you don't keep the whole thing in ram, you can index a ton of data with 8 TB of ram, which means sharding can wait.

For almost everyone, sharing can wait until after the business doesn't need it any more. FAANG need to shard. Maybe a few thousand other companies need to shard. I suspect way way more businesses start sharding when realistically spending more on suitable hardware would easily cover the next two orders of magnitude of growth.

One of these boxes maxed out will give you a few TB of ram, 24 cpu cores, and 24x16TB NVMe drives which gives you 380-ish TB of fairly fast database - for around $135k, and you'd want two for redundancy. So maybe 12 months worth of a senior engineer's time.

https://www.broadberry.com/performance-storage-servers/cyber...

Re: MangaDex infrastructure overview

#108

Post by akx »

Tenoke wrote: post #100
hsn915 wrote: post #89

I don't understand. Why is 2k requests/sec supposed to be massive? Try this yourself: write a simple web server in Go, host it on a cheap VPS provider, let's say at the option that costs $20/mo. Your website will be able to handle more than 1k/s requests with hardly any resource usage. ok, let's assume you're doing some complicated things. So what? You can scale vertically, upgrade to the $120/mo server. Your website…

>Looking at the website itself, mangadex.org, it doesn't even host the manga itself. They do seem to. Clicking on a random manga on there the images are hosted on their server[0]. Also I guess some of those are much bigger images which is less trivial to serve at that rate than a 10kb static page. 0. blob: https://mangadex.org/e78bd61a-e761-4a73-a27c-5f58394e7ea4

Blob links are scoped to your browser tab, they're not real internet URLs.

Re: MangaDex infrastructure overview

#109

Post by hnlmorg »

rk06 wrote: post #91
adreamingsoul wrote: post #56

Earlier quoted context omitted.

No. I think you have a healthy perspective and we should all be questioning if current trends are beneficial/sustainable. I haven’t read the article, but the headline alone to me seems alarming, $1,500 a month is a lot of money for only 2k rps.

There is more to it that http request response. Mangadex also need to store a lot of images and distribute them.

CDNs have already solved this problem and are much cheaper than $1500/month.

I've ran far more complex sites with much higher traffic for less.

Re: MangaDex infrastructure overview

#110

Post by lamnk »

TekMol wrote: post #28

My cheap $20/month VPS serves tens of thousands a user per day without breaking much of a sweat. Using a good old LAMP stack (Linux, Apache, MariaDB, PHP). I don't know how many requests per second it can handle. Trying a guess via curl: time curl --insecure --header 'Host: www.mysite.com' https://127.0.0.1 > test This gives me 0.03s So it could handle about 30 requests per second? Or 30x the number of CPUs? What do…

A day is 16 * 60 * 60 = 57,600 seconds (night time substracted). So tens thousands users per day is like 1-2 req/s, maybe 50 at peak time.

What is more important is what kind of requests your server has to serve. Nginx can easily serve 50-80k req/s of static content; 100ks range if tuned properly.

Post reply on HN