MangaDex infrastructure overview • Hacker News Board

Re: MangaDex infrastructure overview

#171

Post by vymague » Tue, Sep 07, 2021, 1:19 PM UTC

willvarfar wrote: post #58

jiggawatts wrote: post #40

What kills me is that this was a rather pedestrian outcome on a much cheaper 2-core virtual machine back in 2007 or so. I easily got 3K requests / sec out of my laptop at the same time, and it was not a trivial app! People's expectations have shifted so much it's absurd. If you look at the TechEmpower benchmarks, ordinary VMs can easily push 100K requests per second, no sweat, even with managed languages. Trivial stu…

Excellent post, good technical content, amazing feat. That said, I echo that the amazing feat is that they can fit modern inefficient tool choices with poor mechanical sympathy into that budget. The last decade of web-dev tooling has been pushing the TCO of systems through the roof and this post is all about how to struggle against that whilst using those tools. If they went old-school they'd get another order of mag…

> If they went old-school they'd get another order of magnitude savings. Many veterans know of systems doing 10x that in 10x less cost. Remember C10K was in 1999.

How to learn more about the old-school way without getting a job related to it? Like, topic or book recommendations.

Re: MangaDex infrastructure overview

#172

Post by ctvo » Tue, Sep 07, 2021, 1:20 PM UTC

manigandham wrote: post #164

ctvo wrote: post #159

Earlier quoted context omitted.

> Try this yourself: write a simple web server in Go, host it on a cheap VPS provider, let's say at the option that costs $20/mo. Your website will be able to handle more than 1k/s requests with hardly any resource usage. These people have never heard of Go, obviously. The likely scenario is not that you haven't fully understood their constraints or requirements, it's that you're just smarter than they are. > So what…

There are plenty of people who build here on HN (more than most other sites) and the requirements are pretty clearly described in the article. While it's not as simple as a Go program on a VPS, there is certainly a lot of unnecessary overhead here. I think you underestimate just how much poor and wasteful engineering there is out there.

> While it's not as simple as a Go program on a VPS, there is certainly a lot of unnecessary overhead here. I think you underestimate just how much poor and wasteful engineering there is out there.

I don't under estimate poor and wasteful engineering at all, but that's not what I saw in the article.

Serving traffic is a single element of their design. They also designed for security, redundancy, and observability. All with their own solutions because using a service or a cloud provider would be too costly. With that in mind, it's not a charitable view to think they didn't explore low hanging fruits like "make the server in Go". If you think you can do better, detail in depth how and solve all of their requirements vs. the single piece you're familiar with.

And if you can do the above holistically, for an order of magnitude below their costs, it sounds like I need to get in touch to throw money at you.

Re: MangaDex infrastructure overview

#173

Post by krageon » Tue, Sep 07, 2021, 1:25 PM UTC

ctvo wrote: post #159

hsn915 wrote: post #89

I don't understand. Why is 2k requests/sec supposed to be massive? Try this yourself: write a simple web server in Go, host it on a cheap VPS provider, let's say at the option that costs $20/mo. Your website will be able to handle more than 1k/s requests with hardly any resource usage. ok, let's assume you're doing some complicated things. So what? You can scale vertically, upgrade to the $120/mo server. Your website…

> Try this yourself: write a simple web server in Go, host it on a cheap VPS provider, let's say at the option that costs $20/mo. Your website will be able to handle more than 1k/s requests with hardly any resource usage. These people have never heard of Go, obviously. The likely scenario is not that you haven't fully understood their constraints or requirements, it's that you're just smarter than they are. > So what…

> People who don't do / build

I can and do frequently advise on certain topics in comments specifically because I do build and can in fact speak of such topics authoritatively. Isn't that what this website is for?

That said, the post you are replying to is perhaps overly dismissive of the criteria that this website operates under. Other comment chains have some really good advice though.

Re: MangaDex infrastructure overview

#174

Post by politelemon » Tue, Sep 07, 2021, 1:25 PM UTC

arethuza wrote: post #167

maxk42 wrote: post #145

Earlier quoted context omitted.

(1) Simple beats complex. (2) You can spend weeks building complex infrastructure or caching systems only to find out that some fixed C in your equation was larger than your overhead savings. In other words: Measure everything. In other other words: Premature optimization is the root of all evil. (3) Fewer moving parts equals less overhead. (Again: Simple beats complex.) It also makes things simpler to reason about.…

"Simple beats complex." In the very first lecture of the Computer Science degree I did in the 1980s the lecturer emphasised KISS, and said that while we almost certainly wouldn't believe it at first eventually we'd realise that this is the most important design principle of all. Probably took me ~15 years... ;-)

Sadly I think this is a lesson that we as an industry consistently keep unlearning.

Re: MangaDex infrastructure overview

#175

Post by arethuza » Tue, Sep 07, 2021, 1:26 PM UTC

kiba wrote: post #168

arthur_sav wrote: post #165

Earlier quoted context omitted.

> Simple beats complex. > Fewer moving parts equals less overhead. Took me almost a decade to really comprehend this. I used to include all sorts of libraries, try out all the fancy patterns/architectures etc... After countless of hours debugging production issues... the best code i've ever written is the one with the fewer moving parts. Easier to debug and the issues are predictable.

"The best part is no part." is an engineering quote I heard.

I'm sure I've heard something like "engineering is solving problems while doing as little new as possible".

Re: MangaDex infrastructure overview

#176

Post by krageon » Tue, Sep 07, 2021, 1:30 PM UTC

nine_k wrote: post #99

bob1029 wrote: post #86

Earlier quoted context omitted.

>20k RPS. If this metric is what you are chasing, there are ways to reliably break 1 million RPS using a single box if you don't play the shiny BS tech game. The moment you involve multiple computers and containers, you are typically removed from this level of performance. Going from 2,000 to 2,000,000 RPS (serialized throughput) requires many ideological sacrifices. Mechanical sympathy (ring buffers, batching, minim…

I frankly don't see where containers could lower the performance. Basically a container is a glorified chroot. It has the same networking unless you asked for isolation, then packets have to follow a local (inside the host) route. It has exactly no CPU or kernel interface penalty. Maybe you wanted to say about container orchestration like k8s, with its custom network fabric, etc.

> I frankly don't see where containers could lower the performance.

Have you seen most k8s deployments? It's not the containers, it's the thoughtspace that comes with them. Even just using bare containers invites a level of abstraction and generally comes with a type of developer that just isn't desirable.

Re: MangaDex infrastructure overview

#177

Post by m45t3r » Tue, Sep 07, 2021, 1:31 PM UTC

AviKav wrote: post #94

m45t3r wrote: post #12

Earlier quoted context omitted.

There is the MangaDex@Home, where users can serve part of disk space/bandwidth to help serve (mainly old) manga chapters. It does need to be something that is running 24/7 (e.g.: not a PC that is shutdown frequently), so something like a VPS or a service is recommended.

Virtually every chapter is served via MD@H now. Client doesn't really need much availability, as long as it can do a graceful shutdown. Even in the event of a sudden shutdown, the trust penalties are much lower than H@H and in practice go away after a trickle of traffic to raise your score

Nice, didn't know about this (there isn't much information about MD@H after the rewrite).

BTW, how can I register my VPS on MD@H? Before we had a dedicated form on the page to register interest, at least after the rewrite I didn't find it. Is it only using something like Discord?

Re: MangaDex infrastructure overview

#178

Post by m45t3r » Tue, Sep 07, 2021, 1:41 PM UTC

creshal wrote: post #121

germanjoey wrote: post #93

Many manga fans have a love/hate relationship with mangadex. On one hand, it's provided hosting for countless hours of entertainment over the years. Their "v3" version of the site was basically perfect from a usability point of view, to the point that the entire community chose to unite itself under its flag. On the other hand, directly because of the above, their hasty self-inflicted take down earlier this year near…

Yeah. This whole mess pushed me to moving everything I had (or could remember, anyway) to Tachiyomi¹, so I can hop between hosting websites freely without losing progress or access to old chapters (as long as I don't run out of local storage). And while it works fine for reading, it kills any interaction with the hosting sites. No chance for monetization, socialization or anything else that can help sites survive lon…

Before MangaDex we had Batoto (the old Batoto before some sketchy company bought the name), that was kinda of the same: serving high quality manga for most scanlators that wanted it (and also avoiding hosting pirated chapters from official sources, so kinda the same as MangaDex nowadays). As far I remember Batoto closed because of pressure from companies and also because of the high costs related to run the site.

So yeah, considering how fragile maintaining a site like this is, it is always a good idea to sync your progress in a third party so it is easier to migrate if something goes wrong.

> And while it works fine for reading, it kills any interaction with the hosting sites. No chance for monetization, socialization or anything else that can help sites survive long-term.

BTW, MangaDex doesn't have monetization because it is strict a hobby and also because it is a gray area to monetize about this kinda of work [1]. Also, their Tachiyomi client is official (MangaDex v5 API was tested primarily via their Tachiyomi client before they finished the Web interface).

[1]: both for companies (that has the copyright from the works hosted on those sites) and the scanlators (the fans that does actual work of translating those chapters). Sites that host those chapters and monetize are pretty much monetizing on work from other people.

Re: MangaDex infrastructure overview

#179

Post by genewitch » Tue, Sep 07, 2021, 1:43 PM UTC

sigstoat wrote: post #74

jmiserez wrote: post #69

Earlier quoted context omitted.

640KiB is very little and I'm wondering if it's a typo, given that the servers had 1-2TiB available. Postgres 9.0 released in 2010 already had 32MiB as the default for shared_buffers (with a minimum of 128KiB): https://www.postgresql.org/docs/9.0/runtime-config-resource.... and 8.1 released in 2005 used 8MB (1000*8KiB): https://www.postgresql.org/docs/8.1/runtime-config-resource....

i interpreted it as "we wanted to turn the shared buffers ~off, but in a hilarious way that would suggest to someone reading the configuration file that something was going on" (bill gates, mumble mumble) but, wtf do i know, i'm the crazy guy who tries to interpret comments generously.

Yes, it was a direct reference to Bill Gates "640 kilobytes is enough for anyone" and i typed the comment right before i fell asleep.

Re: MangaDex infrastructure overview

#180

Post by genewitch » Tue, Sep 07, 2021, 1:47 PM UTC

winrid wrote: post #82

genewitch wrote: post #42

Earlier quoted context omitted.

In 2011 a company i contracted for was testing some new dell 1U servers with around 1-2TB of ram. There was a postgres database with 4000qps that could fit into tmpfs, and so i restricted postgres to 640Kb of memory and we got replication working, it took about 6 hours of babysitting. We threw the switch and watched as postgres, with 640Kb of ram and a tmpfs backed store proceeded to handle all of the query traffic.…

Postgres handles low memory situations well. It'll kill memory intensive queries before it crashes. I wonder if your application was getting a lot of errors back instead of successful queries :)

the application performed fine, even though we made the switch around 15:00 PST. The DBA was concerned because of the few long queries.

Obviously the tmpfs was doing the heavy lifting, there - and if i had to do a postmortem, i'd wager that filling the OS caches was the main reason the long queries took so long. We didn't do any sort of performance tracing.

The main purpose was to show that these $35k servers could essentially replace the older machines if need be, even though the old ones had FusionIO. I just removed the middleman of the PCIe bus between the application and the memory. It was a near constant argument on the floor about whether or not we could feasibly switch to SSDs in some configuration over spinning rust or even FusionIO, i wanted a third option.

Basically, serve out of registered, ECC memory in front, replicate to the fusionIO and let those handle the spindled backups, which iirc was a pain point.