Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MangaDex infrastructure overview

mangadex.dev

Post reply on HN
Print view View original

51–60 of 241 posts

Re: MangaDex infrastructure overview

#51

Post by ApolloFortyNine »

bytearray64 wrote: post #14
deathanatos wrote: post #3

It's a nice article, I guess, but the site is down (the one discussed in the article, not the blog post itself) for me.

You probably have Verizon - they've started null routeing traffic to sites "like this". https://old.reddit.com/r/mangadex/comments/nvj7qf/is_verizon...

Ah shucks I thought we had mostly avoided that stuff in the US.

I'm guessing though they're using some old spam ip/block though, there's a lot more obvious piracy sites then a Manga site. For instance, I can access all the major torrent sites.

Re: MangaDex infrastructure overview

#52

Post by chime »

novok wrote: post #35
chime wrote: post #15

Not familiar with the project but it is great to see a counterpart to over-provisioned enterprise infrastructure. $10 in 2021 can do what $100 in 2011 did, what $1000 in 2001 did, and that is not solely due to hardware. Well-designed deployments of K8s, KVM/LXC, Ceph, LBs like this project can handle so much more traffic than poorly configured Wordpress storefronts. They're using battle-tested tech from Redis and Rab…

I think enterprise and more optimize for business flexibility and ability to A/B test very rapidly vs a finely crafted piece of efficiency, for better or worse. The people behind this probably do this for their day job, or are teens that are about to do it for their day job.

I agree with you. I mostly work in enterprise and understand that it has different needs and ROI requirements. However, my personal mindset is that computers and networks are really really fast now and it's a tragedy that most of these gains are nullified due to unoptimized layers of abstraction or over-architecting. So it's a welcome sight to read about well-designed infrastructure like this.

Re: MangaDex infrastructure overview

#53

Post by pedrosorio »

kalev wrote: post #41
latch wrote: post #19

I've done things at scale (5-10K req/s) on a budget ($1000 USD) and I've done things at much smaller scales that required a much larger budget. _How_ you hit scale on a budget is one part of the equation. The other part is: what you're doing. Off the top of my head, the "how" will often involve the following (just to list a few): 1 - Baremetal 2 - Cache 3 - Denormalize 4 - Append-only 5 - Shard 6 - Performance focuse…

The 1-7 list you mention definitely deserves it’s own blogpost and how to implement these. I’m currently not using any of these except 1, and probably don’t need the rest for a while but I do want to know what I should do when I need it. For example: what and how should things be cached? When and how to denormalize, why is it needed? Why append-only and how? Never ‘sharded’ before, no idea how that works. Heard some…

As a sibling comment mentioned, read DDIA: https://dataintensive.net/

Re: MangaDex infrastructure overview

#54

Post by sigstoat »

bagels wrote: post #47
genewitch wrote: post #42

Earlier quoted context omitted.

In 2011 a company i contracted for was testing some new dell 1U servers with around 1-2TB of ram. There was a postgres database with 4000qps that could fit into tmpfs, and so i restricted postgres to 640Kb of memory and we got replication working, it took about 6 hours of babysitting. We threw the switch and watched as postgres, with 640Kb of ram and a tmpfs backed store proceeded to handle all of the query traffic.…

For real, 640 kilobits?

K isn't the abbreviation for kilo, so if you're going to rag on the fellow for the 'b', then you should at least be asking what a Kelvin*bit is.

Re: MangaDex infrastructure overview

#55

Post by z3t4 »

jiggawatts wrote: post #40

What kills me is that this was a rather pedestrian outcome on a much cheaper 2-core virtual machine back in 2007 or so. I easily got 3K requests / sec out of my laptop at the same time, and it was not a trivial app! People's expectations have shifted so much it's absurd. If you look at the TechEmpower benchmarks, ordinary VMs can easily push 100K requests per second, no sweat, even with managed languages. Trivial stu…

2k sockets on a test bed vs 2k real user request in production is very different. I doubt you ran a top 1000 Alexa site on your laptop. Today we need to deal with SSL which eats from the performance budget.

Re: MangaDex infrastructure overview

#56

Post by adreamingsoul »

jiggawatts wrote: post #40

What kills me is that this was a rather pedestrian outcome on a much cheaper 2-core virtual machine back in 2007 or so. I easily got 3K requests / sec out of my laptop at the same time, and it was not a trivial app! People's expectations have shifted so much it's absurd. If you look at the TechEmpower benchmarks, ordinary VMs can easily push 100K requests per second, no sweat, even with managed languages. Trivial stu…

No. I think you have a healthy perspective and we should all be questioning if current trends are beneficial/sustainable.

I haven’t read the article, but the headline alone to me seems alarming, $1,500 a month is a lot of money for only 2k rps.

Re: MangaDex infrastructure overview

#57

Post by reilly3000 »

adreamingsoul wrote: post #56
jiggawatts wrote: post #40

What kills me is that this was a rather pedestrian outcome on a much cheaper 2-core virtual machine back in 2007 or so. I easily got 3K requests / sec out of my laptop at the same time, and it was not a trivial app! People's expectations have shifted so much it's absurd. If you look at the TechEmpower benchmarks, ordinary VMs can easily push 100K requests per second, no sweat, even with managed languages. Trivial stu…

No. I think you have a healthy perspective and we should all be questioning if current trends are beneficial/sustainable. I haven’t read the article, but the headline alone to me seems alarming, $1,500 a month is a lot of money for only 2k rps.

Maybe its a lot of money just for the web servers, but for the entire infrastructure stack its pretty reasonable IMHO.

Re: MangaDex infrastructure overview

#58

Post by willvarfar »

jiggawatts wrote: post #40

What kills me is that this was a rather pedestrian outcome on a much cheaper 2-core virtual machine back in 2007 or so. I easily got 3K requests / sec out of my laptop at the same time, and it was not a trivial app! People's expectations have shifted so much it's absurd. If you look at the TechEmpower benchmarks, ordinary VMs can easily push 100K requests per second, no sweat, even with managed languages. Trivial stu…

Excellent post, good technical content, amazing feat.

That said, I echo that the amazing feat is that they can fit modern inefficient tool choices with poor mechanical sympathy into that budget. The last decade of web-dev tooling has been pushing the TCO of systems through the roof and this post is all about how to struggle against that whilst using those tools.

If they went old-school they'd get another order of magnitude savings. Many veterans know of systems doing 10x that in 10x less cost. Remember C10K was in 1999.

Re: MangaDex infrastructure overview

#59

Post by antupis »

kragen wrote: post #48
jiggawatts wrote: post #40

What kills me is that this was a rather pedestrian outcome on a much cheaper 2-core virtual machine back in 2007 or so. I easily got 3K requests / sec out of my laptop at the same time, and it was not a trivial app! People's expectations have shifted so much it's absurd. If you look at the TechEmpower benchmarks, ordinary VMs can easily push 100K requests per second, no sweat, even with managed languages. Trivial stu…

I think even a distributed cache in front of a database shouldn't have any trouble handling 2000 requests per second. The issue is not really the number of requests per second, probably, but the number of bytes, which they don't talk about at all in the article; reading manga with no ads is a pretty static kind of application, which could be satisfied amply with a web browser or even a much simpler program loading im…

Also that 2000 request per second has to happen 24/7 not only quick demo session.

Re: MangaDex infrastructure overview

#60

Post by chairmanwow1 »

trampi wrote: post #39
latch wrote: post #19

I've done things at scale (5-10K req/s) on a budget ($1000 USD) and I've done things at much smaller scales that required a much larger budget. _How_ you hit scale on a budget is one part of the equation. The other part is: what you're doing. Off the top of my head, the "how" will often involve the following (just to list a few): 1 - Baremetal 2 - Cache 3 - Denormalize 4 - Append-only 5 - Shard 6 - Performance focuse…

Reads like a small excerpt out of "Designing Data-Intensive Applications" :)

This is an amazing book that improved my effectiveness as an engineer by an undefinable amount. Instead of just randomly picking components for a cloud application, I learned that I could pick the right tools for the job. This book does a really good job communicating the trade-offs between different designs and tools.
Post reply on HN