Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MangaDex infrastructure overview

mangadex.dev

Post reply on HN
Print view View original

161–170 of 241 posts

Re: MangaDex infrastructure overview

#161

Post by manigandham »

vymague wrote: post #153
maxk42 wrote: post #145

Earlier quoted context omitted.

(1) Simple beats complex. (2) You can spend weeks building complex infrastructure or caching systems only to find out that some fixed C in your equation was larger than your overhead savings. In other words: Measure everything. In other other words: Premature optimization is the root of all evil. (3) Fewer moving parts equals less overhead. (Again: Simple beats complex.) It also makes things simpler to reason about.…

> But for the frameworks that aren't doing those things, the benchmark is solid. Any example of such frameworks?

(ASP).NET is solid. Extremely fast, very reliable, and highly productive.

https://dotnet.microsoft.com/apps/aspnet

Re: MangaDex infrastructure overview

#162

Post by bytearray64 »

ApolloFortyNine wrote: post #51
bytearray64 wrote: post #14

Earlier quoted context omitted.

You probably have Verizon - they've started null routeing traffic to sites "like this". https://old.reddit.com/r/mangadex/comments/nvj7qf/is_verizon...

Ah shucks I thought we had mostly avoided that stuff in the US. I'm guessing though they're using some old spam ip/block though, there's a lot more obvious piracy sites then a Manga site. For instance, I can access all the major torrent sites.

They also block nyaa, an Anime/Manga focused tracker. It's not a very aggressive list though, as you're right that major torrent sites are still accessible.

Re: MangaDex infrastructure overview

#163

Post by hnlmorg »

nrabulinski wrote: post #160
hnlmorg wrote: post #146

Earlier quoted context omitted.

Privacy reasons? It's all static content that is publicly accessible. I don't understand what the privacy reasons could be under this context. Are they worried about CDNs logging the images their visitors access? Seems like an absurd edge case to worry about in my opinion. > however, those options are unlikely to be free I wasn't even talking about free CDNs :)

They’re basically hosting illegal content, or at least a good chunk of it is copyright-infringing so they cannot use cloudflare or any of the other off the shelf offerings

I see. That does complicate things somewhat.

I wonder if there's merit in them approaching studios with a proper business plan?

Re: MangaDex infrastructure overview

#164

Post by manigandham »

ctvo wrote: post #159
hsn915 wrote: post #89

I don't understand. Why is 2k requests/sec supposed to be massive? Try this yourself: write a simple web server in Go, host it on a cheap VPS provider, let's say at the option that costs $20/mo. Your website will be able to handle more than 1k/s requests with hardly any resource usage. ok, let's assume you're doing some complicated things. So what? You can scale vertically, upgrade to the $120/mo server. Your website…

> Try this yourself: write a simple web server in Go, host it on a cheap VPS provider, let's say at the option that costs $20/mo. Your website will be able to handle more than 1k/s requests with hardly any resource usage. These people have never heard of Go, obviously. The likely scenario is not that you haven't fully understood their constraints or requirements, it's that you're just smarter than they are. > So what…

There are plenty of people who build here on HN (more than most other sites) and the requirements are pretty clearly described in the article.

While it's not as simple as a Go program on a VPS, there is certainly a lot of unnecessary overhead here. I think you underestimate just how much poor and wasteful engineering there is out there.

Re: MangaDex infrastructure overview

#165

Post by arthur_sav »

maxk42 wrote: post #145
maxk42 wrote: post #132

Earlier quoted context omitted.

My approach to what?

(1) Simple beats complex. (2) You can spend weeks building complex infrastructure or caching systems only to find out that some fixed C in your equation was larger than your overhead savings. In other words: Measure everything. In other other words: Premature optimization is the root of all evil. (3) Fewer moving parts equals less overhead. (Again: Simple beats complex.) It also makes things simpler to reason about.…

> Simple beats complex. > Fewer moving parts equals less overhead.

Took me almost a decade to really comprehend this.

I used to include all sorts of libraries, try out all the fancy patterns/architectures etc...

After countless of hours debugging production issues... the best code i've ever written is the one with the fewer moving parts. Easier to debug and the issues are predictable.

Re: MangaDex infrastructure overview

#166

Post by bob1029 »

nine_k wrote: post #99
bob1029 wrote: post #86

Earlier quoted context omitted.

>20k RPS. If this metric is what you are chasing, there are ways to reliably break 1 million RPS using a single box if you don't play the shiny BS tech game. The moment you involve multiple computers and containers, you are typically removed from this level of performance. Going from 2,000 to 2,000,000 RPS (serialized throughput) requires many ideological sacrifices. Mechanical sympathy (ring buffers, batching, minim…

I frankly don't see where containers could lower the performance. Basically a container is a glorified chroot. It has the same networking unless you asked for isolation, then packets have to follow a local (inside the host) route. It has exactly no CPU or kernel interface penalty. Maybe you wanted to say about container orchestration like k8s, with its custom network fabric, etc.

Even loopback is significantly slower than a direct method invocation.

Re: MangaDex infrastructure overview

#167

Post by arethuza »

maxk42 wrote: post #145
maxk42 wrote: post #132

Earlier quoted context omitted.

My approach to what?

(1) Simple beats complex. (2) You can spend weeks building complex infrastructure or caching systems only to find out that some fixed C in your equation was larger than your overhead savings. In other words: Measure everything. In other other words: Premature optimization is the root of all evil. (3) Fewer moving parts equals less overhead. (Again: Simple beats complex.) It also makes things simpler to reason about.…

"Simple beats complex."

In the very first lecture of the Computer Science degree I did in the 1980s the lecturer emphasised KISS, and said that while we almost certainly wouldn't believe it at first eventually we'd realise that this is the most important design principle of all. Probably took me ~15 years... ;-)

Re: MangaDex infrastructure overview

#168

Post by kiba »

arthur_sav wrote: post #165
maxk42 wrote: post #145

Earlier quoted context omitted.

(1) Simple beats complex. (2) You can spend weeks building complex infrastructure or caching systems only to find out that some fixed C in your equation was larger than your overhead savings. In other words: Measure everything. In other other words: Premature optimization is the root of all evil. (3) Fewer moving parts equals less overhead. (Again: Simple beats complex.) It also makes things simpler to reason about.…

> Simple beats complex. > Fewer moving parts equals less overhead. Took me almost a decade to really comprehend this. I used to include all sorts of libraries, try out all the fancy patterns/architectures etc... After countless of hours debugging production issues... the best code i've ever written is the one with the fewer moving parts. Easier to debug and the issues are predictable.

"The best part is no part." is an engineering quote I heard.

Re: MangaDex infrastructure overview

#169

Post by Cipater »

ignoramous wrote: post #32
latch wrote: post #19

I've done things at scale (5-10K req/s) on a budget ($1000 USD) and I've done things at much smaller scales that required a much larger budget. _How_ you hit scale on a budget is one part of the equation. The other part is: what you're doing. Off the top of my head, the "how" will often involve the following (just to list a few): 1 - Baremetal 2 - Cache 3 - Denormalize 4 - Append-only 5 - Shard 6 - Performance focuse…

> Twitter's a good example. Mike Cvet's talk about Twitter's fan-in/fan-out problem and its solution makes for a fascinating watch: https://www.youtube-nocookie.com/embed/WEgCjwyXvwc

I appreciate the no-cookie embed.

Learned something new today.

Re: MangaDex infrastructure overview

#170

Post by manigandham »

bawolff wrote: post #25
urlgrey wrote: post #6

I'm amazed that their architecture doesn't include a CDN. These days I expect nearly all high traffic websites to make use of a CDN for all kinds of content, even content that's not cached. They cited Cloudflare not being used due to privacy concerns. It'd be interesting to hear more about that, as well as why other CDNs weren't worth evaluating too.

What's the benefit of a cdn if nothing is cacheable? Slightly lower latency on the tcp/tls handshake? That seems pretty insignificant.

Latency makes a bigger impact on UX than throughput for general browsing. A TLS handshake can be multiple roundtrips that greatly benefit from lower latency, especially mobile devices.

Modern CDNs also provide lots of functionality from security (firewall, DDOS) to application delivery (image optimization, partial requests).

Post reply on HN