Live data from Hacker News

MangaDex infrastructure overview

mangadex.dev

211–220 of 241 posts

Re: MangaDex infrastructure overview

#211
post #209
post #208

Earlier quoted context omitted.

it's timing based. you can always reply by going to the permalink of the comment you want to reply to.

Couldn't reply to this comment - but sure enough, the permalink gives me the option. Thank you for the info!

Yeah, it's a somewhat well-meaning feature (supposed to slow down flamewars) that is extremely unintuitive

Re: MangaDex infrastructure overview

#212
post #195
post #152

Earlier quoted context omitted.

>A fresh load of the home page generates over 100 requests. I see 17 requests, all over either h2 or h3. 4 of them JS, and 2 images.

Then you're not doing a fresh load of the page. There are over 30 images visible on the front page, so your measure doesn't pass the smell test, does it?

>Then you're not doing a fresh load of the page

Nope. Different problem.

The article was linked to a page under the domain "mangadex.dev".

Without any other context, I had assumed "home page" meant http://mangadex.dev , or what I got when clicking "Home" on the linked article.

Apparently not.

Re: MangaDex infrastructure overview

#213
post #133
post #45

Earlier quoted context omitted.

Okay, but isn't most of their content stolen? Why would you want to contribute to that?

Just curious if anyone reading this knows the answer: Would it be illegal to contribute man-hours on e.g. implementing features or fixing bugs on a project like this, or does that only apply to whoever actually hosts the content?

A good lawyer would probably say something like "it depends".

It's entirely possible for a copyright owner to construe some kind of secondary liability based on your conduct, even if the underlying software is legal. This is how they ultimately got Grokster, for example - even if the software was legal, advertising it's use for copyright infringement makes you liable for the infringement. I could also see someone alledging contributory liability for, say, implementing features of the software that have no non-infringing uses. Even if that turned out to ultimately not be illegal, that would be at the end of a long, expensive, and fruitless legal defense that would drain your finances.

In other words, "chilling effects dominate".

Re: MangaDex infrastructure overview

#214

Earlier quoted context omitted.

You're basically dumping down a database to the web browser, including all of the internal metadata that's likely irrelevant to rendering the HTML. For example, user role memberships: { "id": "c80b68c5-09ae-4a50-a447-df7c5a4a6d01", "type": "user", "attributes": { "username": "kinshiki", "roles": [ "ROLE_MEMBER", "ROLE_GROUP_MEMBER", "ROLE_POWER_UPLOADER" ], "version": 1 } } Also record timestamp dates like created/ch…

As I said, it's not so much that we ask that data to be fetched -- it is there in the first place, and pulled from Elasticsearch, not a SQL database Because of this model, we also make sure that Elasticsearch merely works a search cache, not as an authoritative content database (hence everything we add in there is considered public, on purpose, and what isn't meant to be public is just not indexed in ES) However the…

I have to say I'm glad this is being talked about in a public forum. Outsiders rarely get to see brainstorming, troubleshooting & group discussion of technological issues like this.

Someone who is focused on the performance aspect & someone who is focused on stack stability discussing the real world input & output of a business system and showing why performance & UX are not the only metrics that matter is a good thing for us to see.

Re: MangaDex infrastructure overview

#216

Earlier quoted context omitted.

Yeah. This whole mess pushed me to moving everything I had (or could remember, anyway) to Tachiyomi¹, so I can hop between hosting websites freely without losing progress or access to old chapters (as long as I don't run out of local storage). And while it works fine for reading, it kills any interaction with the hosting sites. No chance for monetization, socialization or anything else that can help sites survive lon…

Totally a self plug, but if you're looking to take it a step further, Kavita is a great program to host your own, Plex-like manga server. https://kavitareader.com

That actually looks really interesting, thanks!

Re: MangaDex infrastructure overview

#217
post #52
post #35

Earlier quoted context omitted.

I think enterprise and more optimize for business flexibility and ability to A/B test very rapidly vs a finely crafted piece of efficiency, for better or worse. The people behind this probably do this for their day job, or are teens that are about to do it for their day job.

I agree with you. I mostly work in enterprise and understand that it has different needs and ROI requirements. However, my personal mindset is that computers and networks are really really fast now and it's a tragedy that most of these gains are nullified due to unoptimized layers of abstraction or over-architecting. So it's a welcome sight to read about well-designed infrastructure like this.

It happens because business are optimizing for resources that are ultimately more expensive or slower, which is staffing levels and the ability to respond to the market so the business can grow or survive longer. Inefficient computing architecture as a side effect is a worthwhile tradeoff in light of that to them.

But as a craftsman, it is definitely nice :)

Re: MangaDex infrastructure overview

#218
post #205
post #48

Earlier quoted context omitted.

I think even a distributed cache in front of a database shouldn't have any trouble handling 2000 requests per second. The issue is not really the number of requests per second, probably, but the number of bytes, which they don't talk about at all in the article; reading manga with no ads is a pretty static kind of application, which could be satisfied amply with a web browser or even a much simpler program loading im…

> distributed cache in front of a database Already an over kill. Think smaller. Think simpler. A single machine serving files directly from the file system (yes, from the SSD attached to the machine) will be able to handle a LOT more.

Well, that's what httpdito does: it serves files directly from the filesystem. That's why I mentioned it. But, for some applications, such as the website you're using right now, it's useful to display pages that haven't been previously stored in the filesystem.

Re: MangaDex infrastructure overview

#220
post #80

Earlier quoted context omitted.

> Never ‘sharded’ before, no idea how that works. Sharding sucks, but if your database can't fit on a single machine anymore, you do what you've got to do. The basic idea is instead of everything in one database on one machine (or well redundant group of machines anyway), you have some method to decide for a given key what database machine will have the data. Managing the split of data across different machines is, o…

> you have some method to decide for a given key what database machine will have the data Super simple example, splitting there phone book into two volumes, A-K and L-Z. (Hmmmm, is a "phonebook" a thing that typical HN readers remember?) > you can fit a lot of database in 8 TB of ram, and if you don't keep the whole thing in ram, you can index a ton of data with 8 TB of ram, which means sharding can wait. For almost…

> So maybe 12 months worth of a senior engineer's time.

In America. When the salaries are 2/3 times lower, people spend more time to use less hardware.

Post reply on HN