Earlier quoted context omitted.
it's timing based. you can always reply by going to the permalink of the comment you want to reply to.
Couldn't reply to this comment - but sure enough, the permalink gives me the option. Thank you for the info!
MangaDex infrastructure overview
211–220 of 241 posts
Re: MangaDex infrastructure overview
#212Earlier quoted context omitted.
>A fresh load of the home page generates over 100 requests. I see 17 requests, all over either h2 or h3. 4 of them JS, and 2 images.
Then you're not doing a fresh load of the page. There are over 30 images visible on the front page, so your measure doesn't pass the smell test, does it?
Nope. Different problem.
The article was linked to a page under the domain "mangadex.dev".
Without any other context, I had assumed "home page" meant http://mangadex.dev , or what I got when clicking "Home" on the linked article.
Apparently not.
Re: MangaDex infrastructure overview
#213Earlier quoted context omitted.
Okay, but isn't most of their content stolen? Why would you want to contribute to that?
Just curious if anyone reading this knows the answer: Would it be illegal to contribute man-hours on e.g. implementing features or fixing bugs on a project like this, or does that only apply to whoever actually hosts the content?
It's entirely possible for a copyright owner to construe some kind of secondary liability based on your conduct, even if the underlying software is legal. This is how they ultimately got Grokster, for example - even if the software was legal, advertising it's use for copyright infringement makes you liable for the infringement. I could also see someone alledging contributory liability for, say, implementing features of the software that have no non-infringing uses. Even if that turned out to ultimately not be illegal, that would be at the end of a long, expensive, and fruitless legal defense that would drain your finances.
In other words, "chilling effects dominate".
Re: MangaDex infrastructure overview
#214Earlier quoted context omitted.
You're basically dumping down a database to the web browser, including all of the internal metadata that's likely irrelevant to rendering the HTML. For example, user role memberships: { "id": "c80b68c5-09ae-4a50-a447-df7c5a4a6d01", "type": "user", "attributes": { "username": "kinshiki", "roles": [ "ROLE_MEMBER", "ROLE_GROUP_MEMBER", "ROLE_POWER_UPLOADER" ], "version": 1 } } Also record timestamp dates like created/ch…
As I said, it's not so much that we ask that data to be fetched -- it is there in the first place, and pulled from Elasticsearch, not a SQL database Because of this model, we also make sure that Elasticsearch merely works a search cache, not as an authoritative content database (hence everything we add in there is considered public, on purpose, and what isn't meant to be public is just not indexed in ES) However the…
Someone who is focused on the performance aspect & someone who is focused on stack stability discussing the real world input & output of a business system and showing why performance & UX are not the only metrics that matter is a good thing for us to see.
Re: MangaDex infrastructure overview
#215It's basically pirating content
Re: MangaDex infrastructure overview
#216Earlier quoted context omitted.
Yeah. This whole mess pushed me to moving everything I had (or could remember, anyway) to Tachiyomi¹, so I can hop between hosting websites freely without losing progress or access to old chapters (as long as I don't run out of local storage). And while it works fine for reading, it kills any interaction with the hosting sites. No chance for monetization, socialization or anything else that can help sites survive lon…
Totally a self plug, but if you're looking to take it a step further, Kavita is a great program to host your own, Plex-like manga server. https://kavitareader.com
Re: MangaDex infrastructure overview
#217Earlier quoted context omitted.
I think enterprise and more optimize for business flexibility and ability to A/B test very rapidly vs a finely crafted piece of efficiency, for better or worse. The people behind this probably do this for their day job, or are teens that are about to do it for their day job.
I agree with you. I mostly work in enterprise and understand that it has different needs and ROI requirements. However, my personal mindset is that computers and networks are really really fast now and it's a tragedy that most of these gains are nullified due to unoptimized layers of abstraction or over-architecting. So it's a welcome sight to read about well-designed infrastructure like this.
But as a craftsman, it is definitely nice :)
Re: MangaDex infrastructure overview
#218Earlier quoted context omitted.
I think even a distributed cache in front of a database shouldn't have any trouble handling 2000 requests per second. The issue is not really the number of requests per second, probably, but the number of bytes, which they don't talk about at all in the article; reading manga with no ads is a pretty static kind of application, which could be satisfied amply with a web browser or even a much simpler program loading im…
> distributed cache in front of a database Already an over kill. Think smaller. Think simpler. A single machine serving files directly from the file system (yes, from the SSD attached to the machine) will be able to handle a LOT more.
Re: MangaDex infrastructure overview
#219Re: MangaDex infrastructure overview
#220Earlier quoted context omitted.
> Never ‘sharded’ before, no idea how that works. Sharding sucks, but if your database can't fit on a single machine anymore, you do what you've got to do. The basic idea is instead of everything in one database on one machine (or well redundant group of machines anyway), you have some method to decide for a given key what database machine will have the data. Managing the split of data across different machines is, o…
> you have some method to decide for a given key what database machine will have the data Super simple example, splitting there phone book into two volumes, A-K and L-Z. (Hmmmm, is a "phonebook" a thing that typical HN readers remember?) > you can fit a lot of database in 8 TB of ram, and if you don't keep the whole thing in ram, you can index a ton of data with 8 TB of ram, which means sharding can wait. For almost…
In America. When the salaries are 2/3 times lower, people spend more time to use less hardware.