MangaDex infrastructure overview

Does image processing, runs our analytics, runs our Sentry, runs our gitlab-ci runners, and quite a few other things not mentioned expressly

> which would be extreme overkill

That's an interesting argument against k8s ; if anything I find it much easier to work with -- once accustomed to its idioms, ofc -- than alternatives like dedicated VMs, Docker Swarm etc

Getting HA and auto-healing for free is possible without it, of course, but does require much more work, especially if you aim for a somewhat minimised amount of statefulness (as in deviation from the template of your system)

Also S3-compatible storage backends are really aplenty, from commercial offering to simpler ones like MinIO. Ceph just happens to be a bit higher of a deployment investment with the benefit of fantastic performance, flexibility and resiliency. Somewhat like k8s itself, it's a bit daunting at first but does actually make things simpler in the long run (imo)

> 18k metrics/samples [...] are nothing

Well yes and no, the number of metrics isn't relevant per se, but its cardinality is very relevant, and managing that in a single prometheus instance will quickly require some serious vertical scaling, especially if you want to look at data on longer ranges (which, in contrary to logs, we are interested in)

> 7k logs per second [...] are nothing

That's an interesting take . Surely this isn't a world-record-shattering amount indeed, but no one seems to have such a great non-SaaS-or-cheap solution to storing, sorting and querying this amount of logs either (at the resource efficiency of Loki anyway), so maybe we just have a different set of expectations for log management

> If you don't like a pull-based architecture [...] why use one at all!? There are many more push-based setups out there that are simpler to set up and less complex.

Are there really? That is non-SaaS and with as widespread 3rd-party software support as Prometheus does? ie great integration with essentially any database, webserver, runtime, OS, etc?

Because if we talk only about node metrics like CPU etc then yeah, sure there are plenty of options. But (maybe not so) obviously the diagram showing only node exporter doesn't mean that this is the only integration we use -- we collect prometheus metrics for MySQL, PHP-FPM, Varnish, Nginx, HAProxy, Elasticsearch, Redis, RabbitMQ etc (essentially every single piece of software we use).

Fwiw I found very little in the way of open-source solutions to that problem that ticked as many boxes as Prometheus.

As for "simpler to set up and less complex", both Cortex and Loki would be really annoying to manage outside of Kubernetes, I'll happily give you that. But... being able to easily deploy and manage such systems once you have Kubernetes is precisely one of the reasons to use it. You can't say it's complex to deploy itself but then ignore the fact that it largely outweighs this by making reliable operation of complex-but-powerful software on top it, that is precisely one of the upsides of using it in the first place :)

> Does image processing, runs our analytics, [...]

Fair enough, I was strictly going by the diagrams. From my experience with a somewhat similar setup (HA Loki, HA Prom + Thanos with a MinIO storage backend using Terraform + Ansible and docker) I have to say that the most complex and frustrating part was configuring Loki (this was way before they expanded their documentation, which still isn't great). I'd imagine this would be even more challenging under k8s at least if you stray from the vanilla deployment and/or charts. I agree with your statement regarding Ceph, we use it extensively in production (probably on a much bigger scale). However, I think Ceph, unlike MinIO, just adds unnecessary complexity to your setup.

> Well yes and no, the number of metrics isn't relevant per se, but its cardinality is very relevant [...]

Cardinality is something you should avoid when using Prometheus - for exactly that reason. There are, in my opinion, very few good reasons for dynamic labels (ignoring the baked-in cardinality from a setup like k8s). On first impulse I'd say you're doing metrics wrong but then again, I do not know enough about your use case. Maxing out a single Instance of Prometheus is no easy feat however, especially if your infra isn't that complex and/or big. I've used Thanos for so long now, how does the Cortex compactor handle range queries? Does it also compact and create additional 5m & 1h resolution metrics? These might help with your larger range queries.

Just out of curiosity, have you had any look at alternatives like Victoriametrics?

> 7k logs per second [...] are nothing

My remark was just regarding the added complexity as this depends solely on the size of your log messages. If you don't need or use the (awesome!) capabilities of Loki + Grafana and just need a place for long-term storage of your logs, a 'simple' rsyslog server will do just fine.

> we collect prometheus metrics for MySQL, PHP-FPM, Varnish [...]

Many (if not all) of these can be handled by Telegraf or Fluentd plus InfluxDB (not that I'd used that myself, I absolutely love Prometheus and its Eco-system). My tongue-in-cheek comment was mostly about the Prometheus instance you deploy on every server just to scrape metrics locally and remote-write them into Cortex. Why not the more usual setup of (one or more) Prometheus instances scraping their targets and writing to Cortex?

It performs well and doesn't randomly go down when someone posts a link to it on HN or tries to put in bad input.

> I'm sure you can optimize it to be as efficient as you'd like.

Wrong! I've said nothing about optimizing things.

All I'm advocating is simple solutions that are proven to work.

A web server in Go is far from efficient. An optimized server in C/C++ can probably perform 20x better than a Go server. If not more.

However, a web server in Go makes far more reasonable use of system resources to achieve the desired goals. It's also pretty reliable.

> Bespoke, custom solutions that are not the core of the business are not solutions. Repeat that a dozen times.

I don't understand the point of this sentence.

Are you saying that Kubernetes or Elastic Search or AWS or any of the other buzzwords are at the core of their business?

Clearly they are not.

> MangaDex's business is not to be the most efficient website possible. Their business, I assume, is content and features for their users. They pick off the shelf technologies to do it because it's well documented, proven, and most importantly already built.

It's in the interest of their business to lower their cost of operations. Building on a complicated infrastructure when you don't need is incurring a lot of cost. Not just the monthly cost ($1500/mo) but the cost of the staff needed to understand and maintain this infrastructure.

It's not the kind of thing that is easy to maintain.

To be completely frank with you, I myself am not capable of understanding or maintaining such a system. And every company I've been almost had no one who understood how the system really works. Someone set things up sometime by following some tutorials. When things go wrong, people panic and go into fire fighting mode. They spend hours trying to make sense of what's going on, usually involving multiple people - because it's not a task that a single individual can handle.

> They compose these technologies to solve their business problems. Often there's a mismatch or an overlap in functionality that introduces inefficiency (complexity, cost, performance), but that's a trade-off MangaDex and many other businesses make. We can judge how poorly or well they've made some trade-offs base on their business and overall requirements.

You are talking as if these off the shelf technologies are reliable and easy to implement or integrate.

From what I've seen, these solutions are a lot more complicated than what I'm proposing.

Every place I've been to that tries to take this approach ends up burning too much money and resources trying to make their thing work.

It's not as if these companies don't have to write code to make their product work. You still have to write code anyway. So, why not, instead of writing tons of glue code and configuration files to hopelessly integrate a hodge podge of tools and frameworks ... why not just write the simple code that just does the thing you want?

> I'm going on and on in this thread mainly because I'm tired of people assuming they know better than the people in the trenches making these decisions. You're assuming you're more knowledgable and skilled than them in their own problem space! They're obviously unaware that using Go to make a server would solve their problems otherwise why wouldn't they have done it?

The first company I've been to that was doing this kind of thing was spending upwards of $10k/mo on the most beefed up server that AWS provides to host the database server, and they still struggled to server more than 1000 users concurrently.

According to you, I'm not in a position to give them suggestions or adivce about how to fix this problem!!

> I'm sorry, but who thinks only serving 2000 requests per second is hard? Or do you assume they think it's hard because you misunderstood or are unaware of their 100s of other requirements that they need to solve for in addition to serving 2000 requests per second?

What are the other 100 requirements that are not fulfilled by the thing I'm proposing?!