How We Found 7 TiB of Memory Just Sitting Around

Reading comments like yours makes me realise that I should just leave commercial programming and never come back. Your framing is terrifying.

You know why? I am not saying that what you said does not make sense. Of course it does make sense, financially so. But! You the manager one day come to me and my team and say "How could we allow to have 7TB of unused memory sitting around and we paid for it?!" and we'll then have multiple follow-up meetings where we'll be scolded and "trained" how to avoid things like this. We'll get sent articles and told to improve.

And believe me when I tell you, _all_ the techies in these meetings want to roll their eyes through it all. Because many of them likely asked "Can I take a closer look at our infra, it seems expensive and we can potentially optimise it?" and were said no by managers like yourself.

As an engineer you just can't win. So I don't blame myself or any other techie who sometimes goes cowboy mode to find such problems without asking.

Finally, "my contribution for the month" is technological work and nothing else. If I wanted to be a cofounder or have a seat in the board so I have fiduciary duty, I would have said so. It's your job as a manager to put this barrier between stakeholders and front soldiers so the latter can do their thing without disruption, so the organisation can succeed.

Are you doing your job well?

Here's the extent of my interest: I take my understanding of your use case and specifications, then I write source code that tries to generate as few instructions to suit your needs as possible while still being comprehensible to the next maintainer.

The app should write records to a database? Fine. Here's where you configure the connection. The app in production is slow because the database server is weak? Not my problem, talk to your DBA.

The app should expose an HTTP endpoint for liveness probes? Fine. It's served from the path you specified. Your reused it for an external outage check, and that's reporting the service is down because the route timed out due to your ops team screwing up the reverse proxy? Literally not my problem, I could not care less.

DBAs don't necessarily need telemetry in an app to diagnose an issue with the app's behavior. They can run a trace and see some SELECT is running a thousand times a second and deduce that it's being called in a loop over the result set of an earlier query. And they'd be right to say hey, this is an app issue, open a ticket with the developer.

If you put that responsibility on the developer--meaning you expect the dev to diagnose an issue that they introduced in the first place--what kind of result do you think you're going to get?

Layering these demands takes away from the overall quality of the application in my experience. You want an app developer to learn all about Prometheus so the app can have an endpoint with all these custom metrics, okay, and you want structured logging and expect the dev to learn how to use Kibana effectively? All that's a huge cognitive burden that eats a slice of the same pie (their brains) as domain knowledge, language & runtime knowledge, etc.

Get maybe one app developer to specialize, get maybe one app developer to cross-train with ops or monitoring even. But leave most of us out of it.

When you flip that expectation of developer involvement in operations, it exposes how unreasonable that arrangement is. Hey, DBA, the app is sucking up resources. Why don't you crack open an IDE and write a patch for it? What do you mean you don't know Go, what do you mean you don't use Git? Every DBA should know how to attach a debugger to a remote process, shouldn't they?

It's just exploitative. Or at least that's been my experience, so there's my bias.