Move Back to Datacenters

In my experience small firms will struggle to hire excellent system engineers that can manage a bare metal setup, it's easier however to hire okay-ish SWE's that can design applications appropriate for cloud.

I think people overestimate the expenses related to maintaining their own infrastructure. I work for a large org that does both. We have a lot of workloads running in the public cloud, and many more on premises (for various reasons, including compliance).

Having worked on these for years, building solutions, diagnosing problems and so on I'd risk saying we have reached the point where the cloud is slowly getting more complex than traditional setups. In theory, it should be simpler, right? A Transit Gateway is a virtual device so easier to configure than a Cisco router, right?

The problem is, as the cloud providers offering gets richer, with time it has to be more complex in response to various customers' requirements, so basically you need to learn both: traditional Linux operations and networking, and a whole new class of interactions between dozens of services with hundreds of API calls. It quickly becomes overwhelming. IaC obviously helps but anybody who has maintained a large repository of CloudFormation code will tell you its not a panacea.

And then a million small quirks that basically come from you not owning the infra. E.g. Amazon is pushing the Control Tower as a model for new setups especially for larger orgs. Guess what, this thing has almost no API. Last time I checked it allows you to maybe check a state of a control. This means you need to set it up manually! In 2023! After 5 years of development! And if you add an OU, you need to register it (you guessed it - also manually!). Say you want to create a dozen of VPC endpoints - they won't have names because there is no API, you need to name them manually. And so on and so forth. Why? because you are living in someone else's place, so you you are totally dependent on them.

I'd say the attack surface on rolling your own tech in a datacenter is substantially higher.

Yes DynamoDB has an API, but I'd wager a AWS engineer with good security skills has looked at it carefully. Do you have an equally skilled security expert on hand to look at the datacenters stack and then same for whatever you're deploying on it?

Not all internet exposure is equal. Moving out of cloud often makes sense, but security isn't the right motivation for it.

The real business case for the cloud is it replaces capex with opex.

Which is great if you have limited capital (bootstrapping, new startup, etc), or are the CEO of a large company that wants to show better cash flow to earn a larger bonus.

In reality, the idea that the cloud somehow means you can outsource your infrastructure team rarely works out in practice.

At best you’ll get much more expensive developers having to spend time on infrastructure which might look like a couple of fewer salaries to pay, but you’re likely now paying for it through dev time anyways.

Would you mind giving an example of something that is easy in cloud but hard on prem (all I can think of is scaling up (down?) which seems kinda obviously hard)