We reduced our server costs by moving away from AWS

>This even covers basic things: for example, look at all of the reports of Hetzner customers losing data due to something like a drive failure. It's not like there isn't a way to deal with that yourself but now you're in the business of monitoring the drives, setting up software RAID and testing/benchmarking it to make sure your configuration is robust, scheduling downtime to reflash servers with storage updates following some process which depends on the hardware & software combo you're using, etc. Your competitor using AWS isn't paying their sysadmins to do that because they do it for you, and it's completely transparent.

If you're using dumb boxes to deploy something like a k8s cluster why do you care about drive failure ? You rent out an extra machine to account for the downtime (and peaks) and you're still well below AWS infra cost.

>Similarly, the major cloud providers can live migrate VMs off of failing hosts without your awareness — another thing you have to build and operate yourself on bare metal which isn't trivial to get right. When Spectre et al. came out, the major cloud providers' customers were all patched on the day of the annoucement. Bare metal hosting users had to schedule downtime, roll patches, and make sure nothing broke.

Again cattle not pets.

>say you went all-in on Hetzner

But that's what I am saying - going "all in on Hetzner" doesn't really mean much if you're using dumb blocks available elsewhere (plain container/VM hosting, network object storage).

>This is the opposite of my experience: with a few exceptions, you're looking at significantly more work to build an equivalent service yourself, especially if you need to worry about reliability, security, etc. That last part is important if you need to be able to make strong statements about who has access to data, whether logging can be tampered with, etc. — those are all things you _can_ setup yourself but the cost of doing so is greater than years of your usage until you're at a pretty large scale.

That's the problem I guess - I haven't built a on-prem system in 10 years now, maybe I forgot the pain points, maybe the automation tools really don't work outside of big cloud providers.

But right now all I'm seeing is insane margins on cloud services and we still have a bunch of devops that are busting their ass off to monkeypatch everything together.

> 3 people at 0.10% of their time is already 24K, assuming an 80K salary for each of you.

which is less than 80k, yes.

> You don't mention patching systems, or the time spend replacing the hardware racks every x years. It is very easy to underestimate the cost of maintenance.

Please stop assuming stupid shit, ask for clarification if you're not sure what I included in it.

Racking 2 racks took us 2 days, one on site, on on configuration.

Patching is also included in that 10%. We track that time. If you're patching manually you already fucked up, cloud or no cloud. Only time spent actually patching is "difficult" customers that need paperwork for taking down even one server of HA pair.

And it's utterly ignorant to assume cloud somehow saves you on this point, you just have more time used dealing on security stuff on dev side instead of ops side. Hell, I dare to say average VM with auto-updates will be more secure than average container deployment. Or maybe that's our devs that didn't get kicked enough to set their container pipeline correctly...

You can hardly make quantitative statements about how many applications need this or that. It's not like we're in the 60s and IBM can tell that they'll need to make exactly seven of a certain computer.

I can tell you that massive amounts of storage are used by eg Mercedes Benz during simulations they need when developing cars. Back when I was reading about this - maybe a decade ago - they were using mind boggling amounts of storage. Still all before SSDs. You can assume all other world-class engineering firms in every kind of difficult engineering that requires physical robustness or safety have a similar need. But Mercedes Benz at least used an on-premises solution, simply because storing stuff on AWS would have created latency and throughput issues.

> labyrinthine bureaucracy

This is a reference to "the rule of the desk" [0], and "an elaborate, confusing structure", which centers on a monster [1], in this case a bureaucrat at a desk

[0] https://www.workhuman.com/blog/a-brief-history-of-bureaucrac...

[1] https://en.wikipedia.org/wiki/Labyrinth

> Once you get to a certain size I think you can attract a team who can build out rather than buy in, and in so doing reduce costs.

I don't think this true for the vast majority of applications. It's probably true for the Youtube, Netflix, OneDrive, etc critical data path, but when your ownership is more like two dozen applications ranging from tiny (one consumer on some infrequent interval) to mid sized (under 100k requests a minute, for text, json, images, generated UI components), it's damn near impossible to beat from a pricing standpoint.

Going back to on prem servers, even doing kubernetes/docker orchestration would likely increase my current team's monthly capital expenditure from ~$14000 a month to somewhere in the realm of 50-60k a month. The sad thing about those prices are that our on prem offerings are very cheap compared to the last price I worked, a $BIGBANK that has multiple data centers in multiple countries. As an internal client with the same footprint we have now, $BIGBANK global infrastructure folks provisioning would take $90-110k and 6 to God even knows how many months depending on how fast you could get through the churn of governance, compliance, security, and controls.

Edit: just realized I forgot to mention I'm talking about serverless projects almost exclusively. That's important to the cost savings, because I agree that I'd rather host something in the 9 year old server in my basement and run my own reverse proxy than use EC2/EKS from a cost perspective.