> but the tools that have been brought into the mainstream like Kubernetes
unless you are running real steel, kubernetes is at best a complication layer over the underlying cloud tools. In 2018 me and my team built a real time, ML-based processing API. It was hugely scalable, optimised for speed, price and uptime.
At no point did we use need to use kubernetes. Not only would that have eaten most of our innovation budget, all it would have done is add latency and driven up the money we spent on inter VPC transfers.
> I remember the days of Nagios, Zabbix, SolarWinds etc
So do I, they were recognised as shit even in 2015. $work allowed me to lead the migration away from them and by then we were half way through replacing most of splunk with graphite/grafana based metrics instead. The rest of the alerting was mostly handled by a custom inhouse /__status end point parser.
> The previous iteration of platform at this company was all self-managed VMs with the old school sysadmin mindset. They were managing hundreds of VMs by hand or using some light ansible that was retrofitted to work with pre-existing machines, we were using corporate F5 load balancers and VMware which are immediately out of date the moment you deploy them.
That, again, is just bad sysadmin. Unless you have complete control over your stack with ansible, then its not ansible, its just sparkling bash script.
I worked at a VFX company that deployed to all real machines for virtually everything. They used a GNU configuration system that was older than SSH (I can't remember what its called, it used to use rsh, that's how old it was.) All the machines were fungible, We could replace a file server at the drop of a hat (the only limit was how fast we could copy 160tbs over 2x10gig nics) Most of the place ran on perl, in one git repo for just configuration. OS updates were applied weekly, and we pushed system changes daily. This was in 2013.
salt made all of that easier, especially as we had way too many hosts to sensibly run ansible at the time(I think we worked out it would take >1day to run a basic role).
They had automated it to the point that you could plug in an unconfigured blade host, turn it on, and within 30 minutes, the bios was updated, networking configured and each blade was starting to be installed.
Automation is an attitude. You can either invent new tools to solve hypothetical problems badly (k8s supposedly is high scale, but cant practically get above 1k hosts, its also balls deep in ipv4 ), or actually use/create tools to solve the problems you have now.