> This almost is not ever true for software startups, who get more value from speed, flexibility, and not needing to pay a team to manage hardware.
It's not that simple, some costs on AWS can be high and some tasks don't need the availability/reliability guarantees AWS provides. So even for small startups sometimes having own hardware can be a good cost saver, through with some risk attached to it.
For some mid-sized companies with relative fixed resource utilization sometimes renting rack space or renting bar metal (in a server center, with service for parts replacement) can be a good cost saving measure.
In the end while AWS is flexible and convenient with a lot of services easily accessible it is also expensive (at least when it comes to compute power) so the question is most times how much money do the flexible, convenience and services save. For many companies this is a lot. For some companies (of any size) this is not enough.
I mean this is why there are such funny things as being able to buy a "GPU compute min server center in a container" (through that's outside of the price range of an small company, but smaller versions similar to this exist, too).