Like 1/3 to 1/2 of the problems in this list are just problems with using AWS or other cloud providers. I say this all the time, and nobody takes it seriously: Don't use the cloud. The cloud costs more, requires more work to set up, and provides you with machines with broken IO and 1/50th the capacity you would get if you just racked servers yourself.
Let me say it again:
Compared to buying servers from Dell (or whoever) and driving them down to the local COLO and plugging them in yourself, the cloud:
- Costs more (between 5x and 20x more over the course of a 4 year depreciation for hardware).
- Is more work in the end, by a large factor (you are going to do a ton of stuff you don't need and are never going to use with the cloud. When something goes wrong you are going to spend days or months trying to fix it. Performance is so bad that you have to build very complex solutions to problems that it is very easy to just throw hardware at, like having a db server with millions of IOps, which is extremely easy if you rack hardware and basically unreasonable due to cost and hard limits in the cloud (Well over $10K/month for Postgres at 80,000 iops, which is the most you can provision).
- It's more work up front, even if you have no idea how to do anything to start with. For your deployments to be secure you have to know what you are doing either way, in AWS you just have less choices of how to set things up. Most things in AWS are at least as hard or harder than doing them yourself, and you can't fix problems you run into. Their tier 1 and tier 2 support are completely useless and just do keyword matching against their scripts.
- It costs WAY more. You pay for more than the cost to buy much better hardware and the cost to colo that hardware every 4 months. Yes that is right, if you go buy a pile of servers from dell and rack them in a colo, or buy the same capacity from EC2, the EC2 capacity you get will be far lower. Far far far lower. And you can't customize it or redeploy it for other uses as it gets older.
- When things break, you can't fix them. You just have to live with it or stop caring. IO to disk stalls randomly leaving processes in uninterruptible IO wait status for 100ms? Get fucked, they not only won't admit it's a problem, they will also hide this fact from you in all their cloudwatch metrics, and deny it is happening. Maybe it's a noisy neighbor, maybe it's their terrible networking stack, who knows, you'll never figure it out and you probably can't fix it even if you do. I have weird failure scenarios in AWS all the time that I have never seen or even contemplated as reasonable to consider on my own hardware.
- Cloud deployments are stupid. K8 is an awful system built by children who don't what is important. I don't even directly deal with K8 as my actual job, but I spend more time worrying about K8 to get code shipped today than I ever spent maintaining deployment scripts when I had to own deployments end-to-end.