> We ended up porting everything to k8's. That was a long & arduous process but it gave us total flexibility at significant cost savings. The benefits were great, but not everyone has access to the engineers/skillset needed to be successful with k8's.
I think the argument from most people advocating against using k8s from the get go isn't so much that you'll never need it, but more that it's better to pay this cost later when you have a demonstrated problem that needs to be solved, even if it's a bit more expensive (and I would bet that the total time expenditure isn't really all that much more moving to k8s later, it's just in one chunk instead of spread over your history).
By deferring the cost of building more complex infrastructure:
- You can use those resources to advance your product (arguably some companies that have hit a wall with PaaS vendors may never have gotten to the point where they outgrew those vendors if they spent more of their time on infrastructure vs. company value)
- You'll have a better idea of what you'll need and where your scaling hotspots are going to be if you've already encountered them. Building infrastructure that allows for flexibility in a few key areas is infinitely easier than building infrastructure that can scale in every way imaginable.
- You can potentially take advantage of new technologies if you defer the decision to when you need it. It would be silly to assume k8s is the final word. If you wait a few years, things will have evolved and you gain the advantage of using those few years of advancement rather than cementing a choice early on.