I've always been hesitant about adopting frameworks, and I still favor Flask over Django. Yet, I spent a couple of years in a Go-centric department where the senior members were so against using frameworks that we ended up building everything ourselves from scratch. This included creating our own load-balancer, event store, and aggregation engine. When I attempted to point out that we were investing substantial time…
The thing I've seen the most in this industry is the opposite of that: teams that would mostly use third-party tools and frameworks, but the result was effectively the same, with complexity exploding and becoming so unbearable that the teams inflated to compensate, with productivity grinding to a halt. It happens with third-party tools as often as it does with NIH.
Like you say, Kafka, Kubernetes and Spark would add a higher-degree of complexity. They aren't really that simple to manage in real world production environments. That's the problem here: the complexity. It doesn't matter where it comes from, it will bite you in the ass.
One extreme of that is off-the-shelf enterprise software, that often requires a team of consultants to integrate. I've seen a few disaster, one software specifically started with a 500.000 price, but things became so complex the project ended up costing 4x that. There is no panacea against complexity. It costs money and takes time.
The main problem here seems to be the "questionably designed and lacks readily available support" part, which is something universally bad, even when using third-party software. You can design your infrastructure badly. You can make questionable design using third-party tools just as much as you can with your own code. And support is also very often not readily available.
"Similarly, if everyone agrees on using Kubernetes and Docker"
I'm totally sympathetic to things becoming standards, and I don't have an axe to grind with those two tools, but the reason for the pushback against tools like Kubernetes and Docker is grounded on reality. Not only they introduce complexity by themselves, they are often just band-aids for accidental complexity introduced by teams and by third-party things like programming languages and frameworks.