AI won't be put in important positions of responsibility within an organization because AI providers will never accept liability for bad decisions. You can't fire Claude if it fucks up, and it's got very limited ability to learn from its mistakes. It's also incapable of making good decisions where doing so requires synthesizing more than a few hundred thousand tokens worth of domain knowledge/experience in something that doesn't have an infinite amount of synthetic verifiable training data like code and math.
In theory continuous learning (live weight updates) could help to some degree. But there's essentially no progress towards that because it requires solving a few hard, currently completely unsolved problems. 1. Weights drift over time and there's no way to re-merge them after a few tens of thousands of updates, so when a new model version was released there'd be no way to update existing continuously-learned models to that. 2. It'd allow permanent jailbreaking. And 3. A model can't learn new things without forgetting existing things, unlike humans brains which have hardware plasticity (like London taxi drivers having larger hippocampi due to having to memorize so many streets).