Earlier quoted context omitted.
Such an AI would be unusable in production because there is no bound on the work that can be created and external side effects that could happen. All projects have real world constraints and side effects, budget, customer requirements, etc. Imagine a non-technical person prompting the AI "make the app faster and fix all the bugs" and the AI autonomously decides to rewrite the whole production app and all its dependen…
You are essentially describing an over-eager developer having to run decisions up the chain of command to senior devs / product owners / company execs. I assure you, every one of those roles is being trained on to balance their own wide breadth of concerns, and is unlikely to advise something like a live rewrite of a production app or an entire product pivot - unless it actually made sense from an economics/managemen…
The reason human judgement is necessary is not because it's "better". It's because the entire reason the business exists is to execute the human's judgement. There is no "right or wrong" way to build a startup if different paths are all profitable, but the human decides what they want to do.
Could Bun have survived and done well even if they didn't rewrite the codebase in Rust? Yes, it would've been fine. But the human decided they wanted Rust, that's a different path they chose with different tradeoffs. The AI has no skin in the game , no ownership, it doesn't belong to the AI. The AI only exists to serve the human. And if the human wants to make good decisions, inevitably they need engineering expertise (or an engineer) to weigh the decision.