I think the idea is you'd basically have it take a look at your current system, it would learn what features you're actually using at all, it'd check company emails for past and current pain points or stuff you wish was possible or just simpler, it'd Slack everyone in the company asking what their biggest wish and biggest pet peeves are currently, it'd do a small interview with Joe himself presenting the above to see if it's gotten the right idea, create a very detailed spec and then implement it.
Of course both models and tooling will need to be far more powerful for all this, but it doesn't exactly seem sci-fi to me.
Once system is built it could run detailed analysis on its usage and figure out what parts seem to be confusing or slow for users, and simply refine, deploy, keep analyzing, rinse and repeat.
The biggest upside is probably that workers could also simply request features, have Joe sign off on them (would get messy otherwise) and minutes later they actually roll out.
To me anyways most systems are a PITA because they do so much and your own organization only utilizes a small subset. Good systems actually let you turn off stuff you don't use so that users don't even know it's possible and don't have to drown in menu options, but that's still rare enough. And good luck getting dev focus on your specific requests regarding the parts of the system most important to your specific company, since there are a zillion other things and hundreds or thousands of other customers.
Something literally tailored to what you need will surely be the norm eventually. In five years or whatever I'm sure we'll be plenty on our way towards something like that.
But again just like LLM training in general this all requires having something existing to analyze and work off of. So yeah nobody will be going from paper to custom agent-built system.