Congratulations on an interesting project. There is a lost opportunity with your natural language only approach. The issue is natural language will never be efficient as an interface. Natural language helps with low domain knowledge. That's the plus side as it allows the end user to say a variety of phrases to get the desired result. Commands allow for surgical precision and efficiency/less voice strain for its end user. So there needs to be an approach that allows for both elements natural language and commands. As users develop their own process and workflow they will create actions as commands. (high domain knowledge)
Since these commands are self created by the end user they remember them for their specific purposes. These often are high frequency of use commands where low use would still leverage large language model. You have an opportunity here to leverage this workflow. Being able to create commands with large language model is not something many projects have explored.