I get that they're wary of an LLM-based Siri making embarrassing mistakes. It would be a
bad look if some kid's phone provided helpful instructions for killing themselves when asked. Mummy and Daddy would sue.
However, directly answering queries is not the only way LLMs can be used. In fact, when ChatGPT 4 came out, the first thing I thought of was: "Wow, this could make Siri so much better!"
For example, an LLM like GPT coupled with voice recognition was used to create Whisper, an AI that has nearly perfect text-from-audio recognition. One of my biggest gripes with Siri is that it is basically useless in a car because even slight background noise confuses its voice recognition. An LLM would fix this.
Another point is that many people don't realise that LLMs re-read their entire input for every word they generate! Their writing speed is so-so, but they can read really fast even when running on mobile device hardware. Think 10K to 100K words per second. An LLM could read through all of the text on your device when prompted for search queries in a fraction of a second. As long as this was carefully set up, it wouldn't be able to produce "bad output", because it would just be matching data to your prompt.
E.g.: Imagine GPT being prompted with: "Does this email match the query ? Say only YES if it does or NO if it does not. "
It doesn't matter if it occasionality hallucinates and outputs gibberish, you just mark that as a "NO" and move on. This is also very easy to train out of a specialised version of the model using reinforcement learning.
PS: I just played around with GPT 4 to see how it behaves when asked to recognise requests for creating calendar entries, and it's pretty good. For example, it can correctly compute things like "next long weekend". Interestingly, ChatGPT 4 is already doing some similar prompt injection, and I can't override its sense of "current time".
Apple's Siri team has failed really badly. Everyone else is sprinting away while they're not even aware there's a race going on.