Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor
1–10 of 251 posts
Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor
#2Author here. I ship both Apple speech engines plus WhisperKit side by side in a transcription app, which made it possible to run all five through identical production code on the same audio: LibriSpeech test-clean and test-other, 5,559 utterances, fully on-device on an M2 Pro.
Apple published no accuracy numbers for SpeechAnalyzer (or for SFSpeechRecognizer, ever, as far as I can tell), so the migration question has been guesswork. Short version: the new API cuts WER 3.5-4x vs the old one (2.12% vs 9.02% on test-clean), and it also beat Whisper Small on both splits at about 3x the speed. The old API came in last on clean speech, behind even Whisper Tiny.
On "why should I trust a vendor benchmark": the Whisper column reproduces OpenAI's published LibriSpeech WERs within +0.11 to +0.42 on all six measurements (same corpus, same normalizer, same scorer for every engine), and the raw per-utterance transcripts are downloadable from the article if anyone wants to rescore with their own normalizer.
Limitations worth stating up front: English only, read speech rather than meeting audio, one machine. Precise per-engine timing isn't in the article yet because the accuracy runs shared the machine with a dev workload; WER is load-independent, timing isn't.
Two things that might interest people migrating: SFSpeechRecognizer sends audio to Apple's servers unless you set requiresOnDeviceRecognition, and with SpeechAnalyzer, finishing your input stream is not enough to end a session. If you never call finalizeAndFinishThroughEndOfInput(), the results sequence never terminates and your await hangs forever. I found that one because it was shipping in my own app.
Happy to answer questions about the harness or the normalizer.
Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor
#3Impressive. Apple said they improved the models in 27 didn’t they? It would be interesting to see the numbers the beta turns in.
Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor
#4I second that! Can you run your benchmarks against the iOS 27 beta?
Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor
#5Still nothing beats OpenAI's VTT. Anthropic's sucks and Apple's isn't even usable.
Edit: Getting downvoted by Apple fanboys for telling the truth is a badge of honor.
Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor
#6Still nothing beats OpenAI's VTT. Anthropic's sucks and Apple's isn't even usable. Edit: Getting downvoted by Apple fanboys for telling the truth is a badge of honor.
Which OpenAI model/API do you mean?
Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor
#7Im hoping Apple gets the new Siri working better on older phones. I was excited to use it but the latest beta / Siri runs too slow on my iPhone Pro Max 15.
Im looking for the same experience I have when talking to chatGPT. As for past two years or more talking to GPT within it's app and on my iPhone Pro Max 15 it runs smooth as butter :-). This is the experience I was and still am hoping with Apple, but Im thinking all the extra layers of privacy and security might be slowing them down?
Overall, Apple who is suing Open AI should just buy them and let me have the best conversational AI out there baked into my old ass iPhone. Because as so far the new Siri on my old phone (tho again GPT works great talking to it and for years) doesnt come close. It's the same old "Could you try that again," Siri. BOO!!!
Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor
#8this is amazing. if i had a mac i would try to reverse engineer the code, extract the weights and port it to something that works on linux/windows like torch or burn. then put the code on github and weights on a torrent site. lifes too short to let apple keep their models exclusive.
Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor
#9this is amazing. if i had a mac i would try to reverse engineer the code, extract the weights and port it to something that works on linux/windows like torch or burn. then put the code on github and weights on a torrent site. lifes too short to let apple keep their models exclusive.
Is that copyright infringement?
Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor
#10Im hoping Apple gets the new Siri working better on older phones. I was excited to use it but the latest beta / Siri runs too slow on my iPhone Pro Max 15. Im looking for the same experience I have when talking to chatGPT. As for past two years or more talking to GPT within it's app and on my iPhone Pro Max 15 it runs smooth as butter :-). This is the experience I was and still am hoping with Apple, but Im thinking a…
Yeah, ChatGPT voice is great experience vs. Siri on that phone. In case you haven't done something like this already:
1. In Shortcuts app, make shortcut named "AI Voice Mode" (or whatever you want, YMMV)
2. Set it to run the ChatGPT action "Voice Mode" (requires at least the minimum paid tier, I think)
3. To trigger, say "Hey Siri, AI Voice Mode" (or whatever you called the shortcut)