Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

get-inscribe.com

Post reply on HN
Print view View original

11–20 of 251 posts

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#11

Post by kridsdale1 »

paul7986 wrote: post #7

Im hoping Apple gets the new Siri working better on older phones. I was excited to use it but the latest beta / Siri runs too slow on my iPhone Pro Max 15. Im looking for the same experience I have when talking to chatGPT. As for past two years or more talking to GPT within it's app and on my iPhone Pro Max 15 it runs smooth as butter :-). This is the experience I was and still am hoping with Apple, but Im thinking a…

I’m on iPhone 17 Pro Max, 27 beta 3.

New Siri is impressive in that it answers satisfactorily now 80% of the time vs 10% with old Siri.

But it’s slow as shit. GPT, Claude, and Gemini can answer me in 5-10 seconds. Google AI Mode can answer in 2 seconds.

New Siri usually takes 25 seconds to respond to me. This morning it timed out (with strong network connection) when asked a simple multiplication question.

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#13

Post by kridsdale1 »

realityfactchex wrote: post #10
paul7986 wrote: post #7

Im hoping Apple gets the new Siri working better on older phones. I was excited to use it but the latest beta / Siri runs too slow on my iPhone Pro Max 15. Im looking for the same experience I have when talking to chatGPT. As for past two years or more talking to GPT within it's app and on my iPhone Pro Max 15 it runs smooth as butter :-). This is the experience I was and still am hoping with Apple, but Im thinking a…

Yeah, ChatGPT voice is great experience vs. Siri on that phone. In case you haven't done something like this already: 1. In Shortcuts app, make shortcut named "AI Voice Mode" (or whatever you want, YMMV) 2. Set it to run the ChatGPT action "Voice Mode" (requires at least the minimum paid tier, I think) 3. To trigger, say "Hey Siri, AI Voice Mode" (or whatever you called the shortcut) This is a pretty slick integratio…

You can also map the Action Button to GPT voice mode.

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#14

Post by scuppernong »

Just ran it against Whisper-Large-V2 on a math lecture (my primary use case for ASR is subtitling math lectures), and it was substantially faster and only slightly worse. Very usable for live transcription though I'll probably stick with whisper for the time being since I don't really need the subtitles to be generated in real time.

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#15

Post by coder543 »

get-inscribe wrote: post #2

Author here. I ship both Apple speech engines plus WhisperKit side by side in a transcription app, which made it possible to run all five through identical production code on the same audio: LibriSpeech test-clean and test-other, 5,559 utterances, fully on-device on an M2 Pro. Apple published no accuracy numbers for SpeechAnalyzer (or for SFSpeechRecognizer, ever, as far as I can tell), so the migration question has…

At this point, I would not recommend ignoring Parakeet TDT 0.6b v2/v3 (english-only versus multilingual). Those models have been out for a year, give or take, and they are both accurate and fast. I would choose Parakeet over Whisper in almost all situations these days. Parakeet works great even on my several year old iPhone 15 Pro Max, so if an app is going to ship a dedicated model, I strongly recommend investigating Parakeet.

On the more cutting edge front, Granite Speech 4.1 has proven to be a reliable workhorse for me, but it is larger than Parakeet. Cohere Transcribe is interesting, but how strong it is seems to vary more from task to task.

Parakeet Unified 0.6B came out a few months ago, combining both online streaming and offline transcription into one model, and that is one that I need to test more, but it seems promising.

As others have mentioned macOS 27/iOS 27 is supposed to have a new model, particularly on devices with 12GB of RAM or more. I have not actually seen the option to enable that new model yet, though, despite being on the beta on a device that meets the requirements. Maybe a benchmark would reveal that it is already active?

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#16

Post by Chu4eeno »

scuppernong wrote: post #14

Just ran it against Whisper-Large-V2 on a math lecture (my primary use case for ASR is subtitling math lectures), and it was substantially faster and only slightly worse. Very usable for live transcription though I'll probably stick with whisper for the time being since I don't really need the subtitles to be generated in real time.

If it was faster but worse, maybe compare it to a smaller whisper model?

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#17

Post by Chu4eeno »

modeless wrote: post #12

Whisper small/tiny/base are almost four years old (they were not updated for Whisper v2 or v3). Is there really nothing better to benchmark against by now?

There's tons, Parakeet was the last I remember seeing which seemed to gain traction (independent lightweight implementations etc).

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#18

Post by nodja »

kridsdale1 wrote: post #9
tancop wrote: post #8

this is amazing. if i had a mac i would try to reverse engineer the code, extract the weights and port it to something that works on linux/windows like torch or burn. then put the code on github and weights on a torrent site. lifes too short to let apple keep their models exclusive.

Is that copyright infringement?

This hasn't been tested in court. But there's a high chance that model weights are not copyrightable, only the code to generate them is.

Cloud models are usually protected by trade secret laws, leaking them would get you in trouble. However if the model is made available publicly, as long as you don't break the law to get them, anything after that would be fair game unless Apple can prove that humans have significant authorship over the weights, which hasn't been tested and is a significant burden to prove/disprove.

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#20

Post by behnamoh »

simonw wrote: post #6
behnamoh wrote: post #5

Still nothing beats OpenAI's VTT. Anthropic's sucks and Apple's isn't even usable. Edit: Getting downvoted by Apple fanboys for telling the truth is a badge of honor.

Which OpenAI model/API do you mean?

Whisper and GPT-4o (for diarization).
Post reply on HN