Live data from Hacker News

Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

get-inscribe.com

21–30 of 251 posts

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#21

Author here. I ship both Apple speech engines plus WhisperKit side by side in a transcription app, which made it possible to run all five through identical production code on the same audio: LibriSpeech test-clean and test-other, 5,559 utterances, fully on-device on an M2 Pro. Apple published no accuracy numbers for SpeechAnalyzer (or for SFSpeechRecognizer, ever, as far as I can tell), so the migration question has…

Why use relatively ancient models like whisper and not e. g. parakeet?

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#22
post #19
post #5

Still nothing beats OpenAI's VTT. Anthropic's sucks and Apple's isn't even usable. Edit: Getting downvoted by Apple fanboys for telling the truth is a badge of honor.

In which world 98% accuracy is not usable?

In a world where you say "tmux" and Apple's VTT writes "T Max".

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#24
post #8

this is amazing. if i had a mac i would try to reverse engineer the code, extract the weights and port it to something that works on linux/windows like torch or burn. then put the code on github and weights on a torrent site. lifes too short to let apple keep their models exclusive.

Aside from the legality of it, I think you are underestimating how complex it can be to do that. It is possible in theory but not something that will be a fun side quest like you are making it seem.

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#25

Whisper small/tiny/base are almost four years old (they were not updated for Whisper v2 or v3). Is there really nothing better to benchmark against by now?

There's tons, Parakeet was the last I remember seeing which seemed to gain traction (independent lightweight implementations etc).

Parakeet isn't as good as whisper large.

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#26

Whisper small/tiny/base are almost four years old (they were not updated for Whisper v2 or v3). Is there really nothing better to benchmark against by now?

There's tons, Parakeet was the last I remember seeing which seemed to gain traction (independent lightweight implementations etc).

Parakeet is way faster (on Nvidia hardware) but not quite as accurate in my experience.

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#27
post #7

Im hoping Apple gets the new Siri working better on older phones. I was excited to use it but the latest beta / Siri runs too slow on my iPhone Pro Max 15. Im looking for the same experience I have when talking to chatGPT. As for past two years or more talking to GPT within it's app and on my iPhone Pro Max 15 it runs smooth as butter :-). This is the experience I was and still am hoping with Apple, but Im thinking a…

> Im hoping Apple gets the new Siri working better on older phones.

Apple would never do that, if anything they did not offer their Siri with the most advanced AI on iPhone 16 Pro Max, which is one year-old only.

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#28
post #7

Im hoping Apple gets the new Siri working better on older phones. I was excited to use it but the latest beta / Siri runs too slow on my iPhone Pro Max 15. Im looking for the same experience I have when talking to chatGPT. As for past two years or more talking to GPT within it's app and on my iPhone Pro Max 15 it runs smooth as butter :-). This is the experience I was and still am hoping with Apple, but Im thinking a…

Yeah, ChatGPT voice is great experience vs. Siri on that phone. In case you haven't done something like this already: 1. In Shortcuts app, make shortcut named "AI Voice Mode" (or whatever you want, YMMV) 2. Set it to run the ChatGPT action "Voice Mode" (requires at least the minimum paid tier, I think) 3. To trigger, say "Hey Siri, AI Voice Mode" (or whatever you called the shortcut) This is a pretty slick integratio…

Ridiculous that for past many years we can talk to GPT on our iPhones without any hiccups and this new Siri is still the same old horse crap (at least for me and this latest beta). Buy them already Apple or possibly be replaced by them as their path & trajectory (working on Ai devices now and they are stealing like Jobs did with Xerox) mirrors yours in the late 1970s.

Thanks for the tip and if Im not mistaken it's similar to asking Siri to ask chatGPT to ask XYZ?

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#30

Just ran it against Whisper-Large-V2 on a math lecture (my primary use case for ASR is subtitling math lectures), and it was substantially faster and only slightly worse. Very usable for live transcription though I'll probably stick with whisper for the time being since I don't really need the subtitles to be generated in real time.

If it was faster but worse, maybe compare it to a smaller whisper model?

I imagine because quality of transcription is what matters.
Post reply on HN