Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

get-inscribe.com

Post reply on HN
Print view View original

41–50 of 251 posts

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#41

Post by wahnfrieden »

satvikpendem wrote: post #34

Whisper is the wrong model to benchmark against, or rather, there are better models that are state of the art now like Nemotron and Parakeet both by Nvidia, as well as Mistral's Voxtral and Cohere Transcribe. However, what's funny is, RIP to a lot of the paid apps that simply wrap Whisper, I'm sure Apple will make a native GUI such as a recorder app for macOS that obviates the need for these wrappers, which everyone…

For multilingual and noisy audio the best right now is MOSS-Transcribe-Diarize which was released just a few days ago

Superwhisper does a lot more than just provide a whisper/parakeet UI so I’m not sure Apple will destroy them so easily

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#42

Post by bilbo0s »

altmanaltman wrote: post #24
tancop wrote: post #8

this is amazing. if i had a mac i would try to reverse engineer the code, extract the weights and port it to something that works on linux/windows like torch or burn. then put the code on github and weights on a torrent site. lifes too short to let apple keep their models exclusive.

Aside from the legality of it, I think you are underestimating how complex it can be to do that. It is possible in theory but not something that will be a fun side quest like you are making it seem.

This.

The Jedi Hand Wave-y nature of the way people talk about AI these days is going to make reigning in the AI superpowers nearly impossible. Because there are people out here who believe models of this quality are easily replicated or reverse engineered. Neither is really doable on any reasonable timeline by people who are not AI experts. Real AI experts. Not TF/PyTorch monkeys or Agent Slop Slingers.

And those people are already highly incentivized to not make anything performing better than SOTA models open source.

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#45

Post by satvikpendem »

Chu4eeno wrote: post #17
modeless wrote: post #12

Whisper small/tiny/base are almost four years old (they were not updated for Whisper v2 or v3). Is there really nothing better to benchmark against by now?

There's tons, Parakeet was the last I remember seeing which seemed to gain traction (independent lightweight implementations etc).

Nvidia's Nemotron subsumes their older Parakeet model now even for real time streaming.

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#47

Post by wahnfrieden »

Tsarp wrote: post #35

Any chance you can benchmark against whisper large and large v3 turbo? These run comfortably on older Macbooks and are still far more accurate in real life dictation compared to even the parakeet models( despite ASR leaderboards) with an RTF < 1.

Try MOSS-Transcribe-Diarize from a few days ago. I’m getting better results than those whisper models. And it’s very fast and small. Better suited to noisy audio too.

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#48

Post by drnick1 »

edude03 wrote: post #40
drnick1 wrote: post #23

If this isn't open source/weights and can't run locally, I don't see how this is a replacement for Whisper or other open models, e.g. within Home Assistant.

It's not open weight, but the point is to be an on device (and thus local, privacy preserving) option. The article mentions that as the caveat > What this means if you just want good transcription > If you are on a current iPhone or Mac, the best on-device transcription engine for English is already in the operating system, and the private option is no longer the compromise option

> It's not open weight, but the point is to be an on device (and thus local, privacy preserving) option.

How can you be sure this isn't leaking data or metadata to Apple? Can Apple really be trusted?

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#49

Post by garblegarble »

Chu4eeno wrote: post #17
modeless wrote: post #12

Whisper small/tiny/base are almost four years old (they were not updated for Whisper v2 or v3). Is there really nothing better to benchmark against by now?

There's tons, Parakeet was the last I remember seeing which seemed to gain traction (independent lightweight implementations etc).

I have tried everything (that will run on a 12GB RTX 4070) and I have yet to find anything with better accuracy than Whisper V2 Large for my dataset (discord audio from TTRPG sessions, isolated per-speaker, mostly non-American accents)

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#50

Post by ChadNauseam »

satvikpendem wrote: post #34

Whisper is the wrong model to benchmark against, or rather, there are better models that are state of the art now like Nemotron and Parakeet both by Nvidia, as well as Mistral's Voxtral and Cohere Transcribe. However, what's funny is, RIP to a lot of the paid apps that simply wrap Whisper, I'm sure Apple will make a native GUI such as a recorder app for macOS that obviates the need for these wrappers, which everyone…

> there are better models that are state of the art now like Nemotron and Parakeet both by Nvidia

Is parakeet state of the art? It always transcribes speech fragments for me, like if I stutter and say "m-m-m-map" parakeet will dutifully transcribe "m m m map". Which I guess could be a good thing or a bad thing depending on what you want. Whisper does not do that however.

I do like cohere transcribe a lot.

Post reply on HN