Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

get-inscribe.com

Post reply on HN
Print view View original

51–60 of 251 posts

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#51

Post by satvikpendem »

get-inscribe wrote: post #2

Author here. I ship both Apple speech engines plus WhisperKit side by side in a transcription app, which made it possible to run all five through identical production code on the same audio: LibriSpeech test-clean and test-other, 5,559 utterances, fully on-device on an M2 Pro. Apple published no accuracy numbers for SpeechAnalyzer (or for SFSpeechRecognizer, ever, as far as I can tell), so the migration question has…

Why do you not use Whisper large models when on macOS? They're still fast even when streaming and yield a much lower WER.

Also, just out of curiosity, seems like everyone and their mother is making Whisper wrappers, how is your app different?

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#52

Post by wahnfrieden »

drnick1 wrote: post #23

If this isn't open source/weights and can't run locally, I don't see how this is a replacement for Whisper or other open models, e.g. within Home Assistant.

It is local

The appeal is that users only have to download it once across all apps that use it. Instead of convincing a user to give a couple gigs for just your one app

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#54

Post by ks2048 »

tancop wrote: post #8

this is amazing. if i had a mac i would try to reverse engineer the code, extract the weights and port it to something that works on linux/windows like torch or burn. then put the code on github and weights on a torrent site. lifes too short to let apple keep their models exclusive.

Probably not worth the effort (or legal trouble), unless you can show it's better than other recent open models like Cohere Transcribe.

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#55

Post by realityfactchex »

paul7986 wrote: post #28
realityfactchex wrote: post #10

Earlier quoted context omitted.

Yeah, ChatGPT voice is great experience vs. Siri on that phone. In case you haven't done something like this already: 1. In Shortcuts app, make shortcut named "AI Voice Mode" (or whatever you want, YMMV) 2. Set it to run the ChatGPT action "Voice Mode" (requires at least the minimum paid tier, I think) 3. To trigger, say "Hey Siri, AI Voice Mode" (or whatever you called the shortcut) This is a pretty slick integratio…

Ridiculous that for past many years we can talk to GPT on our iPhones without any hiccups and this new Siri is still the same old horse crap (at least for me and this latest beta). Buy them already Apple or possibly be replaced by them as their path & trajectory (working on Ai devices now and they are stealing like Jobs did with Xerox) mirrors yours in the late 1970s. Thanks for the tip and if Im not mistaken it's si…

> similar to asking Siri to ask chatGPT to ask XYZ

Effectively, it sort of does that, but really it just listens to the wakeword and opens/switches to the requested app & modality.

FWIW, I get a very different functional result using the Shortcut method vs. asking Siri to delegate natively. To compare, I asked Siri (non-beta here) now to "ask ChatGPT " and I got a top-card with some fairly low quality SEO-ranked weblinks.

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#57

Post by satvikpendem »

wahnfrieden wrote: post #41
satvikpendem wrote: post #34

Whisper is the wrong model to benchmark against, or rather, there are better models that are state of the art now like Nemotron and Parakeet both by Nvidia, as well as Mistral's Voxtral and Cohere Transcribe. However, what's funny is, RIP to a lot of the paid apps that simply wrap Whisper, I'm sure Apple will make a native GUI such as a recorder app for macOS that obviates the need for these wrappers, which everyone…

For multilingual and noisy audio the best right now is MOSS-Transcribe-Diarize which was released just a few days ago Superwhisper does a lot more than just provide a whisper/parakeet UI so I’m not sure Apple will destroy them so easily

Thanks, was looking at a better diarization model.

Even for those sorts of apps, MacParakeet which I've been using is FOSS so no payment needed. In reality these days with AI the ability to spin up a free and/or OSS competitor falls to zero.

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#58

Post by wahnfrieden »

get-inscribe wrote: post #2

Author here. I ship both Apple speech engines plus WhisperKit side by side in a transcription app, which made it possible to run all five through identical production code on the same audio: LibriSpeech test-clean and test-other, 5,559 utterances, fully on-device on an M2 Pro. Apple published no accuracy numbers for SpeechAnalyzer (or for SFSpeechRecognizer, ever, as far as I can tell), so the migration question has…

Please run your benchmark on this new and very impressive model https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize in my testing it outperforms all mentioned especially on noisy audio

MOSS-Transcribe-Diarize

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#59

Post by layer8 »

kridsdale1 wrote: post #9
tancop wrote: post #8

this is amazing. if i had a mac i would try to reverse engineer the code, extract the weights and port it to something that works on linux/windows like torch or burn. then put the code on github and weights on a torrent site. lifes too short to let apple keep their models exclusive.

Is that copyright infringement?

Copyright protects original forms of expression, not arbitrary data. It is very arguable whether it applies to model weights. However, it would likely constitute a license violation.

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#60

Post by summarity »

Vs Voxtral would be a better comparison. No other model, open or closed, has been able to hit such a low AER (Acronym Error Rate ;)) for my meeting transcripts. Seems to understand/infer all the technobabble I use at work. Never have to edit anything. Whisper was catastrophically bad.
Post reply on HN