Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

get-inscribe.com

Post reply on HN
Print view View original

121–130 of 251 posts

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#121

Post by foobarqux »

foobarqux wrote: post #108
parl_match wrote: post #99

Earlier quoted context omitted.

Apple likely needed a model that ran on their NPU natively. - parakeet usually runs on Bfloat16. NPU doesn't support that - CPU is not as fast as the NPU for these ops on A-series, and even on modern CPUs, there's a latency delay - Parakeet latency is fine but "fine" may not be good enough for Apple's UX team. - CPU increases power consumption over dedicated float blocks So I would say that Parakeet was a non-option…

Fluidaudio implements Parakeet on ANE. I'd like to know how SpeechAnalyzer compares in speed. https://github.com/FluidInference/FluidAudio

Just tried test using yap on a single ~1hr mp3: yap/Speechanalyzer is about 50% slower than fluidaudio on M1. yap interface is nicer though.

https://github.com/finnvoor/yap

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#122

Post by clickety_clack »

dclowd9901 wrote: post #116

I'm always confused by these phrases: > The new API cuts word error rate by 3.5 to 4x on the same audio: from 9.02% to 2.12% on clean speech Shouldn't they have said "cuts error rate by 78%" or something?

I don’t like it written that way either, and it always seems like the type of number you put on a slide for a head of sales or something. It rankles because:

- it implies that error could be increased n-times, but a 15x _increase_ in 9% error would be an error rate of 135%, which is nonsensical.

- a reduction from 90% error to 20% error is clearly a bigger improvement in rightness to a reduction from 9% to 2%. One is “almost all wrong to almost all right”, the other is “more right”, but they are both a 4.5x reduction in error which means that the 4.5 quantity doesn’t have a constant meaning.

The answer is something like log odds ratios, but that introduces the additional need for a reader to know what that is, and that would be unusual.

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#123

Post by z2 »

satvikpendem wrote: post #34

Whisper is the wrong model to benchmark against, or rather, there are better models that are state of the art now like Nemotron and Parakeet both by Nvidia, as well as Mistral's Voxtral and Cohere Transcribe. However, what's funny is, RIP to a lot of the paid apps that simply wrap Whisper, I'm sure Apple will make a native GUI such as a recorder app for macOS that obviates the need for these wrappers, which everyone…

I don’t know how Apple divides computation between the GPU and the Neural Engine, but one major benefit, especially for real-time transcription on laptops, is the improved power and thermal efficiency. I noticed better accuracy after switching my app to SpeechAnalyzer, and I suspect part of that improvement for me came from the microphone no longer having to compete with jet-engine fan noise.

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#126

Post by llm_nerd »

satvikpendem wrote: post #34

Whisper is the wrong model to benchmark against, or rather, there are better models that are state of the art now like Nemotron and Parakeet both by Nvidia, as well as Mistral's Voxtral and Cohere Transcribe. However, what's funny is, RIP to a lot of the paid apps that simply wrap Whisper, I'm sure Apple will make a native GUI such as a recorder app for macOS that obviates the need for these wrappers, which everyone…

This particular product used Whisper, so that was obviously the right model to compare it against. Further this is explicitly on device, and Nemotron 3.5, as one example, is 2.5GB for the model.

And if someone were broadly comparing all on-device models (instead of just looking at how this new on-device ones compares to what a specific product uses), Nemotron 3.5's WER are actually a bit higher than what they report for SpeechAnalyzer, for both tests.

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#128

Post by dinfinity »

ChadNauseam wrote: post #50
satvikpendem wrote: post #34

Whisper is the wrong model to benchmark against, or rather, there are better models that are state of the art now like Nemotron and Parakeet both by Nvidia, as well as Mistral's Voxtral and Cohere Transcribe. However, what's funny is, RIP to a lot of the paid apps that simply wrap Whisper, I'm sure Apple will make a native GUI such as a recorder app for macOS that obviates the need for these wrappers, which everyone…

> there are better models that are state of the art now like Nemotron and Parakeet both by Nvidia Is parakeet state of the art? It always transcribes speech fragments for me, like if I stutter and say "m-m-m-map" parakeet will dutifully transcribe "m m m map". Which I guess could be a good thing or a bad thing depending on what you want. Whisper does not do that however. I do like cohere transcribe a lot.

I use Parakeet V3 via this tool and it is actually quite reliable for me (in English): https://github.com/cjpais/Handy

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#129

Post by wgm »

monster_truck wrote: post #127
wgm wrote: post #125

Is this the new dictation engine that I'm not allowed to run on my 1-YEAR-OLD IPHONE 17 because it's not Pro?

Why did you buy the cheap one, that's your own fault

At the time, there was very little advantage to buying the Pro model. Ironically, it's the first time I've ever bought the base model.

Re: Apple's new SpeechAnalyzer API, benchmarked against Whisper and its predecessor

#130

Post by rokkamokka »

seviu wrote: post #33
scuppernong wrote: post #14

Just ran it against Whisper-Large-V2 on a math lecture (my primary use case for ASR is subtitling math lectures), and it was substantially faster and only slightly worse. Very usable for live transcription though I'll probably stick with whisper for the time being since I don't really need the subtitles to be generated in real time.

Been using it for a podcast app I have been developing for half a year lol (I hope I publish it by version 27) and I can confirm it’s real fast. Splitting the audio in multiple segments and firing it up without hitting the maximum limit of concurrent decoding streams makes it blazing fast. Fair enough you loose the cut, but it’s good enough for just podcast. In one minute it chews through one hour of audio. This on a…

You could perhaps run over the segment splitting points (plus a few seconds back and forward) in a second batch then merge the results in the end so you don't miss anything.
Post reply on HN