Omni SenseVoice: High-Speed Speech Recognition with Words Timestamps

#1

Post by ringer007 » Sun, Oct 13, 2024, 12:48 AM UTC

Omni SenseVoice: High-Speed Speech Recognition with Words Timestamps
github.com

Re: Omni SenseVoice: High-Speed Speech Recognition with Words Timestamps

#2

Post by modeless » Sun, Oct 13, 2024, 2:43 AM UTC

Looks cool! Combine this with this new TTS that released today that looks really good and an LLM and you'd have a pretty good all-local voice assistant! https://github.com/SWivid/F5-TTS

Re: Omni SenseVoice: High-Speed Speech Recognition with Words Timestamps

#3

Post by staticautomatic » Sun, Oct 13, 2024, 3:34 AM UTC

I’ve been building a production app on top of ASR and find the range of models kind of bewildering compared to LLMs and video. The commercial offerings seem to be custom or built on top of Whisper or maybe nvidia canary/parakeet and then you have stuff like speechbrain that seems to run on top of lots of different open models for different tasks. Sometimes it’s genuinely hard to tell what’s a foundation model and what isn’t.

Separately, I wonder if this is the model Speechmatics uses.

Re: Omni SenseVoice: High-Speed Speech Recognition with Words Timestamps

#4

Post by leetharris » Sun, Oct 13, 2024, 4:02 AM UTC

staticautomatic wrote: post #3

I’ve been building a production app on top of ASR and find the range of models kind of bewildering compared to LLMs and video. The commercial offerings seem to be custom or built on top of Whisper or maybe nvidia canary/parakeet and then you have stuff like speechbrain that seems to run on top of lots of different open models for different tasks. Sometimes it’s genuinely hard to tell what’s a foundation model and wha…

We released a new SOTA ASR as open source just a couple of weeks ago. https://www.rev.com/blog/speech-to-text-technology/introduci...

Take a look. We'll be open sourcing more models very soon!

Re: Omni SenseVoice: High-Speed Speech Recognition with Words Timestamps

#5

Post by satvikpendem » Sun, Oct 13, 2024, 4:24 AM UTC

Can it diarize?

Re: Omni SenseVoice: High-Speed Speech Recognition with Words Timestamps

#6

Post by mkl » Sun, Oct 13, 2024, 4:27 AM UTC

leetharris wrote: post #4

staticautomatic wrote: post #3

I’ve been building a production app on top of ASR and find the range of models kind of bewildering compared to LLMs and video. The commercial offerings seem to be custom or built on top of Whisper or maybe nvidia canary/parakeet and then you have stuff like speechbrain that seems to run on top of lots of different open models for different tasks. Sometimes it’s genuinely hard to tell what’s a foundation model and wha…

We released a new SOTA ASR as open source just a couple of weeks ago. https://www.rev.com/blog/speech-to-text-technology/introduci... Take a look. We'll be open sourcing more models very soon!

> These models are accessible under a non-commercial license.

That is not open source.

Re: Omni SenseVoice: High-Speed Speech Recognition with Words Timestamps

#7

Post by staticautomatic » Sun, Oct 13, 2024, 4:29 AM UTC

leetharris wrote: post #4

staticautomatic wrote: post #3

I’ve been building a production app on top of ASR and find the range of models kind of bewildering compared to LLMs and video. The commercial offerings seem to be custom or built on top of Whisper or maybe nvidia canary/parakeet and then you have stuff like speechbrain that seems to run on top of lots of different open models for different tasks. Sometimes it’s genuinely hard to tell what’s a foundation model and wha…

We released a new SOTA ASR as open source just a couple of weeks ago. https://www.rev.com/blog/speech-to-text-technology/introduci... Take a look. We'll be open sourcing more models very soon!

I'll check it out.

FWIW, in terms of benchmarking, I'm more interested in benchmarks against Gladia, Deepgram, Pyannote, and Speechmatics than whatever is built into the hyperscaler platforms. But I end up doing my own anyway so whatevs.

Also, you guys need any training data? I have >10K hrs of conversational iso-audio :)

Re: Omni SenseVoice: High-Speed Speech Recognition with Words Timestamps

#8

Post by yalok » Sun, Oct 13, 2024, 4:34 AM UTC

leetharris wrote: post #4

staticautomatic wrote: post #3

I’ve been building a production app on top of ASR and find the range of models kind of bewildering compared to LLMs and video. The commercial offerings seem to be custom or built on top of Whisper or maybe nvidia canary/parakeet and then you have stuff like speechbrain that seems to run on top of lots of different open models for different tasks. Sometimes it’s genuinely hard to tell what’s a foundation model and wha…

We released a new SOTA ASR as open source just a couple of weeks ago. https://www.rev.com/blog/speech-to-text-technology/introduci... Take a look. We'll be open sourcing more models very soon!

that's great to hear! amazing performance of the model!

for voice chat bots, however, shorter input utterances are a norm (anywhere from 1-10 sec), with lots of silence in between, so this limitation is a bit sad:

> On the Gigaspeech test suite, Rev’s research model is worse than other open-source models. The average segment length of this corpus is 5.7 seconds; these short segments are not a good match for the design of Rev’s model. These results demonstrate that despite its strong performance on long-form tests, Rev is not the best candidate for short-form recognition applications like voice search.