Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Omnilingual ASR: Advancing automatic speech recognition for 1600 languages

ai.meta.com

Post reply on HN
Print view View original

11–20 of 48 posts

Re: Omnilingual ASR: Advancing automatic speech recognition for 1600 languages

#11

Post by woodson »

samat wrote: post #5

How hard is it to make TTS out of this? A few independent journalists from Belarus asked for TTS in their language, but I am no expert, was thinking about re-using Mozilla's work. What's the easiest way to get working TTS for a language?

EDIT: My bad, please disregard; As akreal pointed out, the MMS TTS models aren’t using the SSL models.

Original post:

You can use the OmniASR SSL models instead of their older MMS models to create TTS models: https://github.com/ylacombe/finetune-hf-vits

Re: Omnilingual ASR: Advancing automatic speech recognition for 1600 languages

#12

Post by lostmsu »

stuffoverflow wrote: post #6

This seems like a massive improvement for openly available local ASR. Even the 300M model outperforms whisper-large-v3 according to the paper's benchmarks.

Not sure, I recorded 3 seconds of voice (a single sentence) and the hf demo misrecognized about half of the words.

Re: Omnilingual ASR: Advancing automatic speech recognition for 1600 languages

#13

Post by willwade »

woodson wrote: post #11
samat wrote: post #5

How hard is it to make TTS out of this? A few independent journalists from Belarus asked for TTS in their language, but I am no expert, was thinking about re-using Mozilla's work. What's the easiest way to get working TTS for a language?

EDIT: My bad, please disregard; As akreal pointed out, the MMS TTS models aren’t using the SSL models. Original post: You can use the OmniASR SSL models instead of their older MMS models to create TTS models: https://github.com/ylacombe/finetune-hf-vits

Meta cheated with the mms models. That is they didn’t use a phonemeizsr step. This means they just won’t work or sound very strange. ASR data is usually not quite right for tts. But anyhow - not really answering your question but many of these languages already done in mms. Try them https://huggingface.co/spaces/willwade/sherpa-onnx-tts

Re: Omnilingual ASR: Advancing automatic speech recognition for 1600 languages

#14

Post by prodigycorp »

cadamsdotcom wrote: post #9

Only a few gb of weights will recognize speech in 1600+ languages. Freely downloadable and usable by anyone for almost anything. We truly live in the future.

Seeing the absurd number of languages made me think of the norm macdonald joke:

Music is the universal language, but one day soon it will be replaced by Chinese.

Re: Omnilingual ASR: Advancing automatic speech recognition for 1600 languages

#15

Post by akreal »

woodson wrote: post #11
samat wrote: post #5

How hard is it to make TTS out of this? A few independent journalists from Belarus asked for TTS in their language, but I am no expert, was thinking about re-using Mozilla's work. What's the easiest way to get working TTS for a language?

EDIT: My bad, please disregard; As akreal pointed out, the MMS TTS models aren’t using the SSL models. Original post: You can use the OmniASR SSL models instead of their older MMS models to create TTS models: https://github.com/ylacombe/finetune-hf-vits

As far as I understand, the MMS TTS models are trained from scratch (section 7.1 of [1]), they do not employ any SSL models. So the OmniASR SSL models are not useful here.

What might be interesting is the newly released OmniASR data, because the MMS data, which was used for the MMS TTS, was never released.

Also, the OmniASR can be used to transcribe some untranscribed speech to train a TTS on it.

[1] MMS paper: https://arxiv.org/pdf/2305.13516

Re: Omnilingual ASR: Advancing automatic speech recognition for 1600 languages

#16

Post by momojo »

Does anyone else feel like they buried the lead?

> Omnilingual ASR was designed as a community-driven framework. People around the world can extend Omnilingual ASR to new languages by using just a few of their own samples.

The world just got smaller

Re: Omnilingual ASR: Advancing automatic speech recognition for 1600 languages

#17

Post by woodson »

akreal wrote: post #15
woodson wrote: post #11

Earlier quoted context omitted.

EDIT: My bad, please disregard; As akreal pointed out, the MMS TTS models aren’t using the SSL models. Original post: You can use the OmniASR SSL models instead of their older MMS models to create TTS models: https://github.com/ylacombe/finetune-hf-vits

As far as I understand, the MMS TTS models are trained from scratch (section 7.1 of [1]), they do not employ any SSL models. So the OmniASR SSL models are not useful here. What might be interesting is the newly released OmniASR data, because the MMS data, which was used for the MMS TTS, was never released. Also, the OmniASR can be used to transcribe some untranscribed speech to train a TTS on it. [1] MMS paper: https…

You’re completely right, I misremembered. I edited my post.

Re: Omnilingual ASR: Advancing automatic speech recognition for 1600 languages

#20

Post by nshm »

lostmsu wrote: post #12
stuffoverflow wrote: post #6

This seems like a massive improvement for openly available local ASR. Even the 300M model outperforms whisper-large-v3 according to the paper's benchmarks.

Not sure, I recorded 3 seconds of voice (a single sentence) and the hf demo misrecognized about half of the words.

This model is actually expected to be bad for popular languages, just like previous MMS it is not accurate at all, it wins by supporting something rare well but never had good ASR accuracy even for Swedish etc. It is more a research thing than a real tool. Unlike Whisper.
Post reply on HN