This is a really impressive project – looking forward to trying it out!
Show HN: Dia, an open-weights TTS model for generating realistic dialogue
141–150 of 202 posts
Re: Show HN: Dia, an open-weights TTS model for generating realistic dialogue
#142[dead]
Re: Show HN: Dia, an open-weights TTS model for generating realistic dialogue
#143Very cool! Insane how much low hanging fruit there is for Audio models right now. A team of two picking things up over a few months can build something that still competes with large players with tons of funding
This is amazing. Is it possible to build in a chosen voice, a bit like Eleven Labs does? ...This may be on the git summary, being lazy and asking anyway :=) Thanks for your work.
Re: Show HN: Dia, an open-weights TTS model for generating realistic dialogue
#144I'm lost for words. This is extremely impressive!
Re: Show HN: Dia, an open-weights TTS model for generating realistic dialogue
#145Training an audio model this good from 0 prior experience is really amazing. I would love to read a blog post about how you guys approached ramping up knowledge and getting practical quickly. Any plans?
Re: Show HN: Dia, an open-weights TTS model for generating realistic dialogue
#146Fantastic model. I'm going to write a Unity plugin for this.
Have been using ElevenLabs for my VR game side project, but this appears to be better
Re: Show HN: Dia, an open-weights TTS model for generating realistic dialogue
#147Earlier quoted context omitted.
I have a sneaking suspicion it's because they lifted the model architecture almost directly from Parakeet: https://jordandarefsky.com/blog/2024/parakeet/ Parakeet references WhisperD which is at https://huggingface.co/jordand/whisper-d-v1a and doesn't include a full list of non-speech events that it's been trained with, except "(coughs)" and "(laughs)". Not saying the authors didn't do anything interesting here. They…
You're absolutely right. We used Jordan's Whisper-D, and he was generous enough to offer some guidance along the way. It's also a valid criticism that we haven’t yet audited the dataset for existing list of tags. That’s something we’ll be improving soon. As for Dia’s architecture, we largely followed existing models to build the 1.6B version. Since we only started learning about speech AI three months ago, we chose n…
I’m curious what differentiates it from Parakeet? I was listening to some of the demos on the parakeet announcement and they sound very similar to your examples - are they trained on the same data? Are there benefits to using Dia over Parakeet?
Re: Show HN: Dia, an open-weights TTS model for generating realistic dialogue
#148Earlier quoted context omitted.
Could one usecase be generating an audiobook with this from existing books? I wonder if I could fine-tune the "characters" that speak these lines since you said it's a single pass whole the whole convo. Wonder if that's a limitation for this kind of a usecase (where speed is not imperative).
Yes! But you would need to put together a LLM system that created scripts from the book content. There is an open source project called OpenNotebookLM ( https://github.com/gabrielchua/open-notebooklm ) that does something similar. If you hook the Dia model to that kind of system, it will be very possible :) Thanks for the interest!
Another project, specifically for creating audiobooks: https://github.com/prakharsr/audiobook-creator
Re: Show HN: Dia, an open-weights TTS model for generating realistic dialogue
#149This is really impressive; we're getting close to a dream of mine: the ability to generate proper audiobooks from EPUBs. Not just a robotic single voice for everything, but different, consistent voices for each protagonist, with the LLM analyzing the text to guess which voice to use and add an appropriate tone, much like a voice actor would do. I've tried "EPUB to audiobook" tools, but they are really miles behind wh…
Wouldn’t it be more desirable to hear an actual human on an audiobook? Ideally the author?
Why a human? There are many cases where I like a book but dislike the audiobook speaker, so I essentially can't listen to that book anymore. With a machine, I can tweak the voice to my heart's content.
Re: Show HN: Dia, an open-weights TTS model for generating realistic dialogue
#150This is really impressive; we're getting close to a dream of mine: the ability to generate proper audiobooks from EPUBs. Not just a robotic single voice for everything, but different, consistent voices for each protagonist, with the LLM analyzing the text to guess which voice to use and add an appropriate tone, much like a voice actor would do. I've tried "EPUB to audiobook" tools, but they are really miles behind wh…
Realistic voice acting for audio books, realistic images for each page, realistic videos for each page, oh wait I just created a movie, maybe I can change the plot? Oh wait I just created a video game
Now do it in VR and make it fully interactive.