MIT license, supported languages beyond english: ar, ch, de, es, fr, it, ja, pl, pt.
TADA: Speech generation through text-acoustic synchronization
21–30 of 31 posts
Re: TADA: Speech generation through text-acoustic synchronization
#22Will this run on CPU? (as opposed to GPU)
This is bound to be a question that will be increasingly harder to answer. For instance, Apple processors have at least two different neural accelerators/matrix coprocessors (ANE and AMX) in addition to the integrated GPU. Do these count as "CPU"?
I think the answer is rather simple and boring -- only the CPU type commonly used in cheap cloud machines counts. This still is x86 only.
The homes at home, such as by Apple, don't count for serious workflows that must run reliably.
Re: TADA: Speech generation through text-acoustic synchronization
#23To me, the speech sounds impressively expressive, but there is something off about the audio quality that I can't quite put my finger on. The "Anger Speech" has an obvious lisp (Maybe a homage to Elmer Fudd?). But I hear a similar, but more subtle, speech impediment in the "Adoration Speech". The "Fearful Speech" might have a slight warble to it. And the "Long Speech" is difficult to evaluate because the speaker has…
> speaker has vocal fry to an extent that I find annoying.
Was it trained on Sam Altman?
Re: TADA: Speech generation through text-acoustic synchronization
#24Will this run on CPU? (as opposed to GPU)
I could not get my Mac to successfully do anything with the script from their GitHub; set device to mps, downloaded llama for the first time in a year, and it just .. hangs. I presume this is sortable, but I'm not sure I care enough.
All that said, I think it likely this has been built and trained only on Nvidia
Re: TADA: Speech generation through text-acoustic synchronization
#25"Long speech" is a faithful synthesis of a fairly irritating modern American English speech pattern.
"Vocal fry", aka "creaky voice". It's stereotypically associated with irritating young women.
I like me a good rabbit hole that's interesting and also digs into stereotypes.
Turns out, like many memes, it's not just that. It's (also?) a normal speech pattern, used by different genders, ages, and social groups, in many languages.
This doesn't mean that vocal fry isn't used as social signaling. But complaining about it, well, isn't that social signalling too?
Geoff Lindsey - Vocal Fry: what it is, who does it, and why people hate it! - https://www.youtube.com/watch?v=Q0yL2GezneU
Re: TADA: Speech generation through text-acoustic synchronization
#26the 0.09 RTF is wild but i wonder how much of that speed advantage disappears once you need voice cloning or fine grained prosody control. i use cartesia sonic for TTS in a video pipeline and the thing that actually matters for content creation isnt raw speed - its whether you can get consistent emotional delivery across like 50+ scenes without it drifting. the 1:1 text-acoustic alignment should help with hallucinati…
Given that it's one-to-one audio and text tokens, you'd get mid-sentence pauses if you just stopped feeding it.
Re: TADA: Speech generation through text-acoustic synchronization
#27"Long speech" is a faithful synthesis of a fairly irritating modern American English speech pattern.
"Vocal fry", aka "creaky voice". It's stereotypically associated with irritating young women. I like me a good rabbit hole that's interesting and also digs into stereotypes. Turns out, like many memes, it's not just that. It's (also?) a normal speech pattern, used by different genders, ages, and social groups, in many languages. This doesn't mean that vocal fry isn't used as social signaling. But complaining about it…
Not the fry, the cadence that makes everything sound like the same list of three or four things
Re: TADA: Speech generation through text-acoustic synchronization
#28To me, the speech sounds impressively expressive, but there is something off about the audio quality that I can't quite put my finger on. The "Anger Speech" has an obvious lisp (Maybe a homage to Elmer Fudd?). But I hear a similar, but more subtle, speech impediment in the "Adoration Speech". The "Fearful Speech" might have a slight warble to it. And the "Long Speech" is difficult to evaluate because the speaker has…
There's a subtle modulation that happens on all of the samples. It sounds almost like some kind of harmonic or phase shift? This is something I notice with every AI generated speech out there.
Re: TADA: Speech generation through text-acoustic synchronization
#29Earlier quoted context omitted.
This is bound to be a question that will be increasingly harder to answer. For instance, Apple processors have at least two different neural accelerators/matrix coprocessors (ANE and AMX) in addition to the integrated GPU. Do these count as "CPU"?
I think the answer is rather simple and boring -- only the CPU type commonly used in cheap cloud machines counts. This still is x86 only. The homes at home, such as by Apple, don't count for serious workflows that must run reliably.
Personally, I love synthesis that can be generated on the client machine, in real time. For some applications, like screen readers, this is a really important feature.
Of course, the big interest these days is in cloud based assistants, where synthesizing on server and piggybacking on the rest of the answer is quite reasonable.
Re: TADA: Speech generation through text-acoustic synchronization
#30okay so they say text continuation only without fine tuning. I assume that means that we can't use it as a replacement for TTS in an AI agent chat? Because it will not work without enough context? Could you maybe trick it into thinking it was continuing a sample for an assistant use case if the sample was generic enough? I appreciate them being honest about it though because otherwise I might spend two days trying to…
What this means is that it does not support things like acting instructions or creating a voice from a text description. If you prompt it with a matching text+voice sample it will be able to generate more speech based on more text, just like a TTS. It can also generate it's own text on the fly but it won't be as good as your frontier model.