what is up with the first sample? and/or am I having a stroke
Hertz-dev, the first open-source base model for conversational audio
11–20 of 58 posts
Re: Hertz-dev, the first open-source base model for conversational audio
#12Earlier quoted context omitted.
So essentially this is voice input to voice output? Can you change gender/age/accent? Does it track prosodic information? I've been waiting for something like this.
Hertz-dev is a base model, meaning it's just trained to predict the next token of audio. If your prompt is an old male voice with a British accent, the model will most likely continue speaking in an old male voice with a British accent. Being a base model, hertz-dev is easily finetunable for specific tasks - it would be a simple change to add manual configurations for the gender/age/accent.
I assume this mirroring is due to symmetry being more typical than not among the training data, and if instead trained with contrived diversity (e.g., males only conversing with females) then the output of the base model would follow suit without pulling any levers?
It's interesting to think about what complete diversity (i.e., no tendencies toward homogeneous conversation partners whatsoever among training data) would yield, given that it's trying to deliver whatever is most probable.
Re: Hertz-dev, the first open-source base model for conversational audio
#13This is really cool. FWIW, existing open-source TTS engines are really bad in comparison to what you have here: I know this is voice-to-voice, but I think there'd be a lot of appetite to get this to also be multimodal and accept text (essentially making it a really good TTS model, in addition to a great voice-to-voice model).
I suppose someone could hack their way around the problem by finetuning it to essentially replay Piper (or whatever) output, only with more natural prosody and intonation. And then have the text LLM pipe to Piper, and Piper pipe to Hertz-dev. But it would be pretty useful to have it accept text natively!
Re: Hertz-dev, the first open-source base model for conversational audio
#14Gotta say I was confused for a second but yeah apparently si.inc and ssi.inc are the domains for two different AGI companies and I can only assume it’s intentional?
Re: Hertz-dev, the first open-source base model for conversational audio
#15Congrats, team. Does Hertz support multi-lingual audio right now?
Yes
Re: Hertz-dev, the first open-source base model for conversational audio
#16They say Hertz is first of its kind but Moshi is another duplex audio model from earlier this year that seems to perform similarly (and it runs on a MacBook):
https://github.com/kyutai-labs/moshi
Re: Hertz-dev, the first open-source base model for conversational audio
#17They say Hertz is first of its kind but Moshi is another duplex audio model from earlier this year that seems to perform similarly (and it runs on a MacBook): https://github.com/kyutai-labs/moshi
Moshi is a good model to build chat applications on, this is designed to be more of a proper base model with all the quirkiness, naturalness, and researcher-friendliness of base modeling.
Re: Hertz-dev, the first open-source base model for conversational audio
#18Earlier quoted context omitted.
So essentially this is voice input to voice output? Can you change gender/age/accent? Does it track prosodic information? I've been waiting for something like this.
Hertz-dev is a base model, meaning it's just trained to predict the next token of audio. If your prompt is an old male voice with a British accent, the model will most likely continue speaking in an old male voice with a British accent. Being a base model, hertz-dev is easily finetunable for specific tasks - it would be a simple change to add manual configurations for the gender/age/accent.
I'm interested to hear more detail about approaches to adding manual controls for speaker characteristics or emotion or other things you might want to vary. What techniques do you have in mind?
Re: Hertz-dev, the first open-source base model for conversational audio
#19the One-channel generation seems to be speaking gibberish english. I'm not sure what it is supposed to represent?
And is the interactive generation just doing an ELIZA? i.e. "P: tell us about how AI will be interesting", "A: Yeah AI will, yeah, be interesting".
Re: Hertz-dev, the first open-source base model for conversational audio
#20what is up with the first sample? and/or am I having a stroke
Pay attention to the given prompt length in the examples. The first 2 seconds of the first example is a real human speaking. Everything after is generated by the model. It produces what almost sounds like real human speech mimicking the voice of the input but it's currently at a level of something like GPT-2 in terms of meaningful words.