Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hertz-dev, the first open-source base model for conversational audio

si.inc

Post reply on HN
Print view View original

21–30 of 58 posts

Re: Hertz-dev, the first open-source base model for conversational audio

#21

Post by a2128 »

blixt wrote: post #16

They say Hertz is first of its kind but Moshi is another duplex audio model from earlier this year that seems to perform similarly (and it runs on a MacBook): https://github.com/kyutai-labs/moshi

Moshi never released the base model, only two conversationally finetuned models. They also never released training code except for the codec. Though I don't see any training code for Hertz either, just 3 inference notebooks, and model code full of no_grad. No paper either to help me understand how this was trained and what the architecture is like. So I'm not too sure about researcher-friendliness unless I'm missing something.

Re: Hertz-dev, the first open-source base model for conversational audio

#22

Post by nicholas-cc »

a2128 wrote: post #21
blixt wrote: post #16

They say Hertz is first of its kind but Moshi is another duplex audio model from earlier this year that seems to perform similarly (and it runs on a MacBook): https://github.com/kyutai-labs/moshi

Moshi never released the base model, only two conversationally finetuned models. They also never released training code except for the codec. Though I don't see any training code for Hertz either, just 3 inference notebooks, and model code full of no_grad. No paper either to help me understand how this was trained and what the architecture is like. So I'm not too sure about researcher-friendliness unless I'm missing…

We're working on a HuggingFace release that will help with finetuning. We'd like to do a paper, after a larger release - we're a team of 4.

Re: Hertz-dev, the first open-source base model for conversational audio

#23

Post by netdevnet »

reissbaker wrote: post #13

This is really cool. FWIW, existing open-source TTS engines are really bad in comparison to what you have here: I know this is voice-to-voice, but I think there'd be a lot of appetite to get this to also be multimodal and accept text (essentially making it a really good TTS model, in addition to a great voice-to-voice model). I suppose someone could hack their way around the problem by finetuning it to essentially re…

They are a team of 4. At that size, it's better for them to be focused on one thing than stretched out

Re: Hertz-dev, the first open-source base model for conversational audio

#24

Post by netdevnet »

nicholas-cc wrote: post #22
a2128 wrote: post #21

Earlier quoted context omitted.

Moshi never released the base model, only two conversationally finetuned models. They also never released training code except for the codec. Though I don't see any training code for Hertz either, just 3 inference notebooks, and model code full of no_grad. No paper either to help me understand how this was trained and what the architecture is like. So I'm not too sure about researcher-friendliness unless I'm missing…

We're working on a HuggingFace release that will help with finetuning. We'd like to do a paper, after a larger release - we're a team of 4.

Very impressive for just 4 people. What's the team background and how long have you been working on this?

Re: Hertz-dev, the first open-source base model for conversational audio

#25

Post by imjonse »

blixt wrote: post #14

Gotta say I was confused for a second but yeah apparently si.inc and ssi.inc are the domains for two different AGI companies and I can only assume it’s intentional?

According to whois records si.inc was registered 5 days after ssi.inc in June. So yes, maybe intentional.

Re: Hertz-dev, the first open-source base model for conversational audio

#27

Post by unit149 »

netdevnet wrote: post #24
nicholas-cc wrote: post #22

Earlier quoted context omitted.

We're working on a HuggingFace release that will help with finetuning. We'd like to do a paper, after a larger release - we're a team of 4.

Very impressive for just 4 people. What's the team background and how long have you been working on this?

For a rag-tag group of transcendental audiophiles operating electronic circuitry, it ionizes and atomizes well.

Re: Hertz-dev, the first open-source base model for conversational audio

#28

Post by codedokode »

The voice sounds a little bit distorted, and there is often a noise in the background (especially noticeable when this noise disappears when the voice pauses). I wonder, is it model limitations or is it the problem with quality of training data?

Re: Hertz-dev, the first open-source base model for conversational audio

#30

Post by PenisBanana »

reissbaker wrote: post #13

This is really cool. FWIW, existing open-source TTS engines are really bad in comparison to what you have here: I know this is voice-to-voice, but I think there'd be a lot of appetite to get this to also be multimodal and accept text (essentially making it a really good TTS model, in addition to a great voice-to-voice model). I suppose someone could hack their way around the problem by finetuning it to essentially re…

Yes, yes. This. Piper is already pretty good . . . and then this.

It may not be _them_ doing it, though.

Post reply on HN