Anything more about the company, founders, affiliations..?
Hertz-dev, the first open-source base model for conversational audio
41–50 of 58 posts
Re: Hertz-dev, the first open-source base model for conversational audio
#42Anything more about the company, founders, affiliations..?
Some commits are by `nicholascc` (https://github.com/nicholascc); via Twitter, he seems to be Nicholas Charette. Nicholas is a first year student at Stanford. For such a young group, this is a really impressive effort!
Re: Hertz-dev, the first open-source base model for conversational audio
#43Earlier quoted context omitted.
Hertz-dev is a base model, meaning it's just trained to predict the next token of audio. If your prompt is an old male voice with a British accent, the model will most likely continue speaking in an old male voice with a British accent. Being a base model, hertz-dev is easily finetunable for specific tasks - it would be a simple change to add manual configurations for the gender/age/accent.
I'm interested to hear more detail about approaches to adding manual controls for speaker characteristics or emotion or other things you might want to vary. What techniques do you have in mind?
I’ll jump in here - as a former new englander, the cheerful helping tone of all modern voice llms infuriates me. And the slow speed. And the over explanations. ChatGPT advanced can be induced to talk more quickly, less sycophantically and if I like in a not-bad regional accent; essentially I want it to mirror my tone better. But those inducements don’t stick between sessions.
On the technical side having some sort of continuation or summarization loop on seems interesting to me as a product feature. It’s not enough to build a company off of though. But it would be nice.
Re: Hertz-dev, the first open-source base model for conversational audio
#44Earlier quoted context omitted.
They are a team of 4. At that size, it's better for them to be focused on one thing than stretched out
Eh, that depends. A small model that's voice-and-text is probably more useful to most people than scaling up a voice-only model: the large voice-only model will have to compete on intelligence with e.g. Qwen and Llama, since it can't be used in conjunction with them; whereas a small voice+text model can be used as a cheap frontend hiding a larger, smarter, but more expensive text-only model behind it. This is an 8b m…
Their goal is not working on what's most useful for most people though. That's the domain of the big AI players. They are small and so specialising works best as that's where they can have an edge as a company.
At the end of the day, the released product needs to be good and needs to be done in a reasonable amount of time. I highly doubt they can do a generic model as well as a more specialised one.
But if you think you know better than them, you could try to contact them even though it looks they are crazy laser focused (their public email addresses are either for investors or employee candidates).
Re: Hertz-dev, the first open-source base model for conversational audio
#45Cool, looks like this is trained on 16 million hours of audio (500B tokens at ~.11 seconds per token).
Even the large open source TTS models (see F5 TTS, Mask GCT) are mostly trained on very small audio datasets (say 100k hours) relative to the amount of audio available on the internet, so it's cool to see an open source effort to scale up training significantly.
Re: Hertz-dev, the first open-source base model for conversational audio
#46Can one of the authors explain what this actually means from the post?
hertz-vae: a 1.8 billion parameter transformer decoder which acts as a learned prior for the audio VAE. The model uses a context of 8192 sampled latent representations (17 minutes) and predicts the next encoded audio frame as a mixture of gaussians. 15 bits of quantized information from the next token act as semantic scaffolding to steer the generation in a streamable manner.
Re: Hertz-dev, the first open-source base model for conversational audio
#47Earlier quoted context omitted.
We're working on a HuggingFace release that will help with finetuning. We'd like to do a paper, after a larger release - we're a team of 4.
Very impressive for just 4 people. What's the team background and how long have you been working on this?
I'm not part of their team, but lived with them for a couple months. They've been working on it for ~5 months, and their background is 16-20 year olds who are too smart for university.
Re: Hertz-dev, the first open-source base model for conversational audio
#48Can one of the authors explain what this actually means from the post? hertz-vae: a 1.8 billion parameter transformer decoder which acts as a learned prior for the audio VAE. The model uses a context of 8192 sampled latent representations (17 minutes) and predicts the next encoded audio frame as a mixture of gaussians. 15 bits of quantized information from the next token act as semantic scaffolding to steer the gener…
[deleted]
Re: Hertz-dev, the first open-source base model for conversational audio
#49Gotta say I was confused for a second but yeah apparently si.inc and ssi.inc are the domains for two different AGI companies and I can only assume it’s intentional?
According to whois records si.inc was registered 5 days after ssi.inc in June. So yes, maybe intentional.
But the company si.inc (Standard Intelligence) was founded many months before ssi.inc (Safe Superintelligence), so they likely just didn't want their domain name to get taken.
Re: Hertz-dev, the first open-source base model for conversational audio
#50Tesla’s approach to pure vision-based autonomous driving—temporarily setting aside lidar and other sensors—seems designed to make this technology more accessible and scalable. By focusing on a vision-only model, they can accelerate adoption and gather large datasets for quicker iterations. Once the vision-based system reaches a mature stage, I imagine Tesla might reintegrate additional sensor data, like lidar or rada…
I think you're describing ChatGPT Advanced Voice Mode (or Realtime API) in your second paragraph.
They were so busy inventing they forgot to do a basic Google search to see if it had already been done.