Was looking for a demo of it on YouTube and fell over this hilarious one from a few months ago: https://youtu.be/coroLWOS7II?si=TeVghP_Zi0P9exQh . I’m sure it’s improved since :-)
Wow, it's so worth watching just for a laugh.
I'm sorry.
Moshi: A speech-text foundation model for real time dialogue
61–68 of 68 posts
Re: Moshi: A speech-text foundation model for real time dialogue
#62I've been building solutions for real-time voice -> llm -> voice output, and I think the most exciting part of what you're building is the streaming neural audio codec since you're never actually really able to stream STT with whisper. However from a product point of view I wouldn't necessarily want to pipe that into an LLM and have it reply, I think in a lot of use-cases there needs to be a tool/function calling ste…
> there needs to be a tool/function calling step before a reply
I built that almost exactly a year ago :) it was good but not fast enough - hence building the joint model.
Re: Moshi: A speech-text foundation model for real time dialogue
#63Earlier quoted context omitted.
This is awesome, are you contributing this to candle or is it a standalone package?
Just trying to stay focused on launching first ( https://docs.mixlayer.com ) and keeping early customers happy, but would love to open source some of this work. It'd probably be a separate crate from candle. If you haven't checked it out yet, mistral.rs implements some of these things ( https://github.com/EricLBuehler/mistral.rs ). Eric hasn't done multi-GPU inference yet, but I know it's on his roadmap. Not sure if…
Hey, mixlayer is really cool.
I also have a Rust LLM inference project. The overlap is very high between what mixlayer is doing and what my project is doing. It's actually crazy how we basically have the same features. [1] Right now I'm still using llama.cpp on the backend, but eventually want to move to candle via mistral.rs.
Re: Moshi: A speech-text foundation model for real time dialogue
#64Do app running in an a-shell terminal on the iPad have a convenient way provide a tts interface?
Re: Moshi: A speech-text foundation model for real time dialogue
#65Was looking for a demo of it on YouTube and fell over this hilarious one from a few months ago: https://youtu.be/coroLWOS7II?si=TeVghP_Zi0P9exQh . I’m sure it’s improved since :-)
this video made my day, thanks for posting it
Re: Moshi: A speech-text foundation model for real time dialogue
#66After a quick glance, I was curious about the 3 "inference stacks" for PyTorch, Rust, and MLX. Unsurprising there's a Rust version given who Kyutai's CTO is. But a quick question for him or anyone else who knows: was a standalone Rust version trained purely from scratch (Candle?), or was there just one training regime in PyTorch?
Re: Moshi: A speech-text foundation model for real time dialogue
#67Earlier quoted context omitted.
Super interested. Do you have an equivalent of vLLM? Did you have to rewrite batching, paged attention…?
Yeah, I’ve had to rewrite continuous batching and other scheduling logic. That and multi-GPU inference have been the hardest things to build. I’ll need to get paged attention working as well, but I think I can launch without it.
Are you aiming for Nvidia hardware with rust-cuda, or looking to integrate with non-Nvidia hardware?
Re: Moshi: A speech-text foundation model for real time dialogue
#68Earlier quoted context omitted.
Yeah, I’ve had to rewrite continuous batching and other scheduling logic. That and multi-GPU inference have been the hardest things to build. I’ll need to get paged attention working as well, but I think I can launch without it.
Are you aiming for Nvidia hardware with rust-cuda, or looking to integrate with non-Nvidia hardware?
We used candle[0], which uses cudarc and the metal crate under the hood. That means we run on nvidia hardware in production and can test locally on macbooks with smaller models.
I would certainly like to use non nvidia hardware but at this point it's not a priority. The subset of tensor operations needed to run the forward pass of LLMs isn't as large as you'd think though.