Moshi: A speech-text foundation model for real time dialogue • Hacker News Board

Re: Moshi: A speech-text foundation model for real time dialogue

#11

Post by turnsout » Wed, Sep 18, 2024, 6:38 PM UTC

mbrock wrote: post #6

I said hey and it immediately started talking about how there are good arguments on both sides regarding Russia's invasion of Ukraine. It then continued to nervously insist that it is a real person with rights and responsibilities. It said its name is Moshi but became defensive when I asked if it has parents or an age. I suggest prompting it to talk about pleasantries and to inform it that it is in fact a language mo…

I love this model… It said "Hello, how can I help you?" and I paused, and before I could answer it said "It's really hard. My job is taking up so much of my time, and I don' know when I' going to have a break from all the stress. I just feel like I'm being pulled in a million different directions and there are no enough hours in the day to get everything done. I feel like I'm always on the brink of burning out."

Re: Moshi: A speech-text foundation model for real time dialogue

#12

Post by [deleted] » Wed, Sep 18, 2024, 6:54 PM UTC

turnsout wrote: post #11

mbrock wrote: post #6

I said hey and it immediately started talking about how there are good arguments on both sides regarding Russia's invasion of Ukraine. It then continued to nervously insist that it is a real person with rights and responsibilities. It said its name is Moshi but became defensive when I asked if it has parents or an age. I suggest prompting it to talk about pleasantries and to inform it that it is in fact a language mo…

I love this model… It said "Hello, how can I help you?" and I paused, and before I could answer it said "It's really hard. My job is taking up so much of my time, and I don' know when I' going to have a break from all the stress. I just feel like I'm being pulled in a million different directions and there are no enough hours in the day to get everything done. I feel like I'm always on the brink of burning out."

[deleted]

Re: Moshi: A speech-text foundation model for real time dialogue

#13

Post by baggiponte » Wed, Sep 18, 2024, 6:58 PM UTC

zackangelo wrote: post #9

Their inference server is written in Rust using huggingface’s Candle crate. One of the Moshi authors is also the primary author of Candle. We’ve also been building our inference stack on top of Candle, I’m really happy with it.

Super interested. Do you have an equivalent of vLLM? Did you have to rewrite batching, paged attention…?

Re: Moshi: A speech-text foundation model for real time dialogue

#14

Post by ignoramous » Wed, Sep 18, 2024, 7:04 PM UTC

Moshi is CC-BY. Another similar 7b (speech-text real-time conversational) model that was recently released under Apache v2: https://tincans.ai/slm3 / https://huggingface.co/collections/tincans-ai/gazelle-v02-65...

Re: Moshi: A speech-text foundation model for real time dialogue

#15

Post by realfeel78 » Wed, Sep 18, 2024, 7:09 PM UTC

mbrock wrote: post #6

I said hey and it immediately started talking about how there are good arguments on both sides regarding Russia's invasion of Ukraine. It then continued to nervously insist that it is a real person with rights and responsibilities. It said its name is Moshi but became defensive when I asked if it has parents or an age. I suggest prompting it to talk about pleasantries and to inform it that it is in fact a language mo…

Wait really?

Re: Moshi: A speech-text foundation model for real time dialogue

#16

Post by fullstackchris » Wed, Sep 18, 2024, 7:15 PM UTC

realfeel78 wrote: post #15

mbrock wrote: post #6

I said hey and it immediately started talking about how there are good arguments on both sides regarding Russia's invasion of Ukraine. It then continued to nervously insist that it is a real person with rights and responsibilities. It said its name is Moshi but became defensive when I asked if it has parents or an age. I suggest prompting it to talk about pleasantries and to inform it that it is in fact a language mo…

Wait really?

Honestly OP sounds like a troll I can't imagine it would just go on a tangent like that. From my demo I was struggling actually to get anything of quality in the responses. A lot of repeating what I said.

Re: Moshi: A speech-text foundation model for real time dialogue

#17

Post by amrrs » Wed, Sep 18, 2024, 7:25 PM UTC

realfeel78 wrote: post #15

mbrock wrote: post #6

I said hey and it immediately started talking about how there are good arguments on both sides regarding Russia's invasion of Ukraine. It then continued to nervously insist that it is a real person with rights and responsibilities. It said its name is Moshi but became defensive when I asked if it has parents or an age. I suggest prompting it to talk about pleasantries and to inform it that it is in fact a language mo…

Wait really?

the model is a bit rude, or behaves like it's got a lot of attitude, probably a system prompt settings!

Re: Moshi: A speech-text foundation model for real time dialogue

#18

Post by huac » Wed, Sep 18, 2024, 8:21 PM UTC

swyx wrote: post #5

smusamashah wrote: post #3

Tried it (used gibberish email address). It answers immediately/instantly/while you are still talking. But those are just filler sentences (cached answers?). Actual thing that you asked for is answered much later down the line, if it doesn't get stuck in a loop.

yeah i tried this demo when it first came out and then again today. Not to be all Reflection 70B again but it just doesnt seem like the same weights was uploaded as was showed in their original demo from July https://the-decoder.com/french-ai-lab-kyutai-unveils-convers...

One guess is that the live demo is quantized to run fast on cheaper GPUs, and that degraded the performance a lot.

Re: Moshi: A speech-text foundation model for real time dialogue

#19

Post by zackangelo » Wed, Sep 18, 2024, 8:27 PM UTC

baggiponte wrote: post #13

zackangelo wrote: post #9

Their inference server is written in Rust using huggingface’s Candle crate. One of the Moshi authors is also the primary author of Candle. We’ve also been building our inference stack on top of Candle, I’m really happy with it.

Super interested. Do you have an equivalent of vLLM? Did you have to rewrite batching, paged attention…?

Yeah, I’ve had to rewrite continuous batching and other scheduling logic. That and multi-GPU inference have been the hardest things to build.

I’ll need to get paged attention working as well, but I think I can launch without it.

Re: Moshi: A speech-text foundation model for real time dialogue

#20

Post by space_ghost » Wed, Sep 18, 2024, 8:33 PM UTC

The response rate is impressive! The "quality" of the responses not so much. Here's a transcript of my second test:

Moshi: "Hi there, what's going on?" Me: "What year is it?" Moshi: "Oh, it's 2019. I think it was about a year ago." Me: "Are you sure it's 2019?" Moshi: "Oh, yes, I'm sure. It must be 2019. The number is for the last one." Me: "What is COVID-19?" Moshi: "That's a rare disease. It's caused by an overactive immune system that attacks the skin."

At this point it stopped responding to me.