Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Real-time AI Voice Chat at ~500ms Latency

github.com

Post reply on HN
Print view View original

61–70 of 238 posts

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#61

Post by thamer »

fintechie wrote: post #40

Quite good, it would sound much better with SOTA voices though: https://github.com/nari-labs/dia

Does Dia support configuring voices now? I looked at it when it was first released, and you could only specify [S1] [S2] for the speakers, but not how they would sound.

There was also a very prominent issue where the voices would be sped up if the text was over a few sentences long; the longer the text, the faster it was spoken. One suggestion was to split the conversation into chunks with only one or two "turns" per speaker, but then you'd hear two voices then two more, then two more… with no way to configure any of it.

Dia looked cool on the surface when it was released, but it was only a demo for now and not at all usable for any real use case, even for a personal app. I'm sure they'll get to these issues eventually, but most comments I've seen so far recommending it are from people who have not actually used it or they would know of these major limitations.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#63

Post by woodson »

jedberg wrote: post #35

I did some research into this about a year ago. Some fun facts I learned: - The median delay between speakers in a human to human conversation is zero milliseconds. In other words, about 1/2 the time, one speaker interrupts the other, making the delay negative. - Humans don't care about delays when speaking to known AIs. They assume the AI will need time to think. Most users will qualify a 1000ms delay is acceptable…

Human-to-human conversational patterns are highly specific to cultural and contextual aspects. Sounds like I’m stating the obvious, but developers regularly disregard that and then wonder why things feel unnatural for users. The “median delay” may not be the most useful thing to look at.

To properly learn more appropriate delays, it can be useful to find a proxy measure that can predict when a response can/should be given. For example, look at Kyutai’s use of change in perplexity in predictions from a text translation model for developing simultaneous speech-to-speech translation (https://github.com/kyutai-labs/hibiki).

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#64

Post by Reason077 »

jedberg wrote: post #35

I did some research into this about a year ago. Some fun facts I learned: - The median delay between speakers in a human to human conversation is zero milliseconds. In other words, about 1/2 the time, one speaker interrupts the other, making the delay negative. - Humans don't care about delays when speaking to known AIs. They assume the AI will need time to think. Most users will qualify a 1000ms delay is acceptable…

The best, most human-like AI voice chat I've seen yet is Sesame (www.sesame.com). It has delays, but fills them very naturally with normal human speech nuances like "hmmm", "uhhh", "hold on while I look that up" etc. If there's a longer delay it'll even try to make a bit of small talk, just like a human conversation partner might.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#65

Post by awesome_dude »

coolspot wrote: post #58
varispeed wrote: post #52

Earlier quoted context omitted.

> AI in KFC drive thrus That right here is an anxiety trigger and would make me skip the place. There is nothing more ruining the day like arguing with a robot who keeps misinterpreting what you said.

They have a fallback to a human operator when stopwords and/or stop conditions are detected.

That right here is an anxiety trigger and would make me skip the place.

There is nothing more ruining the day like arguing with a HUMAN OPERATOR who keeps misinterpreting what you said.

:-)

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#67

Post by amelius »

awesome_dude wrote: post #65
coolspot wrote: post #58

Earlier quoted context omitted.

They have a fallback to a human operator when stopwords and/or stop conditions are detected.

That right here is an anxiety trigger and would make me skip the place. There is nothing more ruining the day like arguing with a HUMAN OPERATOR who keeps misinterpreting what you said. :-)

Maybe talk to the chicken operator then.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#68

Post by com2kid »

jedberg wrote: post #35

I did some research into this about a year ago. Some fun facts I learned: - The median delay between speakers in a human to human conversation is zero milliseconds. In other words, about 1/2 the time, one speaker interrupts the other, making the delay negative. - Humans don't care about delays when speaking to known AIs. They assume the AI will need time to think. Most users will qualify a 1000ms delay is acceptable…

A lot better techniques exist now days than pure silence detection -

1. A special model that predicts when a conversation turn is coming up (e.g. when someone is going to stop speaking). Speech has a rhythm to it and pauses / ends of speech are actually predictable.

2. Generate a model response for every subsequent word that comes in (and throw away the previously generated response), so basically your time to speak after doing some other detection is basically zero.

3. Ask an LLM what it thinks the odds of the user being done talking is, and if it is a high probability, reduce delay timer down. (The linked repo does this)

I don't know of any up to date models for #1 but I haven't checked in over a year.

Tl;Dr the solution to problems involving AI models is more AI models.

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#69

Post by kabirgoel »

This is great. Poking into the source, I find it interesting that the author implemented a custom turn detection strategy, instead of using Silero VAD (which is standard in the voice agents space). I’m very curious why they did it this way and what benefits they observed.

For folks that are curious about the state of the voice agents space, Daily (the WebRTC company) has a great guide [1], as well as an open-source framework that allows you to build AI voice chat similar to OP's with lots of utilities [2].

Disclaimer: I work at Cartesia, which services a lot of these voice agents use cases, and Daily is a friend.

[1]: https://voiceaiandvoiceagents.com [2]: https://docs.pipecat.ai/getting-started/overview

Re: Show HN: Real-time AI Voice Chat at ~500ms Latency

#70

Post by amelius »

LZ_Khan wrote: post #21
smusamashah wrote: post #4

Saying this as a user of these tools (openai, Google voice chat etc). These are fast yes, but they don't allow talking naturally with pauses. When we talk, we take long and small pauses for thinking or for other reasons. With these tools, AI starts taking as soon as we stop. Happens both in text and voice chat tools. I saw a demo on twitter a few weeks back where AI was waiting for the person to actually finish what…

Honestly I think this is a problem of over-engineering and simply allowing the user to press a button when he wants to start talking and press it when he's done is good enough. Or even a codeword for start and finish. We don't need to feel like we're talking to a real person yet.

Or give the AI an Asian accent. If you're talking on the phone to someone on a different continent you accept the delay, so why not here.
Post reply on HN