Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Noise cancellation improves turn-taking for AI Voice Agents

krisp.ai

Post reply on HN
Print view View original

41–47 of 47 posts

Re: Noise cancellation improves turn-taking for AI Voice Agents

#41

Post by BoredPositron »

dgellow wrote: post #38
bartman wrote: post #31

Earlier quoted context omitted.

I've dropped Krisp after they transitioned from their 'old' app to the 'new' app in the most confusing way that I've ever seen. We were paying for a business subscription and suddenly the 'new' app only seemed to worked with a new type of subscription, the 'old' app did not get updates to support the newest macOS versions for months, ... Since then we've been relying solely on Zoom's noise cancelling features and hav…

Yes, I really like NVIDIA broadcast. Unfortunately I work with macOS. I will look again at zoom, last time I tried their noise canceling it would cut my sentences

I had good results with the built in macOS noise canceling.

https://support.apple.com/guide/mac-help/use-mic-modes-on-yo...

Re: Noise cancellation improves turn-taking for AI Voice Agents

#43

Post by kazinator »

YetAnotherNick wrote: post #30
kazinator wrote: post #16

How about having the AI keep talking until you say a specific word like "stop". Background noise will rarely produce a false positive for that word.

They are solving the opposite problem, ie detecting when to start the AI. Stopping the AI is significantly easier problem.

In spite of your assiduously numerous re-readings of the fine article, it appears that the following passage has eluded your attention:

As a result, the VAD mistakenly interprets noise or background voices as active user speech, triggering unintended interruptions. These false triggers negatively impact turn-taking, a core component of natural, human-like conversational interactions.

Re: Noise cancellation improves turn-taking for AI Voice Agents

#44

Post by cubefox »

krisoft wrote: post #40
cubefox wrote: post #34

That's nice, but the main problem with current voice turn-taking is different. It's that these systems don't know when it is their turn to speak. When a human speaks to another, the second person will listen and interpret and guess when the first person is finished talking. For voice agents it doesn't work that way at all. The text-to-speech system just seems to have a hardcoded "pause" detector, e.g. 2 seconds, and…

> Instead of the rigid pause detector, it would actually make more sense for the end-of-message token to be sent when you explicitly say a specific phrase, like literally "over". That doesn’t sound very conversational at all. Instead one could train the network to recognise the appropriate turn-taking points. The simple way to do that is to make the model output a “listen a bit more” token when it is not yet their tu…

The "listen a bit more" token sounds interesting, but I'm not sure whether it would actually work better than the current solution which just waits for a sufficiently long pause. Maybe both could be combined.

Re: Noise cancellation improves turn-taking for AI Voice Agents

#45

Post by soulofmischief »

arthurcolle wrote: post #10
soulofmischief wrote: post #8

Earlier quoted context omitted.

What you're looking for is speaker embeddings. It's an embedding calculated from an audio snippet. As the other commenter mentioned, it should be combined with a robust voice isolation system. My own system automatically detects new speakers and tries to pick up on cues to identify the speaker, and once they are identified by name, the corresponding average embedding is inserted into a vector database so that the age…

Does this work well for multi-user scenarios? I also wanted to as a side effect tag and label people, but not really used to the audio setting. Just found "Speaker Verification with xvector embeddings on Voxceleb" which seems interesting and useful.

Within constraints, yes, it does, but I think there are many improvements I could still make. Speaker diarization and identification are ongoing subjects of research and right now there's not a good end-to-end model, so if your constraints are local inference only or low latency, it can be harder to get amazing results with current hardware and off-the-shelf models. It's still a lot better than nothing.

Re: Noise cancellation improves turn-taking for AI Voice Agents

#46

Post by khip »

dgellow wrote: post #29

Off topic: I hate so much what Krisp (the desktop app) has become. It was the perfect background noise cancelling tool, and now is asking permissions to record your whole screen and audio, for AI meeting companion features I couldn’t care about one bit. And they make it such a frustrating pain to opt-out, with constant modals tricking you in enabling all those AI features. Does anyone know an alternative that achieve…

If you use Linux, we ported the older version of Krisp used by Discord to run natively.

https://codeberg.org/khip/khip

Re: Noise cancellation improves turn-taking for AI Voice Agents

#47

Post by derf_ »

krisoft wrote: post #40
cubefox wrote: post #34

That's nice, but the main problem with current voice turn-taking is different. It's that these systems don't know when it is their turn to speak. When a human speaks to another, the second person will listen and interpret and guess when the first person is finished talking. For voice agents it doesn't work that way at all. The text-to-speech system just seems to have a hardcoded "pause" detector, e.g. 2 seconds, and…

> Instead of the rigid pause detector, it would actually make more sense for the end-of-message token to be sent when you explicitly say a specific phrase, like literally "over". That doesn’t sound very conversational at all. Instead one could train the network to recognise the appropriate turn-taking points. The simple way to do that is to make the model output a “listen a bit more” token when it is not yet their tu…

> Recognising the flow of a conversation is just pattern recognition. That is what machine learning is good at.

And surprisingly hard to do well in practice. My guess is that the problem is that there is very little information in your training dataset (because only the transition from "talking" to "done talking" matters), but the actual knowledge required to perform well is large (up to and including full speech recognition, in theory). So even with over a terabyte of training data, your choices are a small model that performs badly or a large(r) model that overfits severely.

It's possible there was something I was overlooking when I tried it, though. I couldn't think of a good way to confirm my guess experimentally.

Post reply on HN