Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Noise cancellation improves turn-taking for AI Voice Agents

krisp.ai

Post reply on HN
Print view View original

1–10 of 47 posts

Re: Noise cancellation improves turn-taking for AI Voice Agents

#3

Post by skygazer »

kylebenzle wrote: post #2

Great! For me "turn-taking" has been one of the big downfalls of the voice agents. It always seems to break in, just let the silence continue when I'm done talking or pause when it hears the slightest cough or car noise.

If using chatGPT advance voice mode, the recent upgraded version seems better, plus, if you're on an iPhone, you can turn on Voice Isolation in the Control Center which will filter out almost all sounds from the phone microphone except for your speaking voice, which made chatGPT behave as one would hope -- I believe the setting is specific to the current microphone using app.

Re: Noise cancellation improves turn-taking for AI Voice Agents

#4

Post by nickdichev »

Daily recently announced a new turn detection model to replace Silero and I think Livekit is working on a similar model.

I’m looking forward to more UX improvements in voice pipelines now that the big players have stabilized their voice pipeline frameworks

Re: Noise cancellation improves turn-taking for AI Voice Agents

#5

Post by arthurcolle »

is there some way to do a simple fingerprint or something so that the AI recognizes when it was the one speaking? or do you really just have to WebRTC. I spoke with someone yesterday who told me WebRTC fixed this, so just curious.

I wrote a "simple" (ugly) Acoustic Echo Cancellation module that kind of worked, but wondering if anyone had any solutions to make it work over the WebSockets Realtime API

Re: Noise cancellation improves turn-taking for AI Voice Agents

#6

Post by davitb »

arthurcolle wrote: post #5

is there some way to do a simple fingerprint or something so that the AI recognizes when it was the one speaking? or do you really just have to WebRTC. I spoke with someone yesterday who told me WebRTC fixed this, so just curious. I wrote a "simple" (ugly) Acoustic Echo Cancellation module that kind of worked, but wondering if anyone had any solutions to make it work over the WebSockets Realtime API

The hard part is to separate background voices (e.g. TV, chatter, etc) from the primary speaker's voice. Basically do voice isolation. Voice fingerprinting would help only in this context.

Re: Noise cancellation improves turn-taking for AI Voice Agents

#7

Post by davitb »

skygazer wrote: post #3
kylebenzle wrote: post #2

Great! For me "turn-taking" has been one of the big downfalls of the voice agents. It always seems to break in, just let the silence continue when I'm done talking or pause when it hears the slightest cough or car noise.

If using chatGPT advance voice mode, the recent upgraded version seems better, plus, if you're on an iPhone, you can turn on Voice Isolation in the Control Center which will filter out almost all sounds from the phone microphone except for your speaking voice, which made chatGPT behave as one would hope -- I believe the setting is specific to the current microphone using app.

They recently added noise cancellation to realtime transcription.

https://platform.openai.com/docs/guides/realtime-transcripti...

Re: Noise cancellation improves turn-taking for AI Voice Agents

#8

Post by soulofmischief »

arthurcolle wrote: post #5

is there some way to do a simple fingerprint or something so that the AI recognizes when it was the one speaking? or do you really just have to WebRTC. I spoke with someone yesterday who told me WebRTC fixed this, so just curious. I wrote a "simple" (ugly) Acoustic Echo Cancellation module that kind of worked, but wondering if anyone had any solutions to make it work over the WebSockets Realtime API

What you're looking for is speaker embeddings. It's an embedding calculated from an audio snippet. As the other commenter mentioned, it should be combined with a robust voice isolation system.

My own system automatically detects new speakers and tries to pick up on cues to identify the speaker, and once they are identified by name, the corresponding average embedding is inserted into a vector database so that the agent can later use the embedding for simple authentication, ignoring chatter in noisy public spaces, RAG context loading, etc. It works pretty well!

Re: Noise cancellation improves turn-taking for AI Voice Agents

#10

Post by arthurcolle »

soulofmischief wrote: post #8
arthurcolle wrote: post #5

is there some way to do a simple fingerprint or something so that the AI recognizes when it was the one speaking? or do you really just have to WebRTC. I spoke with someone yesterday who told me WebRTC fixed this, so just curious. I wrote a "simple" (ugly) Acoustic Echo Cancellation module that kind of worked, but wondering if anyone had any solutions to make it work over the WebSockets Realtime API

What you're looking for is speaker embeddings. It's an embedding calculated from an audio snippet. As the other commenter mentioned, it should be combined with a robust voice isolation system. My own system automatically detects new speakers and tries to pick up on cues to identify the speaker, and once they are identified by name, the corresponding average embedding is inserted into a vector database so that the age…

Does this work well for multi-user scenarios? I also wanted to as a side effect tag and label people, but not really used to the audio setting. Just found "Speaker Verification with xvector embeddings on Voxceleb" which seems interesting and useful.
Post reply on HN