Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I built a sub-500ms latency voice agent from scratch

ntik.me

Post reply on HN
Print view View original

81–90 of 168 posts

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#81

Post by eru »

nicktikhonov wrote: post #28
brody_hamer wrote: post #27

> Voice is a turn-taking problem It really feels to me like there’s some low hanging fruit with voice that no one is capitalizing on: filler words and pacing. When the llm notices a silence, it fills it with a contextually aware filler word while the real response generates. Just an “mhmm” or a “right, right”. It’d go so far to make the back and forth feel more like a conversation, and if the speaker wasn’t done spea…

100% - I thought about that shortly after writing this up. One way to make this work is to have a tiny, lower latency model generate that first reply out of a set of options, then aggressively cache TTS responses to get the latency super low. Responses like "Hmm, let me think about that..." would be served within milliseconds.

See also any public speaking who starts every answer to a question from the audience (or in a verbal interview) with something like 'that is a good question!' or "thank you for asking me that!"

Same strategy but employed by humans.

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#82

Post by eru »

fragmede wrote: post #53
phkahler wrote: post #37

Earlier quoted context omitted.

Better if it can anticipate its response before you're done speaking. That would be subject to change depending what the speaker says, but it might be able to start immediately.

it's bad enough how to deal with people that don't think before they speak now we gotta make the computers do it as well‽

Huh, the grandfather was suggestion to have the computer think while you speak.

That's different from banning the computer from thinking before they speak, ain't it?

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#83

Post by eru »

com2kid wrote: post #38
modeless wrote: post #14

IMO STT -> LLM -> TTS is a dead end. The future is end-to-end. I played with this two years ago and even made a demo you can install locally on a gaming GPU: https://github.com/jdarpinian/chirpy , but concluded that making something worth using for real tasks would require training of end-to-end models. A really interesting problem I would love to tackle, but out of my budget for a side project.

The advantage is being able to plug in new models to each piece of the pipeline. Is it super sexy? No. But each individual type of model is developing at a different rate (TTS moves really fast, low latency STT/ASR moved slower, LLMs move at a pretty good pace).

You should probably split it up: an end-to-end model for great latency (especially for baked in turn taking), but under the hood it can call out to any old text based model to answer more intricate question. You just need to teach the speech model to stall for a bit, while the LLM is busy.

Just use the same tricks humans are using for that.

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#84

Post by llbbdd »

stavros wrote: post #66
jedberg wrote: post #43

Earlier quoted context omitted.

If your Alexa did that, how quickly would you box it up and send it to me. :) I am serious though about having it sent to me: if anyone has an Alexa they no longer want, I'm happy to take it off your hands. I have eight and have never bought one. Having worked there I actually trust the security more than before I worked there. It was basically impossible for me, even as a Principle Engineer, to get copies of the Tex…

I'm puzzled by this conversation, because Amazon did get on the agent bandwagon with Alexa Plus (I have it, it's buggier than regular Alexa and it's all making me throw my Echos away since they can't even play Spotify reliably). Also, my Alexa does advertise stuff to me when I talk to it. It's not Budweiser, but it'll try to upsell me on Amazon services all the time.

I upgraded to Alexa+ and initially hated it but I've kept it because it's sooo much better at some things. This last December I bought a handful of smart plugs for my Christmas lights all around the house, and I did almost all the setup trivially over voice, e.g. fuzzy run-on stuff like this just worked on the first try:

- "Alexa, name the new unnamed outlet 'Living Room Lights', and the other unnamed one 'Stair Lights', then add them to a new group called 'Christmas Lights', and add the other three outlets as well"

- "Alexa, create a routine to turn off all the Christmas lights if there's nobody in the room and it's after 11pm"

- "Alexa, turn off all the Christmas lights except the tree in this room and the mantle"

That same fuzziness has definitely fucked up things that used to work more reliably like music playback though. Sometimes it works when I fall back to giving it more "robotic" commands in those cases but not always. They've also gone completely overboard with the cutesy responses because it's so trivial to do now ("I've set your spaghetti sauce timer for ten minutes. Happy to help with getting this evening's Italian-inspired dinner ready!")

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#85

Post by esperent »

jedberg wrote: post #75
vcxy wrote: post #74

Earlier quoted context omitted.

oh, interesting, I assumed the data came from interruptions (that seemed obvious) but I'm surprised you had some specific negative measurements. How do you decide the magnitude of the number? Just counting how long both parties are talking?

To be clear, it wasn't my research, I got it from studying some linguistics papers. But it was pretty straightforward. If I am talking, and then you interrupt, and 300ms later I stop talking, then the delay is -300ms. Same the other way. If I stop taking and then 300ms later you start talking, then the delay is 300ms. And if you start talking right when I stop, the delay is 0ms. You can get the info by just listening…

I assume there was a lot of variance? As in, some people interrupt others constantly and some do it rarely. Also probably a lot of adjustment depending on the situation, like depending on the relative status of the people, or when people are talking to a young child or non-native speaker.

All that to say, I'd imagine people are adaptable enough to easily handle 100ms+ delay when they know they're talking to an AI.

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#86

Post by russdill »

jedberg wrote: post #36

Oh, this is really interesting to me. This is what I worked on at Amazon Alexa (and have patents on). An interesting fact I learned at the time: The median delay between human speakers during a conversation is 0ms (zero). In other words, in many cases, the listener starts speaking before the speaker is done. You've probably experienced this, and you talk about how you "finish each other's sentences". It's because you…

I've experimented with having different sized LLMs cooperating. The smaller LLM starts a response while the larger LLM is starting. It's fed the initial response so it can continue it.

The idea of having an LLM follow and continuously predict the speaker. It would allow a response to be continually generated. If the prediction is correct, the response can be started with zero latency.

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#87

Post by Rohunyyy »

brody_hamer wrote: post #27

> Voice is a turn-taking problem It really feels to me like there’s some low hanging fruit with voice that no one is capitalizing on: filler words and pacing. When the llm notices a silence, it fills it with a contextually aware filler word while the real response generates. Just an “mhmm” or a “right, right”. It’d go so far to make the back and forth feel more like a conversation, and if the speaker wasn’t done spea…

I am not sure about the low hanging fruit. Its not easy to make something robotic more human. Based on personal experience I thought it would be a low hanging fruit for text. Take a simple LLM answer to anything and replace the "-" and "its not x its y" thingy that people almost always associate with LLMs to something else. Guess what? Now those answers sound even MORE robotic. Obviously this was a pet project that I cooked up in less than an hour but the more I tried to make it human the more it became ai

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#88

Post by russdill »

modeless wrote: post #14

IMO STT -> LLM -> TTS is a dead end. The future is end-to-end. I played with this two years ago and even made a demo you can install locally on a gaming GPU: https://github.com/jdarpinian/chirpy , but concluded that making something worth using for real tasks would require training of end-to-end models. A really interesting problem I would love to tackle, but out of my budget for a side project.

At least running things locally, such a model completely blows up your latency

Re: Show HN: I built a sub-500ms latency voice agent from scratch

#90

Post by DoctorOetker »

brody_hamer wrote: post #27

> Voice is a turn-taking problem It really feels to me like there’s some low hanging fruit with voice that no one is capitalizing on: filler words and pacing. When the llm notices a silence, it fills it with a contextually aware filler word while the real response generates. Just an “mhmm” or a “right, right”. It’d go so far to make the back and forth feel more like a conversation, and if the speaker wasn’t done spea…

1) if the system misdetected end-of-turn and has swiftly realized its error too late, and if we collect 90% of English syllables and find filler that starts with the syllable, it might allow to terminate the commitment to interrupt the speaker by turning it into background filler

2) if end-of-turn was detected very late, we can randomly select a first phonetic syllable, and then add it in the prompt that the reply should start with this syllable!

Post reply on HN