Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Open source framework OpenAI uses for Advanced Voice

github.com

Post reply on HN
Print view View original

41–50 of 63 posts

Re: Show HN: Open source framework OpenAI uses for Advanced Voice

#41

Post by andrew_eu »

ensignavenger wrote: post #35
TZubiri wrote: post #32

Earlier quoted context omitted.

As costs of humanlike communications decrease, so will Sybil attacks and spam. The IRS is notorious for resistance to tech change, don't be surprised if they unplug the phones and force you to walk in to ask your question. What is the value add here? Save sometime for technocrats and technoadjacents for a whole of 3 years before victims of spam adapt? Also this has been solved already just mail your question like the…

It would be really nice if the IRS would ALLOW you to walk in and ask a question!

Years ago my tax return was flagged as a possible fraud case -- I believe a direct consequence of a big data breach. I had to go into my "local" IRS office and present my passport to prove indeed it was me. Decidedly not nice.

True to form, with an appointment I waited 3 hours at the office and watched the guard staff turn away countless people. Finally saw a person, gave then my passport, and finished in a minute.

Re: Show HN: Open source framework OpenAI uses for Advanced Voice

#42

Post by fidotron »

throw14082020 wrote: post #25

This is really helpful, thanks! OpenAI hired the ex fractional CTO of LiveKit, who created Pion, a popular WebRTC library/tool. I'd expect OpenAI to migrate off of LiveKit within 6 months. LiveKit is too expensive. Also, WebRTC is hard, and OpenAI now being a less open company will want to keep improvements to itself. Not affiliated with any competitors, but I did work at a PaaS company similar to LiveKit but used We…

> LiveKit is too expensive

Most of it is open source, especially the clients, although they do feel quite ad hoc hacked together (a possible side effect of WebRTC evolution).

Would totally agree on OpenAI moving away. The description of the agent here sounds like a big hack just to get around the fact temporarily the model server expects audio over sockets instead.

Re: Show HN: Open source framework OpenAI uses for Advanced Voice

#43

Post by ValentinA23 »

balloob wrote: post #22
rukuu001 wrote: post #20

Earlier quoted context omitted.

Working on this for an internal tool - detecting no speech has been a PITA so far. Interested to see how you go with this.

Use the voice activity detector we wrote for Home Assistant. It works very well: https://github.com/rhasspy/pymicro-vad

What if I'm watching TV and use the AI to control it ? It should only react to my voice (a problem I had that forced me to use a wake word).

Re: Show HN: Open source framework OpenAI uses for Advanced Voice

#44

Post by russ »

spuz wrote: post #23

Is there anyone besides OpenAI working on a speech to speech model? I find it incredibly useful and it's the sole reason that I pay for their service but I do find it very limited. I'd be interested to know if any other groups are doing research on voice models.

There’s Ultravox as well (from one of the creators of WebRTC): https://github.com/fixie-ai/ultravox

Their model builds a speech-to-speech layer into Llama. Last I checked they have the audio-in part working and they’re working on the audio-out piece.

Re: Show HN: Open source framework OpenAI uses for Advanced Voice

#45

Post by russ »

fidotron wrote: post #42
throw14082020 wrote: post #25

This is really helpful, thanks! OpenAI hired the ex fractional CTO of LiveKit, who created Pion, a popular WebRTC library/tool. I'd expect OpenAI to migrate off of LiveKit within 6 months. LiveKit is too expensive. Also, WebRTC is hard, and OpenAI now being a less open company will want to keep improvements to itself. Not affiliated with any competitors, but I did work at a PaaS company similar to LiveKit but used We…

> LiveKit is too expensive Most of it is open source, especially the clients, although they do feel quite ad hoc hacked together (a possible side effect of WebRTC evolution). Would totally agree on OpenAI moving away. The description of the agent here sounds like a big hack just to get around the fact temporarily the model server expects audio over sockets instead.

Which components feel ad hoc?

In most real applications, the agent has additional logic (function calling, RAG, etc) than simply relaying a stream to the model server. In those cases, you want it to be a separate service/component that can be independently scaled.

Re: Show HN: Open source framework OpenAI uses for Advanced Voice

#46

Post by ensignavenger »

daveguy wrote: post #40
ensignavenger wrote: post #35

Earlier quoted context omitted.

It would be really nice if the IRS would ALLOW you to walk in and ask a question!

That is very expensive. Offices all around the country with personnel. We are going to have to fund them instead of gripe about them to get that to happen.

Yeah, that is why I doubt it will happen. Maybe a website where you can sumbit an issue and have it resolved in a reasonable number of days would be fine.

Re: Show HN: Open source framework OpenAI uses for Advanced Voice

#47

Post by ensignavenger »

andrew_eu wrote: post #41
ensignavenger wrote: post #35

Earlier quoted context omitted.

It would be really nice if the IRS would ALLOW you to walk in and ask a question!

Years ago my tax return was flagged as a possible fraud case -- I believe a direct consequence of a big data breach. I had to go into my "local" IRS office and present my passport to prove indeed it was me. Decidedly not nice. True to form, with an appointment I waited 3 hours at the office and watched the guard staff turn away countless people. Finally saw a person, gave then my passport, and finished in a minute.

I am going through that right now. IRS owes me 3 years of refunds, but I can't even get an appointment to see them. They hang up one when I call (after hours on hold), and won't let me just visit the local office. My current attempt is to work with my US Senators office.

Re: Show HN: Open source framework OpenAI uses for Advanced Voice

#48

Post by fidotron »

russ wrote: post #45
fidotron wrote: post #42

Earlier quoted context omitted.

> LiveKit is too expensive Most of it is open source, especially the clients, although they do feel quite ad hoc hacked together (a possible side effect of WebRTC evolution). Would totally agree on OpenAI moving away. The description of the agent here sounds like a big hack just to get around the fact temporarily the model server expects audio over sockets instead.

Which components feel ad hoc? In most real applications, the agent has additional logic (function calling, RAG, etc) than simply relaying a stream to the model server. In those cases, you want it to be a separate service/component that can be independently scaled.

Essentially I think the Livekit value is a SFU that works, with signalling, and the SDKs exist. My experience is people radically overstate how hard signalling is, and underestimate SFU complexity, especially with fast failover.

In terms of being a higher level API arguably it is doomed to failure, thanks to the madness of the domain. (The part that sticks in my mind is audio device switching on Android.) WebRTC products seem to always end up with the consumer needing to know way more of the internals than is healthy. As such I think once you are sufficiently good at using LiveKit you are less likely to pick it for your next product because you will be able to roll your own far more easily. That is unless the value you were getting from it actually was the SFU infrastructure and not the SDKs.

The OpenAI case is so point-to-point that doing WebRTC for that is, honestly, really not hard at all.

Re: Show HN: Open source framework OpenAI uses for Advanced Voice

#49

Post by russ »

throw14082020 wrote: post #25

This is really helpful, thanks! OpenAI hired the ex fractional CTO of LiveKit, who created Pion, a popular WebRTC library/tool. I'd expect OpenAI to migrate off of LiveKit within 6 months. LiveKit is too expensive. Also, WebRTC is hard, and OpenAI now being a less open company will want to keep improvements to itself. Not affiliated with any competitors, but I did work at a PaaS company similar to LiveKit but used We…

Field CTO — hi @Sean-Der :wave:

Fractional CTO sounds like a disaster lol

Re: Show HN: Open source framework OpenAI uses for Advanced Voice

#50

Post by davidz »

rukuu001 wrote: post #20
davidz wrote: post #7

Earlier quoted context omitted.

Currently it does: all audio is sent to the model. However, we are working on turn detection within the framework, so you won't have to send silence to the model when the user isn't talking. It's a fairly straight forward path to cutting down the cost by ~50%.

Working on this for an internal tool - detecting no speech has been a PITA so far. Interested to see how you go with this.

currently we are using silero VAD to detect speech: https://github.com/livekit/agents/blob/main/livekit-plugins/...

it works well for voice activity; though it doesn't always detect end-of-turn correctly (humans often pause mid-sentence to think). we are working on improving this behavior.

Post reply on HN