Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jarvis: A Voice Virtual Assistant in Python (OpenAI, ElevenLabs, Deepgram)

github.com

Post reply on HN
Print view View original

51–60 of 60 posts

Re: Jarvis: A Voice Virtual Assistant in Python (OpenAI, ElevenLabs, Deepgram)

#51

Post by zaptrem »

Spiwux wrote: post #9

I wonder if we're at a point where you could build a voice assistant like that, except almost-realtime and streamed end to end: User speaks and speech to text starts streaming text while the user is still speaking. That text stream is piped into a LLM, which also streams its output text. That output text is streamed to text-to-speech, which also generates audio in a streaming manner.

Available as a phone line API (https://www.vocode.dev) and OS project (https://github.com/vocodedev/vocode-python)

Re: Jarvis: A Voice Virtual Assistant in Python (OpenAI, ElevenLabs, Deepgram)

#52

Post by visarga »

alchemist1e9 wrote: post #12
Alyx1337 wrote: post #2

Here is a video demo of the project: https://youtu.be/aIg4-eL9ATc?si=66ynl4Mlci9v76rU

Nice work! Very impressed. Do you happen to know anything about any open source voice identification software? I’ve noticed with ChatGPT voice and any other voice driven assistant that a massive problem is the background voices and noise. One solution could be advanced pre-processing to ID your voice only. Another idea I’ve had is using something professional with PTT: https://sheepdogmics.com/products/quick-disconne…

Google Gemini was trained on audio and can generate audio directly. Whatever you build now will be replaced by a much better version soon.

Re: Jarvis: A Voice Virtual Assistant in Python (OpenAI, ElevenLabs, Deepgram)

#53

Post by java_beyb »

Alyx1337 wrote: post #14
vessenes wrote: post #4

I'm curious - what made you choose deepgram over just running whisper? I don't have any experience with deepgram but whisper has worked so well in my own tests that I didn't even ever consider there might be API speech recognition-only companies.

Deepgram advertised itself as being the fastest, and I wanted to focus on limiting response delay so I chose it. I hope I did not get misled.

well, deepgram might be the fastest among cloud-dependent APIs, like Speechmatics and Assembly AI mentioned above. -but- it cannot be faster than local or smaller models as you mentioned.

Among local solutions, Whisper SDK doesn't support streaming, I haven't seen any good workarounds or successfully implemented it. VOSK, DeepSpeech, Kaldi, et al were good once upon a time... Picovoice seems to be doing well.

I was planning to work on this: https://picovoice.ai/blog/chatgpt-ai-virtual-assistant-in-py... using Eleven Labs and Cheetah. Hope I can crave some time

Re: Jarvis: A Voice Virtual Assistant in Python (OpenAI, ElevenLabs, Deepgram)

#54

Post by Alyx1337 »

alchemist1e9 wrote: post #42
Alyx1337 wrote: post #17

Earlier quoted context omitted.

Thanks! I don't know a lot about this but someone shared this local voice assistant in the comments: https://github.com/KoljaB/LocalAIVoiceChat Could be a good lead

Yeah github.com/KoljaB is quite a collection of stuff! I agree. It all seems your vision of JARVIS, which I share completely but haven't accomplished what you have, again excellent work and thank you for sharing, is very attainable. Probably combining your work along with KoljaB is very promising.

Thank you very much!

Re: Jarvis: A Voice Virtual Assistant in Python (OpenAI, ElevenLabs, Deepgram)

#56

Post by 3abiton »

modeless wrote: post #46
Spiwux wrote: post #9

I wonder if we're at a point where you could build a voice assistant like that, except almost-realtime and streamed end to end: User speaks and speech to text starts streaming text while the user is still speaking. That text stream is piped into a LLM, which also streams its output text. That output text is streamed to text-to-speech, which also generates audio in a streaming manner.

I implemented this! All local models. And I packaged it up so people can install it with one click: https://apps.microsoft.com/detail/9NC624PBFGB7 The speech recognition part needs work for sure, but when it works you can see the potential. It's very different from the way it feels to talk to Siri or even ChatGPT's voice mode. It won't be long before we are having real conversations with our computers.

But how realtime is it?

Re: Jarvis: A Voice Virtual Assistant in Python (OpenAI, ElevenLabs, Deepgram)

#57

Post by modeless »

3abiton wrote: post #56
modeless wrote: post #46

Earlier quoted context omitted.

I implemented this! All local models. And I packaged it up so people can install it with one click: https://apps.microsoft.com/detail/9NC624PBFGB7 The speech recognition part needs work for sure, but when it works you can see the potential. It's very different from the way it feels to talk to Siri or even ChatGPT's voice mode. It won't be long before we are having real conversations with our computers.

But how realtime is it?

The end-to-end response latency is around 1 second typically. It listens continuously, there are no buttons to press, and you can interrupt it while it's talking.

Re: Jarvis: A Voice Virtual Assistant in Python (OpenAI, ElevenLabs, Deepgram)

#58

Post by modeless »

bjelkeman-again wrote: post #50
modeless wrote: post #46

Earlier quoted context omitted.

I implemented this! All local models. And I packaged it up so people can install it with one click: https://apps.microsoft.com/detail/9NC624PBFGB7 The speech recognition part needs work for sure, but when it works you can see the potential. It's very different from the way it feels to talk to Siri or even ChatGPT's voice mode. It won't be long before we are having real conversations with our computers.

Could you record a demo of this?

I really should! I'm not the type to publish videos of myself usually, but it really does need a video demo.

Re: Jarvis: A Voice Virtual Assistant in Python (OpenAI, ElevenLabs, Deepgram)

#59

Post by jkachmar »

java_beyb wrote: post #53
Alyx1337 wrote: post #14

Earlier quoted context omitted.

Deepgram advertised itself as being the fastest, and I wanted to focus on limiting response delay so I chose it. I hope I did not get misled.

well, deepgram might be the fastest among cloud-dependent APIs, like Speechmatics and Assembly AI mentioned above. -but- it cannot be faster than local or smaller models as you mentioned. Among local solutions, Whisper SDK doesn't support streaming, I haven't seen any good workarounds or successfully implemented it. VOSK, DeepSpeech, Kaldi, et al were good once upon a time... Picovoice seems to be doing well. I was p…

unless i'm misunderstanding `whisper.cpp` seems to support streaming & the repository includes a native example[0] and a WASM example[1] with a demo site[2].

[0]: https://github.com/ggerganov/whisper.cpp/tree/master/example...

[1]: https://github.com/ggerganov/whisper.cpp/blob/master/example...

[2]: https://whisper.ggerganov.com/stream/

Re: Jarvis: A Voice Virtual Assistant in Python (OpenAI, ElevenLabs, Deepgram)

#60

Post by java_beyb »

jkachmar wrote: post #59
java_beyb wrote: post #53

Earlier quoted context omitted.

well, deepgram might be the fastest among cloud-dependent APIs, like Speechmatics and Assembly AI mentioned above. -but- it cannot be faster than local or smaller models as you mentioned. Among local solutions, Whisper SDK doesn't support streaming, I haven't seen any good workarounds or successfully implemented it. VOSK, DeepSpeech, Kaldi, et al were good once upon a time... Picovoice seems to be doing well. I was p…

unless i'm misunderstanding `whisper.cpp` seems to support streaming & the repository includes a native example[0] and a WASM example[1] with a demo site[2]. [0]: https://github.com/ggerganov/whisper.cpp/tree/master/example... [1]: https://github.com/ggerganov/whisper.cpp/blob/master/example... [2]: https://whisper.ggerganov.com/stream/

have you tried it? i mean for fun, it wouldn't hurt for sure and ggerganov is doing amazing stuff. kudos to him.

but whisper is designed to process audio files in 30-second batches if I'm not mistaken. it's been a while since whisper released, lol. These workarounds make the window smaller but it doesn't change the fact that they're workarounds. you can adjust, modify, or manipulate the model. You can't write or train it from scratch. check out the issues referring to the real-time transcription in the repo.

can you use it? yes would it perform better than Deepgram? -although it's an API and probably not the best API- I am not sure. would i use it in my money-generating application? absolutely not.

Post reply on HN