Show HN: Chat with Zephyr 7B over voice message using whisper.cpp and llama.cpp

#1

Post by tynmann » Mon, Oct 16, 2023, 1:48 AM UTC

Just my small weekend project, experimenting with LLMs. I made a Discord bot that uses the Zephyr 7B model to reply, and whisper.cpp to transcribe voice messages into text. So you can send it voice messages and it replies.

Show HN: Chat with Zephyr 7B over voice message using whisper.cpp and llama.cpp
bastianai.com

Re: Show HN: Chat with Zephyr 7B over voice message using whisper.cpp and llama.cpp

#2

Post by wanderingmind » Mon, Oct 16, 2023, 3:47 AM UTC

This looks good, thanks for sharing. Can you share a little more about where the compute is performed, the transcription and llm? Its a local machine or a machine in cloud? If its local, i assume it cant be used by other people in server? If its on a network/cloud machine, how is the communication handled between discord and the machine? Also what modifications need to be done to do longer streams of audio data?

Re: Show HN: Chat with Zephyr 7B over voice message using whisper.cpp and llama.cpp

#3

Post by tynmann » Mon, Oct 16, 2023, 8:09 AM UTC

wanderingmind wrote: post #2

This looks good, thanks for sharing. Can you share a little more about where the compute is performed, the transcription and llm? Its a local machine or a machine in cloud? If its local, i assume it cant be used by other people in server? If its on a network/cloud machine, how is the communication handled between discord and the machine? Also what modifications need to be done to do longer streams of audio data?

I'm running everything on my own PC. With a 3060 12 GB at under $300, it's very cheap to build a system capable of running inference with smaller models.

But even bigger models (13B or 30B) can be handled with a dual GPU setup (24 GB total memory at The Discord bot works as a client (outgoing connection to the Discord public server), so it can run from behind NAT.

Re: Show HN: Chat with Zephyr 7B over voice message using whisper.cpp and llama.cpp

#4

Post by tynmann » Mon, Oct 16, 2023, 8:12 AM UTC

tynmann wrote: post #3

wanderingmind wrote: post #2

This looks good, thanks for sharing. Can you share a little more about where the compute is performed, the transcription and llm? Its a local machine or a machine in cloud? If its local, i assume it cant be used by other people in server? If its on a network/cloud machine, how is the communication handled between discord and the machine? Also what modifications need to be done to do longer streams of audio data?

I'm running everything on my own PC. With a 3060 12 GB at under $300, it's very cheap to build a system capable of running inference with smaller models. But even bigger models (13B or 30B) can be handled with a dual GPU setup (24 GB total memory at The Discord bot works as a client (outgoing connection to the Discord public server), so it can run from behind NAT.

However if one wants to run something like this in production, it would make more sense to use a server with at least an A10, which has better performance per power consumption.