Awesome work.
Honest question, do you think with this marked improvement it could be worth making a wrapped library for this c/c++ version in python - for example like numpy?
Show HN: Port of OpenAI's Whisper model in C/C++
31–40 of 93 posts
Re: Show HN: Port of OpenAI's Whisper model in C/C++
#32I vouch for this. Pretty solid and keeps improving. The OP is in the class of Magic Wizards of programming like Fabrice Bellard!
There are frequent updates and performance improvements. There is also a small community of active users around this.
All most all feedbacks get implemented and the OP is very responsive.
The OP made it possible to do state of the art voice recognition without the PyTorch baggage and in C/C++, pretty incredible! Its one of those rare high value projects.
Very grateful for this project and respect to the OP!
Some day if a ChatGPT open version becomes available, this could mean voice assistants that speak sense and understand the human - as long you have a beefy machine.
The current efficiency is pretty surprising, even on a low spec device it performs faster than real time.
I don't know what to say. But I'm blown away.
I expect to see more magic from the OP in future.
He has even a project for a cool sound modem that works over ultrasonic! Not new stuff, but the implementation is the most robust I have seen.
I recommend hackers here to check out his other project too and maybe contribute with testing and patches and stuff!
Re: Show HN: Port of OpenAI's Whisper model in C/C++
#33This is really cool, and I have been meaning to get my hands dirty with Whisper I looks like you’ve definitely maximized the parallelization on the CPU/AMX here, but have you tried getting it to run on the GPU or the Neural Engine? I love the portability, but I feel like you would get a massive parallelization boost while dramatically cutting energy consumption.
I did some experiments with adding Metal Performance Shaders support, but the performance that I achieved was only marginally better compared to the one I get when using just the Accelerate framework (there is an unmerged PR with the tests). Honestly, I am bit confused with all the different types of processing units available on Apple Silicon. If I understand correctly, we have: CPU, GPU, AMX coprocessor and Neural…
OpenCL would be appreciated much... Opens the door to use this on many more low powered devices but, it could be very difficult as you have already mentioned.
Re: Show HN: Port of OpenAI's Whisper model in C/C++
#34Live demo: https://whisper.ggerganov.com/
Holy crap, I just loaded a random video file into the tiny model and it did it perfectly and quickly too. This is amazing.
Re: Show HN: Port of OpenAI's Whisper model in C/C++
#35Great work. It's a real bowl of fresh air coming from huge framework that use cuda. So many different cuda version, with each framework using its own, that all rely on a different driver, and everything needs a new version every 3 months and takes ~10G, (and don't even talk about cudnn needing some manual logged-in install). Here everything is just two files. For embedded system that don't have a GPU it's perfect. He…
100%.
So much more practical to hack around and or build small apps.
Re: Show HN: Port of OpenAI's Whisper model in C/C++
#36Hi Georgi,
I am experimenting with your code now. Is there a way to force Whisper to only consider a limited vocabulary and then respond with confidence levels? I am working on an app where it is important to restrict answers and I would like to know how confident that a response is one of a set of words. If the answer could be word A with a confidence level of 95% and word B with a level of 50%, I would want to know that so that I could perform context verification.
Thanks!
Bill
Re: Show HN: Port of OpenAI's Whisper model in C/C++
#37Amazing, even more so that you rolled your own inference and didn’t use ONNX runtime. Thanks so much for sharing.
Re: Show HN: Port of OpenAI's Whisper model in C/C++
#38There are tons of open source STT models, what makes whisper so valuable? I especially don't get it on mobile, where the native STT built into the OS is now real-time and includes punctuation (at least for iOS). I love the open-source approach to the model, but it didn't strike me as particularly better than other open-source or built-in models.
Re: Show HN: Port of OpenAI's Whisper model in C/C++
#39There are tons of open source STT models, what makes whisper so valuable? I especially don't get it on mobile, where the native STT built into the OS is now real-time and includes punctuation (at least for iOS). I love the open-source approach to the model, but it didn't strike me as particularly better than other open-source or built-in models.
In my limited testing, I've found Whisper to be much better (accuracy-wise) than other STT models.
Re: Show HN: Port of OpenAI's Whisper model in C/C++
#40Love it, great job!