Very cool, thank you.
Show HN: Port of OpenAI's Whisper model in C/C++
11–20 of 93 posts
Re: Show HN: Port of OpenAI's Whisper model in C/C++
#12This is awesome. I'm sure the fact that most ML models require an insane mess of Python packages is holding applications back.
Hook this up to ChatGPT and you've got something better than Google Assistant with almost no work.
(You can tell ChatGPT an API, and ask it to generate a script in response to a voice assistant query.)
Re: Show HN: Port of OpenAI's Whisper model in C/C++
#13> about 2-3 times faster compared to the current PyTorch implementation This is surprising to me. Is this about CPU only? Then it would make sense. Also, is there a particular reason why the whole code is basically 2 massive files (3k and 8k lines respectively)?
I am not very good with Python, so there is some chance I am doing something wrong. But my explanation of the results that I get is that PyTorch currently does not fully utilize FP16 + Metal or AMX when running on Apple Silicon. In contrast, my implementation stores the weights in 16-bit floating point precision (FP16) and also utilizes the AMX coprocessor through the Accelerate framework. As I mentioned in OP, the latter is very efficient for doing the matrix multiplications. According to my experiments, it is comparable in performance to running them on the Apple GPU via Metal.
Re: Show HN: Port of OpenAI's Whisper model in C/C++
#14Thank you for sharing your project.
Re: Show HN: Port of OpenAI's Whisper model in C/C++
#15I've been watching this repo pretty much since the beginning, and the amount of work you've achieved is incredible.
I've started tinkering with the code about last week and despite knowing nothing about C/C++, I was able to make some edits to fit my use case, and connect it to a custom Python front end (I initially tried to use Qt in C++ but struggled so much to get to it to compile that I've switched to Python instead). This probably means your code is very clean and well documented.
It's a game changer in terms of accessibility: it can caption almost anything in live!
I'm very grateful for the effort that you've lead. Thank you ggerganov, and thanks to everyone who contributed.
Re: Show HN: Port of OpenAI's Whisper model in C/C++
#16This is very cool. Excellent work!
Perhaps in combination with https://www.npmjs.com/package/peertube-plugin-transcription your port of Whisper could be used for generating subtitles for videos in PeerTube?
I just recently set up a PeerTube instance of my own and uploaded my first video on it ("No Brain Required - ChatGPT solves Advent of Code in Rust, episode 1", https://video.nstr.no/w/6z7PxB4J92H3NHhgMmfYVw)
I want to try and make use of your port of Whisper on my PeerTube instance, so that I can have subtitles generated for my videos on it :D
Re: Show HN: Port of OpenAI's Whisper model in C/C++
#17I love this! That talk.wasm package has a lot of potential…
Re: Show HN: Port of OpenAI's Whisper model in C/C++
#18This is really outstanding work, thank you so much for doing this and open sourcing it! I'm sure this will enable many applications in the future!
Re: Show HN: Port of OpenAI's Whisper model in C/C++
#19Why not Rust? C++ is obsolete
I wonder how ChatGPT3 would respond to this?
If they crawled /r/programming and hackernews, it should be annoyed and come up with something along the lines of "fork it and write in in Rust yourself", "C++ has lots of inertia", "his main stack is C/C++", etc
Re: Show HN: Port of OpenAI's Whisper model in C/C++
#2010/10 you're doing god's work my friend, can't wait to spend some time this weekend to try and understand what's going on here. I can't overstate how much I value small libraries. I can't think of a faster way to learn about a concept than to step through someone else's barebones implementation.
Thanks! Indeed, I agree that the project has an educational aspect and value. For me, it helped me get a better understanding of the neural network layers involved in the transformer model. Also, it was a good playground to practice my low-level optimization techniques. I guess another cool thing was that with the help of the community, we came up with a faster way to evaluate the Encoder (at the cost of some accurac…
I liked reading the different implementations of the low-level tensor ops (simple C/AVX/AVX2/WASM128bit/ARM-NEON) -- it will help me learn about how to use x86 ASM. Thank you for writing this! Do you have any other recommendations/examples on how numerical code can be optimized via SIMD routines?