Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Port of OpenAI's Whisper model in C/C++

github.com

Post reply on HN
Print view View original

81–90 of 93 posts

Re: Show HN: Port of OpenAI's Whisper model in C/C++

#81

Post by ProtoAES256 »

GistNoesis wrote: post #24

Great work. It's a real bowl of fresh air coming from huge framework that use cuda. So many different cuda version, with each framework using its own, that all rely on a different driver, and everything needs a new version every 3 months and takes ~10G, (and don't even talk about cudnn needing some manual logged-in install). Here everything is just two files. For embedded system that don't have a GPU it's perfect. He…

Ditto about the CUDA and cuDNN part. My project that was running fine for the past 4 years just "died" after a colleague's oversight on upgrading the GPU(1080Ti -> 3090) which isn't compatible with the new cuDNN. It is just too much of a hassle maintaining that *expletive* jargon so I did the wise decision to kill it.

Re: Show HN: Port of OpenAI's Whisper model in C/C++

#83

Post by ggerganov »

wfcollins wrote: post #36

Hi Georgi, I am experimenting with your code now. Is there a way to force Whisper to only consider a limited vocabulary and then respond with confidence levels? I am working on an app where it is important to restrict answers and I would like to know how confident that a response is one of a set of words. If the answer could be word A with a confidence level of 95% and word B with a level of 50%, I would want to know…

Hi, it's not obvious how to achieve this, but it feels it could be done. I think all the "tools" are available in the existing interface in `whisper.h` - for example, `whisper_get_probs()` gives you the probability for each token.

Re: Show HN: Port of OpenAI's Whisper model in C/C++

#87

Post by Const-me »

Feel free to merge my fork, about 20% faster on my computer (Ryzen 7 5700G CPU, medium.en model): https://github.com/Const-me/whisper.cpp It also contains VS2022 projects to build on Windows, your cmake project results in disabled AVX which is critical for performance.

Also, I didn’t really understand your multithreading code in ggml_graph_compute function, but that custom thread pool implementation IMO looks suspicious. Just too many atomics. Might be possible to improve a lot with a better multithreading strategy.

Re: Show HN: Port of OpenAI's Whisper model in C/C++

#88

Post by moloch-hai »

kgwgk wrote: post #77
moloch-hai wrote: post #63

There is no language called "C/C++". If, to use it, you need C++, it's a C++ system, period. If parts are in C, that is an implementation detail, and is furthermore pointless; modernizing those to C++ would make them more robust and probably faster. Using C anywhere just makes it as bug-prone in those parts as C code everywhere is.

> There is no language called "C/C++". He clearly means that he used two languages: C and C++. "I decided to reimplement the inference of the model from scratch using C/C++. To achieve this I implemented a minimalistic tensor library in C and ported the high-level architecture of the model in C++."

It is obvious what he did just from reading what he wrote, after. It is not clear that he meant anything at all by the expression. You cannot use his thing in any but a C++ program, so mentioning C at all only adds confusion.

Re: Show HN: Port of OpenAI's Whisper model in C/C++

#89

Post by kgwgk »

moloch-hai wrote: post #88
kgwgk wrote: post #77

Earlier quoted context omitted.

> There is no language called "C/C++". He clearly means that he used two languages: C and C++. "I decided to reimplement the inference of the model from scratch using C/C++. To achieve this I implemented a minimalistic tensor library in C and ported the high-level architecture of the model in C++."

It is obvious what he did just from reading what he wrote, after. It is not clear that he meant anything at all by the expression. You cannot use his thing in any but a C++ program, so mentioning C at all only adds confusion.

It is clear what he meant by the distiction:

  implemented a minimalistic tensor library in C

  ported the high-level architecture of the model in C++
The C part - which accounts for most of the lines of code - may be of interest to someone who wants to use it as the foundation for implementing a different model - not necessarily in C++

Re: Show HN: Port of OpenAI's Whisper model in C/C++

#90

Post by NWoodsman »

stevenhuang wrote: post #46
AMICABoard wrote: post #32

I vouch for this. Pretty solid and keeps improving. The OP is in the class of Magic Wizards of programming like Fabrice Bellard! There are frequent updates and performance improvements. There is also a small community of active users around this. All most all feedbacks get implemented and the OP is very responsive. The OP made it possible to do state of the art voice recognition without the PyTorch baggage and in C/C…

Yup this is so magical. I've always felt there was something off about requiring setting up what is essentially a pytorch/ml dev environment everytime end users "just" want to run inference. A single binary that does this all w/o the python stack is just incredible! edit: Got it going in 1 min! I grabbed the prebuilt artifacts (windows) - https://github.com/ggerganov/whisper.cpp/actions/runs/363552... Then downloaded…

Can you expand on your steps a bit more? I've never used Github Actions which seems like step 1. Not sure how to get an installer.
Post reply on HN