Why not zig.
Show HN: I built a tensor library from scratch in C++/CUDA
21–30 of 30 posts
Re: Show HN: I built a tensor library from scratch in C++/CUDA
#22Earlier quoted context omitted.
Yes, when I designed the API I wanted to keep a clear distinction between Python and C. At some point I had two APIs: 1 in Python and the other in high-level C++ and they both shared the same low-level C API. I find this design quite clean and easy to work with if multiple languages are involved. When I'll get to perf I plan to experiment a bit with nanobind ( https://github.com/wjakob/nanobind ) and see if there's a…
The call overhead of using ctypes vs nanobind/pybind is enormous https://news.ycombinator.com/item?id=31378277 Even if the number reported there is off, it's not far off because ctypes just calls out to libffi which is known to be the slowest way to do ffi.
Thanks for pointing this out! I'll definitely have to investigate other approaches. nanobind looks interesting but I don't need to expose complex C++ objects, I just need the 'fastest' way of calling into a C API. I guess the goto for this is CFFI?
Re: Show HN: I built a tensor library from scratch in C++/CUDA
#23It's very C-like, heavy use of macros, prefixes instead of namespaces, raw pointers for arrays etc. Technically you're compiling C++, but... not really. No negative or positive comment on its usability though, I'm not an ML/Neural Network simulation person.
Yes! This was actually one of my initial goals! I actually like to work in a C-style-C++ let's say where I turn off C++ features I don't need and just use the one I actually need like templates, objects ecc...
I find this style to be easy to reason about when it comes to performance.
Re: Show HN: I built a tensor library from scratch in C++/CUDA
#24Earlier quoted context omitted.
Thanks! To be honest, it started purely as a learning project. I was really inspired when llama.cpp first came out and tried to build something similar in pure C++ ( https://github.com/nirw4nna/YAMI ), mostly for fun and to practice low-level coding. The idea for DSC came when I realized how hard it was to port new models to that C++ engine, especially since I don't have a deep ML background. I wanted something that…
If someone wanted to learn the same thing, what material would you suggest is a good place to start?
You just need a foundation of C/C++. If you already have that then just start programming, it's way better than reading books/guides/blogs (at least until you're stuck!). Also, you can read the source code of other similar projects on GitHub and get ideas from them, this is what I did at the beginning.
Re: Show HN: I built a tensor library from scratch in C++/CUDA
#25Why not zig.
Because I happen to know C++ and I just wanted to build something rather than learn a new language. Zig looks very interesting though, there are already other projects in this space that use it with great success (see: https://github.com/zml/zml).
Re: Show HN: I built a tensor library from scratch in C++/CUDA
#26It's very C-like, heavy use of macros, prefixes instead of namespaces, raw pointers for arrays etc. Technically you're compiling C++, but... not really. No negative or positive comment on its usability though, I'm not an ML/Neural Network simulation person.
Yes! This was actually one of my initial goals! I actually like to work in a C-style-C++ let's say where I turn off C++ features I don't need and just use the one I actually need like templates, objects ecc... I find this style to be easy to reason about when it comes to performance.
The proper way to reason about performance is to use a profiler, not second guessing what C like code generates.
Re: Show HN: I built a tensor library from scratch in C++/CUDA
#27Earlier quoted context omitted.
The call overhead of using ctypes vs nanobind/pybind is enormous https://news.ycombinator.com/item?id=31378277 Even if the number reported there is off, it's not far off because ctypes just calls out to libffi which is known to be the slowest way to do ffi.
Thanks for pointing this out! I'll definitely have to investigate other approaches. nanobind looks interesting but I don't need to expose complex C++ objects, I just need the 'fastest' way of calling into a C API. I guess the goto for this is CFFI?
It's the same thing - both nanobind and cffi compile the binding. The fact that nanobind let's you expose cpp doesn't prevent you from only exposing c. And IMHO nanobind is better because you don't need to learn another language to use it (ie you don't need to learn cffi's DSL).
Re: Show HN: I built a tensor library from scratch in C++/CUDA
#28It's very C-like, heavy use of macros, prefixes instead of namespaces, raw pointers for arrays etc. Technically you're compiling C++, but... not really. No negative or positive comment on its usability though, I'm not an ML/Neural Network simulation person.
I've found adherence to C++ conventions in low-level software to be a rather contentious issue, mostly recently when working in an ML compiler group. One set abhorred the use of macros, the other any kind of polymorphism or modern C++ feature. Coming from a background of working with OS kernels and systems software, I don't mind the kind of explicit "C++ lite" style used by the OP. Left to my own devices, I usually w…
If you think that, I encourage you to check out this presentation:
https://www.youtube.com/watch?v=zBkNBP00wJE
About writing a Commodore C64 game in modern(ish) C++
maybe it will sway you a bit :-)
Re: Show HN: I built a tensor library from scratch in C++/CUDA
#29Cool stuff! Is the goal of this project personal learning, inference performance, or something else? Would be nice to see how inference speed stacks up against say llama.cpp
Both uses cublas under the hood. So I think it is similar for prefilling (of course, this framework is too early and don't have FP16 / BF16 support for GEMM it seems). Hand-roll gemv is faster for token generation hence llama.cpp is better.
Unrelated: my man, I loved your C vision library back in the day.
Re: Show HN: I built a tensor library from scratch in C++/CUDA
#30Earlier quoted context omitted.
I've found adherence to C++ conventions in low-level software to be a rather contentious issue, mostly recently when working in an ML compiler group. One set abhorred the use of macros, the other any kind of polymorphism or modern C++ feature. Coming from a background of working with OS kernels and systems software, I don't mind the kind of explicit "C++ lite" style used by the OP. Left to my own devices, I usually w…
If you think that, I encourage you to check out this presentation: https://www.youtube.com/watch?v=zBkNBP00wJE About writing a Commodore C64 game in modern(ish) C++ maybe it will sway you a bit :-)
That is impressive. It certainly shows what is possible _if_ you are familiar enough with the intricacies of modern C++. I'm not sure how I feel about a workflow where one needs to continually address "overhead" introduced by the language environment.