51–60 of 71 posts
Is there a way to convert a Cython module(s) to C++, or at least a .o file? They are so dang close.
I like cython, but I think it pigeon-holes developers: i've seen hardware modules written in Cython, when they could easily switch to C++ and provide a library that could be used as an FFI in any language, but instead locked themselves (and users) into the narrow world of Python. Is there a way to convert a Cython module(s) to C++, or at least a .o file? They are so dang close.
While it is nice that this option is available, it would be much better if Python itself would embrance the necessary runtime capabilties to not have to rely on it.
To write fast Cython, you basically need to write in C and control everything including Python API calls. No runtime will help with this.
Earlier quoted context omitted.
Common Lisp and simlar languages are a prof of what is possible.
Writing a much faster language runtime for a language that looks quite a bit like Python is easy. The hard problem is writing a faster language runtime that is 100% compatible with all current python programs (and their extensions) out in the world.
Yes, we just went through one "tough but important" transition (Python 2->3) and it sucks to have to do this so often, but it's the price we pay from making bad bets (in this case, unnecessarily exposing the entire CPython interpreter as the C-extension API surface on the assumption that it will never be necessary to make CPython faster because people who need speed will just write the slow parts in C).
Earlier quoted context omitted.
To write fast Cython, you basically need to write in C and control everything including Python API calls. No runtime will help with this.
Common Lisp and simlar languages are a prof of what is possible.
The only drawback is that a Cython module still loads the CPython interpreter, so I personally prefer writing performance critical code in Rust instead. Writing in Julia has the same drawbacks of not being embeddable that writing in Cython does.
Julia has multiple dispatch and may seem more appealing but at scale it is a very slow language to develop in. And for scripts it takes FOREVER (try loading Plots, CSV, DataFrames, Makie etc every time you restart. It’s genuinely insane that that’s the norm.)
If the whole Python ecosystem was in Cython (i.e. numpy, scipy, etc) I’d never use another backend language again.
Earlier quoted context omitted.
But this is not necessary. The python developers just need to specify this fast subset of the language, and let people use to create libraries. Over time, we would have a growing set of libraries written in the fast subset.
People have. See PyPy and Pythran for two examples currently under active development. Instagram has such a project as well that they recently released. I know there have been others. None of them seem to catch on. It seems that most people don't actually want a faster subset of python. They want either all of python or none of python (by switching to another language all together)
I would recommend considering using NanoBind, the follow up of PyBind11 by the same author (Wensel Jakob), and move as much performance critical code to C or C++. https://github.com/wjakob/nanobind If you really care about performance called from Python, consider something like NVIDIA Warp (Preview). Warp jits and runs your code on CUDA or CPU. Although Warp targets physics simulation, geometry processing, and proced…
Why would you recommend that? It's all way more effort than just writing Cython, especially in a Jupyter Notebook. And Cython code can be just as fast as C/C++ code unless you're doing something really fancy. It's a bunch of work for no benefit.
>Warp jits and runs your code on CUDA or CPU
If someone's writing Cython it's probably because they found something that couldn't be done efficiently in Numpy because it was sequential, not easily vectorisable. Such code is going to get zero benefit from Cuda or running on the GPU.
In general, all your jitted code is not going to be as fast as code compiled with an ahead-of-time compiler like the C compiler that Cython uses. Moreover if you use a JIT then it makes your code a pain in the ass to embed in a C/C++ application, unlike Cython code.