21–30 of 147 posts
I feel like the #1 downside of Python for the last few years is that you cannot take advantage of multiple cores of a CPU easily. Especially when you think it is heavily used in data analysis. We use Python for data analysis as well, and for 95% of operations we are doing, numpy is fast enough that we don't have any complaints. But sometimes, we do wish to be able to take advantage of all the cores in our CPUs, espec…
From the benchmarks: > Rust is not that fast because it needs to copy data; I'm surprised. Don't know much about Rust, but isn't it hailed as being competitive with C/C++ ?
The culprit here is `a_list: Vec >`, which means the list of lists is copied as a whole, hence why the rust version is slower. In general, rust is at least as fast as C or C++, but this seems to be a special case.
Rust doesn’t need to copy the data. It’s trivial to pass e.g. Numpy arrays to Rust as slices via Cython (let alone originating in Cython!), modify them, and return them, or use them as input for a new returned struct. https://github.com/urschrei/simplification https://github.com/urschrei/lonlat_bng https://github.com/urschrei/pypolyline Each of those repos has links to the corresponding Rust “shim” libraries that pro…
To add, it is trivial to guarantee that Rust code is "zero-copy", so if this is something you care about, Rust allows your program to fail to compile if it tries to make a copy.
Rust doesn’t need to copy the data. It’s trivial to pass e.g. Numpy arrays to Rust as slices via Cython (let alone originating in Cython!), modify them, and return them, or use them as input for a new returned struct. https://github.com/urschrei/simplification https://github.com/urschrei/lonlat_bng https://github.com/urschrei/pypolyline Each of those repos has links to the corresponding Rust “shim” libraries that pro…
When I was interfacing D code with a part of Unreal Engine that I think is garbage collected, I actually just took the L and copied everything into buffers on their malloc when handing stuff off to the engine. It wasn't particularly hot code so the memcpys were worth the peace of mind I found, ugly as it was.
Earlier quoted context omitted.
You left out the explanation from your quote. The full is: > Rust is not that fast beacuse it needs to copy data; using Pyo3 objects would probably lead to similar results as cython, but with an added library. "It needs to copy data", because it's converting Python objects into Rust objects and back again. As the full quote states, it could be written in a different way, although that would make the code look really…
> although that would make the code look really weird. Not it wouldn't? Using Pyo3 is arguably the most idiomatic way of interfacing Rust with python. The "added library" cost mentioned as an excuse not to use it is literally the 1 second it takes me to write `cargo add pyo3` . They didn't do this because it would make their claim that "Julia is the better language for extending Python" kind of moot. They started fro…
I'll defer to your knowledge on that. I generally don't work with Python. I've only looked into interfacing C# with Rust and it looked crazy, so that's where I'm coming from.
But yes, it's pretty clear that the results are more a consequence of what's in the source files than the languages used (if we can even say that e.g. C++ was used at all).
I feel like the #1 downside of Python for the last few years is that you cannot take advantage of multiple cores of a CPU easily. Especially when you think it is heavily used in data analysis. We use Python for data analysis as well, and for 95% of operations we are doing, numpy is fast enough that we don't have any complaints. But sometimes, we do wish to be able to take advantage of all the cores in our CPUs, espec…
> Python for the last few years is that you cannot take advantage of multiple cores of a CPU easily. Same thing in the R ecosystem, it's possible to use multiple cores with the parallel package but there are caveats too.
I feel like the #1 downside of Python for the last few years is that you cannot take advantage of multiple cores of a CPU easily. Especially when you think it is heavily used in data analysis. We use Python for data analysis as well, and for 95% of operations we are doing, numpy is fast enough that we don't have any complaints. But sometimes, we do wish to be able to take advantage of all the cores in our CPUs, espec…
If you are genuinely hitting a wall like that surely it's time to move on from python? Or at least think about it
I feel like the #1 downside of Python for the last few years is that you cannot take advantage of multiple cores of a CPU easily. Especially when you think it is heavily used in data analysis. We use Python for data analysis as well, and for 95% of operations we are doing, numpy is fast enough that we don't have any complaints. But sometimes, we do wish to be able to take advantage of all the cores in our CPUs, espec…
Recent versions of Python (3.8+) have introduced a SharedMemory class for sharing memory between different processes (see https://docs.python.org/3/library/multiprocessing.shared_mem... ). The implementation may still be a bit buggy though, so use with caution! https://bugs.python.org/issue38119
I feel like the #1 downside of Python for the last few years is that you cannot take advantage of multiple cores of a CPU easily. Especially when you think it is heavily used in data analysis. We use Python for data analysis as well, and for 95% of operations we are doing, numpy is fast enough that we don't have any complaints. But sometimes, we do wish to be able to take advantage of all the cores in our CPUs, espec…
Also a big downside with JavaScript. Of course both Python and JS are high-level interpreted languages where high-performance use cases aren't the foremost priority.