Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Test for lists in Cython

github.com

Post reply on HN
Print view View original

11–20 of 147 posts

Re: Test for lists in Cython

#11

Post by yakubin »

ur-whale wrote: post #3

From the benchmarks: > Rust is not that fast because it needs to copy data; I'm surprised. Don't know much about Rust, but isn't it hailed as being competitive with C/C++ ?

You left out the explanation from your quote. The full is:

> Rust is not that fast beacuse it needs to copy data; using Pyo3 objects would probably lead to similar results as cython, but with an added library.

"It needs to copy data", because it's converting Python objects into Rust objects and back again. As the full quote states, it could be written in a different way, although that would make the code look really weird.

In this benchmark there is also no normal C++ involved. There is only Cython using some C++ ints, but operating on Python lists. And that's not handled by writing C++ code that operates on Python structures, but writing Python code in Cython in a "C++ mode".

Apples to oranges.

Re: Test for lists in Cython

#12

Post by andi999 »

ZuLuuuuuu wrote: post #8

I feel like the #1 downside of Python for the last few years is that you cannot take advantage of multiple cores of a CPU easily. Especially when you think it is heavily used in data analysis. We use Python for data analysis as well, and for 95% of operations we are doing, numpy is fast enough that we don't have any complaints. But sometimes, we do wish to be able to take advantage of all the cores in our CPUs, espec…

Why not use the multiprocessing module? (if you can partition your problem space). I find memory sharing much more difficult to reliably implement an algorithm, and speed gain might be lower through cache conflicts.

Re: Test for lists in Cython

#13

Post by mhh__ »

ZuLuuuuuu wrote: post #8

I feel like the #1 downside of Python for the last few years is that you cannot take advantage of multiple cores of a CPU easily. Especially when you think it is heavily used in data analysis. We use Python for data analysis as well, and for 95% of operations we are doing, numpy is fast enough that we don't have any complaints. But sometimes, we do wish to be able to take advantage of all the cores in our CPUs, espec…

If you are genuinely hitting a wall like that surely it's time to move on from python? Or at least think about it

Re: Test for lists in Cython

#14

Post by szubie »

ZuLuuuuuu wrote: post #8

I feel like the #1 downside of Python for the last few years is that you cannot take advantage of multiple cores of a CPU easily. Especially when you think it is heavily used in data analysis. We use Python for data analysis as well, and for 95% of operations we are doing, numpy is fast enough that we don't have any complaints. But sometimes, we do wish to be able to take advantage of all the cores in our CPUs, espec…

Recent versions of Python (3.8+) have introduced a SharedMemory class for sharing memory between different processes (see https://docs.python.org/3/library/multiprocessing.shared_mem...).

The implementation may still be a bit buggy though, so use with caution! https://bugs.python.org/issue38119

Re: Test for lists in Cython

#15

Post by ekianjo »

ZuLuuuuuu wrote: post #8

I feel like the #1 downside of Python for the last few years is that you cannot take advantage of multiple cores of a CPU easily. Especially when you think it is heavily used in data analysis. We use Python for data analysis as well, and for 95% of operations we are doing, numpy is fast enough that we don't have any complaints. But sometimes, we do wish to be able to take advantage of all the cores in our CPUs, espec…

> Python for the last few years is that you cannot take advantage of multiple cores of a CPU easily.

Same thing in the R ecosystem, it's possible to use multiple cores with the parallel package but there are caveats too.

Re: Test for lists in Cython

#16

Post by volta83 »

urschrei wrote: post #9

Rust doesn’t need to copy the data. It’s trivial to pass e.g. Numpy arrays to Rust as slices via Cython (let alone originating in Cython!), modify them, and return them, or use them as input for a new returned struct. https://github.com/urschrei/simplification https://github.com/urschrei/lonlat_bng https://github.com/urschrei/pypolyline Each of those repos has links to the corresponding Rust “shim” libraries that pro…

To add, it is trivial to guarantee that Rust code is "zero-copy", so if this is something you care about, Rust allows your program to fail to compile if it tries to make a copy.

Re: Test for lists in Cython

#18

Post by mhh__ »

ur-whale wrote: post #3

From the benchmarks: > Rust is not that fast because it needs to copy data; I'm surprised. Don't know much about Rust, but isn't it hailed as being competitive with C/C++ ?

You mean C++.

C usually comes out a bit worse because it gives even less information to the optimizer.

And thanks to the GNU/LLVM monoculture most languages should perform roughly the same if they have them as a target e.g. I have found that D makes its it ridiculously easy to write highly specialized code that is both readable and visible to the optimizer (I will be blogging about that part later), but I'm sure I could force the same asm out from C++ or Rust etc.

Re: Test for lists in Cython

#19

Post by volta83 »

yakubin wrote: post #11
ur-whale wrote: post #3

From the benchmarks: > Rust is not that fast because it needs to copy data; I'm surprised. Don't know much about Rust, but isn't it hailed as being competitive with C/C++ ?

You left out the explanation from your quote. The full is: > Rust is not that fast beacuse it needs to copy data; using Pyo3 objects would probably lead to similar results as cython, but with an added library. "It needs to copy data", because it's converting Python objects into Rust objects and back again. As the full quote states, it could be written in a different way, although that would make the code look really…

> although that would make the code look really weird.

Not it wouldn't?

Using Pyo3 is arguably the most idiomatic way of interfacing Rust with python. The "added library" cost mentioned as an excuse not to use it is literally the 1 second it takes me to write `cargo add pyo3` .

They didn't do this because it would make their claim that "Julia is the better language for extending Python" kind of moot.

They started from the conclusion that they wanted to have, and worked their way backwards towards implementation and data that would support it.

Re: Test for lists in Cython

#20

Post by mhh__ »

andi999 wrote: post #12
ZuLuuuuuu wrote: post #8

I feel like the #1 downside of Python for the last few years is that you cannot take advantage of multiple cores of a CPU easily. Especially when you think it is heavily used in data analysis. We use Python for data analysis as well, and for 95% of operations we are doing, numpy is fast enough that we don't have any complaints. But sometimes, we do wish to be able to take advantage of all the cores in our CPUs, espec…

Why not use the multiprocessing module? (if you can partition your problem space). I find memory sharing much more difficult to reliably implement an algorithm, and speed gain might be lower through cache conflicts.

> cache conflicts

This can be an absolute sinkhole for performance, but if you have a large dataset you shouldn't be sharing lines that much one would hope.

Post reply on HN