Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Test for lists in Cython

github.com

Post reply on HN
Print view View original

31–40 of 147 posts

Re: Test for lists in Cython

#31

Post by chalst »

urschrei wrote: post #9

Rust doesn’t need to copy the data. It’s trivial to pass e.g. Numpy arrays to Rust as slices via Cython (let alone originating in Cython!), modify them, and return them, or use them as input for a new returned struct. https://github.com/urschrei/simplification https://github.com/urschrei/lonlat_bng https://github.com/urschrei/pypolyline Each of those repos has links to the corresponding Rust “shim” libraries that pro…

There is no problem at all interfacing Julia's C API to Rust. It's a shame that C++ and Rust are bad fits together, but this actually strengthens the argument for Julia as the extension language rather than Rust, since Julia interfaces easily to not just C but also the C++ code in which much of the world's best numerical algorithms are written, and has an unrivalled FFI to Python, while Rust's object system is a poor fit for either Python or C++.

> As a more general comment, using a GC language as the FFI target from a GC language is begging for difficult-if-not-impossible-to-debug crashes down the line.

As for the FUD about interfacing JIT code to C/C++, this is a problem Julia was designed from the outset to tackle. Incidentally, Julia built on the excellent experience LuaJIT has had. A challenge: can you name a particular extension that would interface better from Rust than from Julia?

Oops, I misunderstood the criticism you were hinting at but failing to justify, sorry. OK, Julia already does interface to Python and this interface sees widespread use. If your suspicion is right, then where are the horror stories from people who were bitten when deploying code that built on the interface?

Re: Test for lists in Cython

#32

Post by andi999 »

mhh__ wrote: post #20
andi999 wrote: post #12

Earlier quoted context omitted.

Why not use the multiprocessing module? (if you can partition your problem space). I find memory sharing much more difficult to reliably implement an algorithm, and speed gain might be lower through cache conflicts.

> cache conflicts This can be an absolute sinkhole for performance, but if you have a large dataset you shouldn't be sharing lines that much one would hope.

Then you shd be able to do process parallelization.

Re: Test for lists in Cython

#33

Post by mhh__ »

ZuLuuuuuu wrote: post #28
mhh__ wrote: post #13

Earlier quoted context omitted.

If you are genuinely hitting a wall like that surely it's time to move on from python? Or at least think about it

The thing is that the current ecosystem (numpy + scipy + pandas + PyCharm) fits our company structure perfectly (very few software engineers, mostly test engineers who are not very proficient coders). And we already have tens of thousands lines of code. So changing the whole ecosystem just because 5% of the problems are slow is too big of a jump for us. For now, it is easier for us to write a bit of C code for places…

Do you know what you are actually stalled on? i.e. Memory or Instructions, if it's the latter you can probably eek out some more performance just by fiddling with compiler flags (you'd be surprised simultaneously how clever and how utterly braindead some compilers are if you play with them on Compiler Explorer for a bit)

Also, if you are running on Intel you might have some luck with Intel's profiling tools as, although I've never used them for python they are by far and away the best things in this area, they do support Python.

I don't know what details they give you in from Python code directly, but if you've used perf to profile code at a low level before, vTune is like a Tom & Jerry shotgun in comparison.

Re: Test for lists in Cython

#34

Post by anentropic »

ZuLuuuuuu wrote: post #29
szubie wrote: post #14

Earlier quoted context omitted.

Recent versions of Python (3.8+) have introduced a SharedMemory class for sharing memory between different processes (see https://docs.python.org/3/library/multiprocessing.shared_mem... ). The implementation may still be a bit buggy though, so use with caution! https://bugs.python.org/issue38119

Thank you, I'll definitely check it out!

There are also some libraries built on top of it that might be useful https://github.com/dillonalaird/shared_numpy

Re: Test for lists in Cython

#35

Post by Sukera »

hobofan wrote: post #22
PartiallyTyped wrote: post #10

Earlier quoted context omitted.

The culprit here is `a_list: Vec >`, which means the list of lists is copied as a whole, hence why the rust version is slower. In general, rust is at least as fast as C or C++, but this seems to be a special case.

It's also repeatedly `push`-ing elements into the `Vec`, which will likely lead to repeated allocation and copying. In a case like this were you pretty clearly know the size beforehand you should crate a Vec of the appropriate size with `Vec::reserve`.

The julia version also does that though

Re: Test for lists in Cython

#36

Post by azangru »

zozbot234 wrote: post #30
ZuLuuuuuu wrote: post #8

I feel like the #1 downside of Python for the last few years is that you cannot take advantage of multiple cores of a CPU easily. Especially when you think it is heavily used in data analysis. We use Python for data analysis as well, and for 95% of operations we are doing, numpy is fast enough that we don't have any complaints. But sometimes, we do wish to be able to take advantage of all the cores in our CPUs, espec…

> I feel like the #1 downside of Python for the last few years is that you cannot take advantage of multiple cores of a CPU easily. Also a big downside with JavaScript. Of course both Python and JS are high-level interpreted languages where high-performance use cases aren't the foremost priority.

> Also a big downside with JavaScript.

Can't Node run multiple processes (or multiple workers, not sure of the proper terminology) on multiple cores? As discussed in this thread on StackOverflow: https://stackoverflow.com/questions/61893497/node-js-on-mult...

Re: Test for lists in Cython

#37

Post by pletnes »

ZuLuuuuuu wrote: post #8

I feel like the #1 downside of Python for the last few years is that you cannot take advantage of multiple cores of a CPU easily. Especially when you think it is heavily used in data analysis. We use Python for data analysis as well, and for 95% of operations we are doing, numpy is fast enough that we don't have any complaints. But sometimes, we do wish to be able to take advantage of all the cores in our CPUs, espec…

Numpy operations release the GIL (usually at least) so you can use a threadpool and, indeed, share memory. Just try it and you may be pleasantly surprised.

Dask is great if you’re processing large amounts of data, and it recommends and supports threads for this reason.

Re: Test for lists in Cython

#38

Post by urschrei »

chalst wrote: post #31
urschrei wrote: post #9

Rust doesn’t need to copy the data. It’s trivial to pass e.g. Numpy arrays to Rust as slices via Cython (let alone originating in Cython!), modify them, and return them, or use them as input for a new returned struct. https://github.com/urschrei/simplification https://github.com/urschrei/lonlat_bng https://github.com/urschrei/pypolyline Each of those repos has links to the corresponding Rust “shim” libraries that pro…

There is no problem at all interfacing Julia's C API to Rust. It's a shame that C++ and Rust are bad fits together, but this actually strengthens the argument for Julia as the extension language rather than Rust, since Julia interfaces easily to not just C but also the C++ code in which much of the world's best numerical algorithms are written, and has an unrivalled FFI to Python, while Rust's object system is a poor…

> There is no problem at all interfacing Julia's C API to Rust

Nobody claimed there was a problem?

> As for the FUD about interfacing JIT code to C/C++

I don't know what you're talking about. Sorry!

Re: Test for lists in Cython

#39

Post by Nimitz14 »

one word: pybind11

I honestly think people are severely underestimating what a massive impact this is currently having in increasing productivity of python devs who know a little C++.

Re: Test for lists in Cython

#40

Post by zozbot234 »

azangru wrote: post #36
zozbot234 wrote: post #30

Earlier quoted context omitted.

> I feel like the #1 downside of Python for the last few years is that you cannot take advantage of multiple cores of a CPU easily. Also a big downside with JavaScript. Of course both Python and JS are high-level interpreted languages where high-performance use cases aren't the foremost priority.

> Also a big downside with JavaScript. Can't Node run multiple processes (or multiple workers, not sure of the proper terminology) on multiple cores? As discussed in this thread on StackOverflow: https://stackoverflow.com/questions/61893497/node-js-on-mult...

> Can't Node run multiple processes

Of course, but you can also run multiple Python processes.

Post reply on HN