Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Test for lists in Cython

github.com

Post reply on HN
Print view View original

41–50 of 147 posts

Re: Test for lists in Cython

#41

Post by chalst »

urschrei wrote: post #38
chalst wrote: post #31

Earlier quoted context omitted.

There is no problem at all interfacing Julia's C API to Rust. It's a shame that C++ and Rust are bad fits together, but this actually strengthens the argument for Julia as the extension language rather than Rust, since Julia interfaces easily to not just C but also the C++ code in which much of the world's best numerical algorithms are written, and has an unrivalled FFI to Python, while Rust's object system is a poor…

> There is no problem at all interfacing Julia's C API to Rust Nobody claimed there was a problem? > As for the FUD about interfacing JIT code to C/C++ I don't know what you're talking about. Sorry!

I misread your comment. I still consider what you wrote contained FUD, at least with respect to the last paragraph.

Re: Test for lists in Cython

#42

Post by BlackFly »

andi999 wrote: post #12
ZuLuuuuuu wrote: post #8

I feel like the #1 downside of Python for the last few years is that you cannot take advantage of multiple cores of a CPU easily. Especially when you think it is heavily used in data analysis. We use Python for data analysis as well, and for 95% of operations we are doing, numpy is fast enough that we don't have any complaints. But sometimes, we do wish to be able to take advantage of all the cores in our CPUs, espec…

Why not use the multiprocessing module? (if you can partition your problem space). I find memory sharing much more difficult to reliably implement an algorithm, and speed gain might be lower through cache conflicts.

A real hard requirement is going to be resource pools such as session linked connection pools with associated data. But there are just plenty of places where you don't want to be forced into a message passing architecture for a service and shared memory may be a better fit.

Re: Test for lists in Cython

#43

Post by short_sells_poo »

I believe the toolchain of Rust is nicer in that you get a relatively small sized and self-contained rust library that can be easily distributed with a python package. Julia can't be easily bundled this way because you need to ship the entire runtime, with all the gubbins this entails.

On the other hand: having used pyo3 to integrate rust with python in the past, the biggest pain is simply to reconcile the dynamism of python with AOT compiled rust code. There's a lot of noise at the interface from the large amount of type checking to unpack the specific types of numpy arrays coming across. Do you want to have your rust code work with all sorts of integer bit lengths? That'll be a code path for each. If you have a number of input types, have fun coercing them all.

This can be alleviated with macros to a degree, but that just hides the problem really.

Julia is JIT compiled, which means that there are no problems with types being determined only at run time. In fact, this one fact makes integration at the code level much nicer with Julia.

So all in all it's a tradeoff really. I also found the tooling for Rust to be much more robust and stable (1 year ago admittedly).

Re: Test for lists in Cython

#44

Post by jeremiecoullon »

ZuLuuuuuu wrote: post #8

I feel like the #1 downside of Python for the last few years is that you cannot take advantage of multiple cores of a CPU easily. Especially when you think it is heavily used in data analysis. We use Python for data analysis as well, and for 95% of operations we are doing, numpy is fast enough that we don't have any complaints. But sometimes, we do wish to be able to take advantage of all the cores in our CPUs, espec…

I've been using JAX (https://jax.readthedocs.io/en/latest/) for scientific computing in general (in particular MCMC algorithms), as it's really fast. Even on a CPU you get massive speedups compared to numpy (can be up to 2 or 3 orders of magnitude faster in some cases).

The main selling point of the library is automatic differential and compilation to XLA, but I've been using it even when I don't need gradients, as it's really fast (due to compilation). I also really like the random number generator as it's very good for reproducibility.

I've played around with Julia in the past and really liked it, but in terms speed Jax has pretty much solved that problem for me

Re: Test for lists in Cython

#45

Post by jakobnissen »

I'm a massive Julia fanboy, but I would not extend Python with Julia if I could choose not to. Julia has a _massive_ runtime with a hello-world script consuming 150 MB of RAM, not to mention the dreaded startup-time.

It's better and easier to use Julia as your main top-level "glue" language and call Python/Rust/C from Julia. Julia is in many ways a better glue language than Python - better multithreading, easier calling into C/Rust, etc. Then, over time, to the extent it is practical, you can replace foreign code with Julia code - because Julia is fast enough that it actually makes sense to do so.

If you already have a large Python code base and can't switch the top-level language to Julia, I would just not use Julia for that project.

Re: Test for lists in Cython

#46

Post by PartiallyTyped »

jeremiecoullon wrote: post #44
ZuLuuuuuu wrote: post #8

I feel like the #1 downside of Python for the last few years is that you cannot take advantage of multiple cores of a CPU easily. Especially when you think it is heavily used in data analysis. We use Python for data analysis as well, and for 95% of operations we are doing, numpy is fast enough that we don't have any complaints. But sometimes, we do wish to be able to take advantage of all the cores in our CPUs, espec…

I've been using JAX ( https://jax.readthedocs.io/en/latest/ ) for scientific computing in general (in particular MCMC algorithms), as it's really fast. Even on a CPU you get massive speedups compared to numpy (can be up to 2 or 3 orders of magnitude faster in some cases). The main selling point of the library is automatic differential and compilation to XLA, but I've been using it even when I don't need gradients, as…

Have you tried numba+numpy? In my experience, it is much faster than Jax and can compile to cuda. It's not caveat free, but it also removes the hustle of labeling arrays as donated in Jax.

You may find this interesting https://github.com/scikit-hep/iminuit/blob/develop/tutorial/...

Re: Test for lists in Cython

#47

Post by CogitoCogito »

ZuLuuuuuu wrote: post #8

I feel like the #1 downside of Python for the last few years is that you cannot take advantage of multiple cores of a CPU easily. Especially when you think it is heavily used in data analysis. We use Python for data analysis as well, and for 95% of operations we are doing, numpy is fast enough that we don't have any complaints. But sometimes, we do wish to be able to take advantage of all the cores in our CPUs, espec…

> We use Python for data analysis as well, and for 95% of operations we are doing, numpy is fast enough that we don't have any complaints. But sometimes, we do wish to be able to take advantage of all the cores in our CPUs, especially now that we can easily get an 8 core CPU for a reasonable price.

I understand the sentiment, but in this case I wonder if it is less a python issue and more an issue with numpy. I don't see any reason why numpy couldn't execute the last line of the following multi-threaded:

    >>> import numpy as np
    >>> a = np.arange(1000000000)
    >>> b = 2 \* a
The same thing applies to matrix multiplication. I'm guessing numpy has chosen not to for good reason though.

edit: Actually it seems numpy is multi-threaded under certain circumstances:

https://stackoverflow.com/questions/16617973/why-isnt-numpy-...

Re: Test for lists in Cython

#48

Post by duckerude »

urschrei wrote: post #9

Rust doesn’t need to copy the data. It’s trivial to pass e.g. Numpy arrays to Rust as slices via Cython (let alone originating in Cython!), modify them, and return them, or use them as input for a new returned struct. https://github.com/urschrei/simplification https://github.com/urschrei/lonlat_bng https://github.com/urschrei/pypolyline Each of those repos has links to the corresponding Rust “shim” libraries that pro…

I'm also getting a 30% speedup simply from manipulating &PyList references instead of Vecs (without parallelism).

Re: Test for lists in Cython

#49

Post by galangalalgol »

jakobnissen wrote: post #45

I'm a massive Julia fanboy, but I would not extend Python with Julia if I could choose not to. Julia has a _massive_ runtime with a hello-world script consuming 150 MB of RAM, not to mention the dreaded startup-time. It's better and easier to use Julia as your main top-level "glue" language and call Python/Rust/C from Julia. Julia is in many ways a better glue language than Python - better multithreading, easier call…

You seem to understand Julia well. Is there a reason there is no nexus plugin or way to mirror the julia repo for dev networks that dont have unrestricted access to the internet, or are even airgapped. I see multiple people asking this online, so it is a common enough problem, but it seems like people are saying the Julia approach makes it hard to support.

Re: Test for lists in Cython

#50

Post by gbrown »

curiousgal wrote: post #27
ekianjo wrote: post #15

Earlier quoted context omitted.

> Python for the last few years is that you cannot take advantage of multiple cores of a CPU easily. Same thing in the R ecosystem, it's possible to use multiple cores with the parallel package but there are caveats too.

I haven't run into any issues with that package honestly (mclapply ftw!). Other than not being able to use it on Windows of course.

It’s process based, so communication overhead is heavy and the infrastructure is minimal - if you’re distributing jobs which are individually large it works fine, but it can’t really be used to replace multithreading in a shared memory model.
Post reply on HN