Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Test for lists in Cython

github.com

Post reply on HN
Print view View original

61–70 of 147 posts

Re: Test for lists in Cython

#61

Post by vchuravy »

Gravityloss wrote: post #59

I was a huge fan of Matlab way back. I wrote a hundred small Matlab programs for usage in the research department of the company I worked in. Doing data operations in Matlab was way more elegant than in say, Numpy which I tried later. Development was fast and ergonomics were good. After using Ruby for years, returning to Matlab style code in Julia felt somewhat awkward. Instead of my_array.length you have length(my_a…

Broadcasting and map are two different operations. If all the inouts have the same shape broadcast is equivalent to map, but in Julia you can also just use map.

`map(el->el^2, a)`

and inspired by Ruby

``` map(a) do element log(element) end ```

The latter being syntax sugar for the former.

Re: Test for lists in Cython

#62

Post by galangalalgol »

jakobnissen wrote: post #53
galangalalgol wrote: post #49

Earlier quoted context omitted.

You seem to understand Julia well. Is there a reason there is no nexus plugin or way to mirror the julia repo for dev networks that dont have unrestricted access to the internet, or are even airgapped. I see multiple people asking this online, so it is a common enough problem, but it seems like people are saying the Julia approach makes it hard to support.

I'm sorry, I have no idea, but I'm sure someone else has insight. Why would you want to mirror the Julia repo instead of just downloading already-compiled binaries? Do you want to maintain your own fork with your own set of patches?

Figuring the whole dependency tree without the package manager would be difficult and the build file sometimes has other side effects besides building for many packages. And downloading the binaries rather than letting the source go through nexus would be sidestepping most of the network security (and maybe legal) reasons people use nexus.

Re: Test for lists in Cython

#63

Post by leephillips »

Gravityloss wrote: post #59

I was a huge fan of Matlab way back. I wrote a hundred small Matlab programs for usage in the research department of the company I worked in. Doing data operations in Matlab was way more elegant than in say, Numpy which I tried later. Development was fast and ergonomics were good. After using Ruby for years, returning to Matlab style code in Julia felt somewhat awkward. Instead of my_array.length you have length(my_a…

The standard way to make a histogram in Julia is histogram(data)

Using the latest version (1.6 - although 1.6.1 just came out) the time to first plot is just a few seconds. After that, plotting in the REPL is instantaneous.

I probably don’t understand what you’re getting at when you speak of making frequent changes to code. REPL-based development in Julia is excellent, and there are Pluto notebooks as well.

Re: Test for lists in Cython

#64

Post by bjourne »

urschrei wrote: post #9

Rust doesn’t need to copy the data. It’s trivial to pass e.g. Numpy arrays to Rust as slices via Cython (let alone originating in Cython!), modify them, and return them, or use them as input for a new returned struct. https://github.com/urschrei/simplification https://github.com/urschrei/lonlat_bng https://github.com/urschrei/pypolyline Each of those repos has links to the corresponding Rust “shim” libraries that pro…

> As a more general comment, using a GC language as the FFI target from a GC language is begging for difficult-if-not-impossible-to-debug crashes down the line.

Not true!

What you do is that you keep a registry for objects passed from the host vm to the foreign vm in which you register objects thus transferred. And you use a similar mechanism for objects passed from the foreign vm to the host vm. In CPython, you simply increment the refcount to prevent Python from collecting them prematurely.

This is how Java does it (via JNI) and how many gc:ed runtimes interact with other gc:ed runtimes. This is how you do it in Rust too since Rust can't tell how long an object passed to a foreign vm is supposed to last.

Re: Test for lists in Cython

#65

Post by syntonym2 »

I use python in a scientific context, but have so far not written much extensions for python in any of the languages tested. I'm interested in some guidance which language a) is easy to integrate with python and b) has some good performance, but this benchmark lacks the details to come to any conclusion.

I tried to run the benchmark on my own computer, but the setup documentation was not enough for me to get the julia integration running. I haven't used julia before, so it might just be something very simple.

Similar I haven't used poetry much before, and the given documentation failed to install the necessary setuptools-rust for me. I could fix it on my own, but doesn't make me feel certain about the outcome of the benchmark.

The rust benchmark did not reproduce for me: "Rust (Pyo3) parallel after" has a 1.56 speedup for me, but a 2.6x slowdown for the author. Also I don't understand what the difference between "after" and "before" is, the code just calls the same code twice. Might be a JIT/Cache thing, but it's unclear to me. One sentence what before/after refers to would be very helpful.

Generally all measurements are only done once. Measuring at least thrice gives one at least a chance to detect an outlier and gives possibility for statistics, e.g. is a difference betwee the different cython annotations even meanginful?

The "C Cython (pure-python mode)" is reported faster then "C Cython (.pyx)". The Cython project itself says that using pyx files should be faster, so something strange is going on.

"Cython is fast, but none of these methods are able to release the GIL. " (A) this is not true (B) this seems to be mostly over single threaded performance, so why is that meaningful?

"Rust is not that fast beacuse it needs to copy data; using Pyo3 objects would probably lead to similar results as cython, but with an added library." The rust code already contains Pyo3, so an "added library" is not necessary as far as I understand.

I'd guess the performance stems more from conversions between different types then anything else. Maybe Julia (and the python-julia bridge) is particular smart about it and thus it's super easy to use, while pyo3 (and cython) needs some more work to interface with python. Even if that is true, I couldn't say it from the presented data.

With these caveats resolved I'd be interested in the benchmark, but without it I can't really say anything from it.

Re: Test for lists in Cython

#66

Post by grej »

cfcf14 wrote: post #51
PartiallyTyped wrote: post #46

Earlier quoted context omitted.

Have you tried numba+numpy? In my experience, it is much faster than Jax and can compile to cuda. It's not caveat free, but it also removes the hustle of labeling arrays as donated in Jax. You may find this interesting https://github.com/scikit-hep/iminuit/blob/develop/tutorial/...

Have you been successful in implementing non-trivial computational code in numba/numpy? I've always found it starts to really break for anything which isn't really trivial, and the errors are mostly non-prescriptive and highly verbose.

We have a very large portion of production code written in numba we’ve been running for about 3 years and I made a small contribution to the library. There are a lot of gotchas to numba when the codebase gets large but the benefits far outweigh the downsides. I highly recommend numba.

Edit: also note that a big part of the umap library is written in numba.

Re: Test for lists in Cython

#67

Post by leephillips »

jhgb wrote: post #60
Gravityloss wrote: post #59

I was a huge fan of Matlab way back. I wrote a hundred small Matlab programs for usage in the research department of the company I worked in. Doing data operations in Matlab was way more elegant than in say, Numpy which I tried later. Development was fast and ergonomics were good. After using Ruby for years, returning to Matlab style code in Julia felt somewhat awkward. Instead of my_array.length you have length(my_a…

> Instead of my_array.length you have length(my_array). In my personal preference the method call is just a nicer way of doing the same thing. Well...they're both method calls, aren't they? (So they're both the nicer way?)

Those who are attached to the class-based OOP model used, for example, in Python will find the dot notation more natural. But Julia’s multiple dispatch is a superset of this, and unarguably more powerful and flexible. Python OOP leads to monstrosities like

       >>> ', '.join(['1', '2', '3'])
     '1, 2, 3'

       >>> '1, 2, 3'.split(', ')
     ['1', '2', '3']

Re: Test for lists in Cython

#68

Post by zozbot234 »

bjourne wrote: post #64
urschrei wrote: post #9

Rust doesn’t need to copy the data. It’s trivial to pass e.g. Numpy arrays to Rust as slices via Cython (let alone originating in Cython!), modify them, and return them, or use them as input for a new returned struct. https://github.com/urschrei/simplification https://github.com/urschrei/lonlat_bng https://github.com/urschrei/pypolyline Each of those repos has links to the corresponding Rust “shim” libraries that pro…

> As a more general comment, using a GC language as the FFI target from a GC language is begging for difficult-if-not-impossible-to-debug crashes down the line. Not true! What you do is that you keep a registry for objects passed from the host vm to the foreign vm in which you register objects thus transferred. And you use a similar mechanism for objects passed from the foreign vm to the host vm. In CPython, you simp…

> This ... is how many gc:ed runtimes interact with other gc:ed runtimes.

It's not that easy. You'd need to register the object as a GC root as long as it's being managed by the foreign GC, and similarly ensure that objects in the foreign VM are properly "de-registered" when the local GC finalizes them. It's far from trivial, particularly when compared with other memory-management strategies.

Re: Test for lists in Cython

#69

Post by leephillips »

CogitoCogito wrote: post #47
ZuLuuuuuu wrote: post #8

I feel like the #1 downside of Python for the last few years is that you cannot take advantage of multiple cores of a CPU easily. Especially when you think it is heavily used in data analysis. We use Python for data analysis as well, and for 95% of operations we are doing, numpy is fast enough that we don't have any complaints. But sometimes, we do wish to be able to take advantage of all the cores in our CPUs, espec…

> We use Python for data analysis as well, and for 95% of operations we are doing, numpy is fast enough that we don't have any complaints. But sometimes, we do wish to be able to take advantage of all the cores in our CPUs, especially now that we can easily get an 8 core CPU for a reasonable price. I understand the sentiment, but in this case I wonder if it is less a python issue and more an issue with numpy. I don't…

Numpy will perform a parallelized calculation when using one of the underlying BLAS routines, if you have the appropriate libraries installed on your machine. So to take advantage of your cores, try to write the calculation as a call to a linear algebra function. Loops or expressions with arithmetic operators as in your example will run on one core. Or, just use Julia.

Re: Test for lists in Cython

#70

Post by urschrei »

zozbot234 wrote: post #68
bjourne wrote: post #64

Earlier quoted context omitted.

> As a more general comment, using a GC language as the FFI target from a GC language is begging for difficult-if-not-impossible-to-debug crashes down the line. Not true! What you do is that you keep a registry for objects passed from the host vm to the foreign vm in which you register objects thus transferred. And you use a similar mechanism for objects passed from the foreign vm to the host vm. In CPython, you simp…

> This ... is how many gc:ed runtimes interact with other gc:ed runtimes. It's not that easy. You'd need to register the object as a GC root as long as it's being managed by the foreign GC, and similarly ensure that objects in the foreign VM are properly "de-registered" when the local GC finalizes them. It's far from trivial, particularly when compared with other memory-management strategies.

Thanks – this is the more informed version of what I wrote.
Post reply on HN