Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Test for lists in Cython

github.com

Post reply on HN
Print view View original

111–120 of 147 posts

Re: Test for lists in Cython

#111

Post by willseth »

zozbot234 wrote: post #30
ZuLuuuuuu wrote: post #8

I feel like the #1 downside of Python for the last few years is that you cannot take advantage of multiple cores of a CPU easily. Especially when you think it is heavily used in data analysis. We use Python for data analysis as well, and for 95% of operations we are doing, numpy is fast enough that we don't have any complaints. But sometimes, we do wish to be able to take advantage of all the cores in our CPUs, espec…

> I feel like the #1 downside of Python for the last few years is that you cannot take advantage of multiple cores of a CPU easily. Also a big downside with JavaScript. Of course both Python and JS are high-level interpreted languages where high-performance use cases aren't the foremost priority.

> Of course both Python and JS are high-level interpreted languages where high-performance use cases aren't the foremost priority.

Modern JS engines are all primarily JIT-based, generally only interpreting code when it's faster than waiting on the JIT. JS JITs are quite good now, and in many cases will produce optimized C-equivalent compiled code from fairly naive JS.

Re: Test for lists in Cython

#112

Post by [deleted] »

[deleted] wrote: post #105
urschrei wrote: post #9

Rust doesn’t need to copy the data. It’s trivial to pass e.g. Numpy arrays to Rust as slices via Cython (let alone originating in Cython!), modify them, and return them, or use them as input for a new returned struct. https://github.com/urschrei/simplification https://github.com/urschrei/lonlat_bng https://github.com/urschrei/pypolyline Each of those repos has links to the corresponding Rust “shim” libraries that pro…

[deleted]

[deleted]

Re: Test for lists in Cython

#113

Post by leephillips »

jhgb wrote: post #110
leephillips wrote: post #109

Earlier quoted context omitted.

Yes, but those are not classes, in the sense of Python or other OOP language classes.

Named types that can be dispatched on are classes, see CLOS. Hell, even in Python, classes are pretty much the named types that you can dispatch on, although of course, in Python, you generally can't create different kinds of types that can't be dispatched on like types satisfying a predicate - at least to my knowledge you can't, so the difference is somewhat less tangible there.

In Julia they are not classes in the sense that the term is normally used in, for example, Python. Classes bind together data and methods, which is what leads to all the problems. In Julia functions are generic, and specialized to methods acting on arbitrary combinations of types; they are freed from the data.

https://arstechnica.com/science/2020/10/the-unreasonable-eff...

Re: Test for lists in Cython

#114

Post by [deleted] »

[deleted] wrote: post #105
urschrei wrote: post #9

Rust doesn’t need to copy the data. It’s trivial to pass e.g. Numpy arrays to Rust as slices via Cython (let alone originating in Cython!), modify them, and return them, or use them as input for a new returned struct. https://github.com/urschrei/simplification https://github.com/urschrei/lonlat_bng https://github.com/urschrei/pypolyline Each of those repos has links to the corresponding Rust “shim” libraries that pro…

[deleted]

[deleted]

Re: Test for lists in Cython

#115

Post by jhgb »

leephillips wrote: post #113
jhgb wrote: post #110

Earlier quoted context omitted.

Named types that can be dispatched on are classes, see CLOS. Hell, even in Python, classes are pretty much the named types that you can dispatch on, although of course, in Python, you generally can't create different kinds of types that can't be dispatched on like types satisfying a predicate - at least to my knowledge you can't, so the difference is somewhat less tangible there.

In Julia they are not classes in the sense that the term is normally used in, for example, Python. Classes bind together data and methods, which is what leads to all the problems. In Julia functions are generic, and specialized to methods acting on arbitrary combinations of types; they are freed from the data. https://arstechnica.com/science/2020/10/the-unreasonable-eff...

> Classes bind together data and methods

Not in CLOS, from which Julia's mechanisms are derived (which leads to the question why not CLOS but Python of all things should be used as a source of "normal sense" of anything Julia-related). In CLOS, classes bind together data, generic functions name abstract operations, and methods represent specific code that deals with implementing a generic function for a particular combination of type arguments.

> which is what leads to all the problems

I don't think anyone is disputing that here. There's a reason why CLOS didn't do any of that.

> In Julia functions are generic, and specialized to methods acting on arbitrary combinations of types; they are freed from the data.

...yes, just like in CLOS (unsurprisingly, given Julia's heritage), and those types are effectively CLOS classes.

Re: Test for lists in Cython

#116

Post by hobofan »

yakubin wrote: post #26
volta83 wrote: post #19

Earlier quoted context omitted.

> although that would make the code look really weird. Not it wouldn't? Using Pyo3 is arguably the most idiomatic way of interfacing Rust with python. The "added library" cost mentioned as an excuse not to use it is literally the 1 second it takes me to write `cargo add pyo3` . They didn't do this because it would make their claim that "Julia is the better language for extending Python" kind of moot. They started fro…

> Not it wouldn't? I'll defer to your knowledge on that. I generally don't work with Python. I've only looked into interfacing C# with Rust and it looked crazy, so that's where I'm coming from. But yes, it's pretty clear that the results are more a consequence of what's in the source files than the languages used (if we can even say that e.g. C++ was used at all).

> interfacing C# with Rust

AFAIK, there is no library that achieves a similar idiomatic interoperability between Rust and C#, like PyO3 for Python.

Re: Test for lists in Cython

#117

Post by leephillips »

jhgb wrote: post #115
leephillips wrote: post #113

Earlier quoted context omitted.

In Julia they are not classes in the sense that the term is normally used in, for example, Python. Classes bind together data and methods, which is what leads to all the problems. In Julia functions are generic, and specialized to methods acting on arbitrary combinations of types; they are freed from the data. https://arstechnica.com/science/2020/10/the-unreasonable-eff...

> Classes bind together data and methods Not in CLOS, from which Julia's mechanisms are derived (which leads to the question why not CLOS but Python of all things should be used as a source of "normal sense" of anything Julia-related). In CLOS, classes bind together data, generic functions name abstract operations, and methods represent specific code that deals with implementing a generic function for a particular co…

Well, it seems as if we were just at cross purposes due to terminology.

Although multiple dispatch in CL (and Perl) predates Julia, I was not aware that Julia’s design derived from it. Do you have a reference that traces this?

Re: Test for lists in Cython

#119

Post by remram »

yakubin wrote: post #11
ur-whale wrote: post #3

From the benchmarks: > Rust is not that fast because it needs to copy data; I'm surprised. Don't know much about Rust, but isn't it hailed as being competitive with C/C++ ?

You left out the explanation from your quote. The full is: > Rust is not that fast beacuse it needs to copy data; using Pyo3 objects would probably lead to similar results as cython, but with an added library. "It needs to copy data", because it's converting Python objects into Rust objects and back again. As the full quote states, it could be written in a different way, although that would make the code look really…

What is the "added library" though? PyO3 is already used to wrap, and is in fact the library doing the copy rather than providing a reference?

Re: Test for lists in Cython

#120

Post by boothby »

syntonym2 wrote: post #65

I use python in a scientific context, but have so far not written much extensions for python in any of the languages tested. I'm interested in some guidance which language a) is easy to integrate with python and b) has some good performance, but this benchmark lacks the details to come to any conclusion. I tried to run the benchmark on my own computer, but the setup documentation was not enough for me to get the juli…

Their .pyx implementation leaves much to be desired. Among other problems, the hot loop uses an untyped python list. Also, they're indexing into the list instead of iterating over the list. In creating the list, they're using .append() instead of list comprehensions.

Fixing those minor issues cuts the runtime in half (see #3). Going for actual high-performance Cython (on my 12-core workstation) cuts the runtime by 20x (see #2).

https://github.com/00sapo/cython_list_test/pull/2 https://github.com/00sapo/cython_list_test/pull/3

Post reply on HN