Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Speed up your Python using Rust

developers.redhat.com

Post reply on HN
Print view View original

91–100 of 102 posts

Re: Speed up your Python using Rust

#91

Post by pc2g4d »

Dowwie wrote: post #68

I never got around to talking about it, but as part of my "month of Rust", I ported permission-based authorization logic from Python to Rust and then ran performance benchmarks of the Rust implementation and a pypy-compiled version. The pypy-compiled python ran slightly faster. I've been told not to expect similar results in other implementations. These findings cannot be used to draw any conclusions about pypy. My r…

Very interesting. I glanced at the code to see if there were obvious performance issues. All I noticed was a triple `map` invocation in https://github.com/YosaiProject/yosai_libauthz/blob/master/s...

Not sure what the compiler does with that, but I'd expect that it means you're running through those elements three times. I imagine it would be better to reduce this to one `map` call.

Re: Speed up your Python using Rust

#92

Post by Dowwie »

pc2g4d wrote: post #91
Dowwie wrote: post #68

I never got around to talking about it, but as part of my "month of Rust", I ported permission-based authorization logic from Python to Rust and then ran performance benchmarks of the Rust implementation and a pypy-compiled version. The pypy-compiled python ran slightly faster. I've been told not to expect similar results in other implementations. These findings cannot be used to draw any conclusions about pypy. My r…

Very interesting. I glanced at the code to see if there were obvious performance issues. All I noticed was a triple `map` invocation in https://github.com/YosaiProject/yosai_libauthz/blob/master/s... Not sure what the compiler does with that, but I'd expect that it means you're running through those elements three times. I imagine it would be better to reduce this to one `map` call.

ah you're looking at the CABI stuff.. that part wasn't benchmarked because I wanted as close to apples to apples as I could get and know that the cffi bridge taxes performance

here's the actual rust library: https://github.com/Dowwie/rust-authz/blob/master/src/authz.r...

the project includes a bench

Re: Speed up your Python using Rust

#93

Post by fnl »

rochacbruno wrote: post #42
emj wrote: post #28

About as fast as numpy.. More tools to create fast code is always great, but the tooling for Rust/C in Python needs to be easier, I just can't be bothered most of the time. This in numpy gets a better relative boost on my machine YMMV. import numpy def count_double_chars_np(val): ng=np.fromstring(val,dtype=np.byte) return np.sum(ng[:-1]==ng[1:]) def test_np(benchmark): benchmark(count_double_chars_np, val)

Hi, can you send a Pull Request including your numpy implementation? https://github.com/rochacbruno/rust-python-example I would like to add it there just for the record and then I will update the article.

Thank you for the very nice, educative article, Bruno!

If performance comparison of counting character pairs really were the issue here, in addition to the already suggested numpy approach, an implementation I'd dare wager to be as competitive is re2, e.g. [1], a drop-in replacement for the standard re package.

But I want to point out that I think all this performance comparison of this trivial character counting distracts from the core idea here: You'd use a low-level implementation in Rust (or C/C++/Cython, for that matter) when such "nifty tricks" are not available, after all. So again thanks for the article, and do think if you really want this performance issues to degrade the article to a only marginally relevant performance "showdown".

https://pypi.python.org/pypi/re2/

Re: Speed up your Python using Rust

#94

Post by jerf »

steveklabnik wrote: post #78
jerf wrote: post #73

Earlier quoted context omitted.

"could you please explain what you mean by C runtime," There's two components to the C runtime, what is specified by the C standard, and what is specified by POSIX and the operating systems. I am not sufficiently familiar with the C world to tell you exactly which thing is defined in which part. Fortunately, for this discussion of how integrating C code into another runtime goes, it doesn't really matter. The C runti…

A great answer! > I am not sufficiently familiar with the C world to tell you exactly which thing is defined in which part. I've got some bits of knowledge here. I could be wrong, as it's not my expertise... > Function calls have a "stack" and there's a "heap", and the language itself distinguishes between them. I don't believe this is true or at least, not literally but the details are interesting! http://www.open-s…

Thank you for the elaboration. Now that you remind me, I remember about the C11. Which also adds "The C memory model" as part of the runtime, IIRC. Other languages have different memory models. Usually simpler, though it's hard to hold that against C11 since it was in the unenviable position of trying to codify decades of implicit and divergent practice in one of the trickiest places in software engineering.

Re: Speed up your Python using Rust

#95

Post by fnl »

stochastic_monk wrote: post #76
Matumio wrote: post #26

For comparison, I just implemented the same as C SWIG extension[1]. It's about 10% faster, but it's cheating by comparing bytes instead of utf-8 encoded characters. The more interesting part to me is the comparison of the amount of boilerplate code required. https://github.com/martinxyz/rust-python-example/commit/f8e3...

I find pybind11 [1] to be perfect for my C++ code. There's so little boilerplate, and I get RAII-guaranteed memory safety and all the speed my C++ development can bring. For example, the binding of an accelerated HyperLogLog implementation only requires tiny amount of work, plus a line in my Makefile: PYBIND11_MODULE(_hll, m) { m.doc() = "pybind11-powered HyperLogLog"; // optional module docstring py::class_ (m, "hll…

If working in C++ land, I'd agree this is the nicest approach. It does, however, require linking against a specific libpython version [1], unlike Milksnake. But I'm not sure that's a bad thing...

[1] http://pybind11.readthedocs.io/en/master/basics.html#creatin...

Re: Speed up your Python using Rust

#96

Post by the_mitsuhiko »

tdbgamer wrote: post #82
the_mitsuhiko wrote: post #18

Earlier quoted context omitted.

But the tooling is terrible in comparison. I find rust sigificantly easier as a python developer than cython. There is so much more rust ecosystem to take advantage of.

To be fair to Cython, they have access to the entire C++ stdlib, so that's a fairly good amount of tooling. The main thing is lacks is good documentation and memory safety.

And C++ has absolutely no package distribution system at this point.

Re: Speed up your Python using Rust

#98

Post by stochastic_monk »

fnl wrote: post #95
stochastic_monk wrote: post #76

Earlier quoted context omitted.

I find pybind11 [1] to be perfect for my C++ code. There's so little boilerplate, and I get RAII-guaranteed memory safety and all the speed my C++ development can bring. For example, the binding of an accelerated HyperLogLog implementation only requires tiny amount of work, plus a line in my Makefile: PYBIND11_MODULE(_hll, m) { m.doc() = "pybind11-powered HyperLogLog"; // optional module docstring py::class_ (m, "hll…

If working in C++ land, I'd agree this is the nicest approach. It does, however, require linking against a specific libpython version [1], unlike Milksnake. But I'm not sure that's a bad thing... [1] http://pybind11.readthedocs.io/en/master/basics.html#creatin...

True. It's not that bad, though -- "python3-config --extension-suffix" or "python-config --extension-suffix" is all it takes to generate the suffix you want, and you can drop it straight in your site-packages folder. At that point, you're just dropping the 3 or not depending on your version.

It's not as simple as milksnake. I would like to see some smarter extensions added to pybind11, but I'm okay with that for now.

Re: Speed up your Python using Rust

#99

Post by sjellis »

vog wrote: post #13

I like the article, but the following advice confused me, especially since this comes from RedHat i.e. Linux people: > Having Rust installed (recommended way is https://www.rustup.rs/ ). This essentially recommends unconditionally using the "curl | sh" anti-pattern. Shouldn't they recommend instead e.g. "apt-get install rustc" for Debian users? Since this doesn't make use of too recent Rust features, using Rust 1.14…

Packaging Rust for Fedora/RH distributions is a work in progress ATM:

https://fedoraproject.org/wiki/Changes/Packaging_Rust_applic...

Re: Speed up your Python using Rust

#100

Post by jD91mZM2 »

blub wrote: post #72
gpm wrote: post #71

Earlier quoted context omitted.

You're planning on downloading binaries and gpg keys from that site anyways. Either you trust it, in which case you might as well curl | bash, or you don't, in which case you shouldn't be running that script no matter how carefully you inspect it. And of course you can inspect the bash script (not that it does you any good), curl > file; bash file. It's just that most people don't so that's not what is recommended .

No, you're completely wrong. The key is available in multiple places and has been available for a while, so there is some verification that can be done. The binary will be checked by gpg, it shouldn't matter where it's from. Finally, if the recommmendation is to run curl foo | sh, the bash script can literally not be inspected.

Just separate the steps? Curl to a file, inspect it, and then execute it? I don't see the problem. Most users just don't care because it's official anyways.
Post reply on HN