Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Speed up your Python using Rust

developers.redhat.com

Post reply on HN
Print view View original

31–40 of 102 posts

Re: Speed up your Python using Rust

#31

Post by heavenlyblue »

ogrisel wrote: post #24
coldtea wrote: post #11

Earlier quoted context omitted.

Cython just transpiles to C. It's not "as fast as C", it's as fast as the runtime support structures it uses, and the communication with Python allows.

You can write very efficient Cython code but it's true that in this case, you tend to adopt a lower level code style that is very close to C/C++. Basically, you need to think about the C/C++ code that will be generated by Cython. C/C++ compilers might be able to generate more optimized native code than what rutsc does though. Actually, this is a question: how good is rustc with numerical / math intensive code? For in…

Most of the time when a developer needs loop unrolling - numpy will work best anyway. Why does everyone always start mentioning this fact when performance is mentioned?

For example, in my case I always need high-performance code to work with strings loaded from loads of CSV files. That includes: merging strings, matching them, comparing them. Loop unrolling/SIMD would not really help here, while an ability to write safe, checked code fast - would.

On the other hand I do need the pythonic dynamics, so that's what I stick to.

Re: Speed up your Python using Rust

#32

Post by steveklabnik »

nabla9 wrote: post #30
Matumio wrote: post #26

For comparison, I just implemented the same as C SWIG extension[1]. It's about 10% faster, but it's cheating by comparing bytes instead of utf-8 encoded characters. The more interesting part to me is the comparison of the amount of boilerplate code required. https://github.com/martinxyz/rust-python-example/commit/f8e3...

I'm not familiar with Rust libraries, but I would guess it's just counting code points and not characters, so strictly speaking both are cheating. I would love to see people showing how to do simple string processing, like counting characters in proper grapheme cluster level in their favorite programming language.

   for (c1, c2) in val.chars().zip(val.chars().skip(1))
chars() iterates by unicode scalar values. It'd be bytes() for bytes.

If you wanted to do it by grapheme clusters, you'd add https://crates.io/crates/unicode-segmentation to your Cargo.toml, add the relevant imports you see on that page to your code, and change the above line to

   for (c1, c2) in UnicodeSegmentation::graphemes(val, true).zip(UnicodeSegmentation::graphemes(val, true).skip(1))
... possibly splitting that up into variables becuase dang, that's a long line.

Then, you're getting &strs instead of chars for the iteration, but I think the body still says the same, as == checks by value.

Re: Speed up your Python using Rust

#33

Post by steveklabnik »

cesarb wrote: post #23
vog wrote: post #13

I like the article, but the following advice confused me, especially since this comes from RedHat i.e. Linux people: > Having Rust installed (recommended way is https://www.rustup.rs/ ). This essentially recommends unconditionally using the "curl | sh" anti-pattern. Shouldn't they recommend instead e.g. "apt-get install rustc" for Debian users? Since this doesn't make use of too recent Rust features, using Rust 1.14…

> using Rust 1.14 of Debian/Stable should be fine, shouldn't it? Same of Fedora, etc. That is a RedHat developer blog. Is rust available on RHEL already? Looking at CentOS (which should have nearly the same packages), it doesn't appear to be available yet.

This stuff is in preview https://developers.redhat.com/blog/2017/10/04/red-hat-adds-g...

Re: Speed up your Python using Rust

#34

Post by b0rsuk »

> Rust is a language that, because it has no runtime, can be used to integrate with any runtime; you can write a native extension in Rust that is called by a program node.js, or by a python program, or by a program in ruby, lua etc. and, however, you can script a program in Rust using these languages. — “Elias Gabriel Amaral da Silva”

Can someone explain why is "having a runtime" problematic for writing extensions and calling them from Python ? From what I gather Go does have a runtime, so implicitly it should be suboptimal for calling from Python. Yet since 2015 (Go 1.5) can be called directly from Python. I'm a Python programmer looking to expand my tool belt. I'm wondering of relative pros and cons of Rust and Go. I have only written small toy programs in C and other compiled languages.

Is Go better suited to completely rewriting software rather than using it for extensions ? Why ?

I would appreciate a benchmark with a Go extension, too.

Re: Speed up your Python using Rust

#35

Post by sigzero »

b0rsuk wrote: post #34

> Rust is a language that, because it has no runtime, can be used to integrate with any runtime; you can write a native extension in Rust that is called by a program node.js, or by a python program, or by a program in ruby, lua etc. and, however, you can script a program in Rust using these languages. — “Elias Gabriel Amaral da Silva” Can someone explain why is "having a runtime" problematic for writing extensions an…

Maybe?

https://matthias-endler.de/2017/go-vs-rust/

Re: Speed up your Python using Rust

#36

Post by somenewacc »

vog wrote: post #13

I like the article, but the following advice confused me, especially since this comes from RedHat i.e. Linux people: > Having Rust installed (recommended way is https://www.rustup.rs/ ). This essentially recommends unconditionally using the "curl | sh" anti-pattern. Shouldn't they recommend instead e.g. "apt-get install rustc" for Debian users? Since this doesn't make use of too recent Rust features, using Rust 1.14…

I think distros package rustup too. Here on Arch it is pacman -S rustup (instead of the curl | sh thing) then proceed normally.

Installing an outdated toolchain makes little sense because after this example people interested in Rust may want to do other things, and will encounter an artificial roadblock when they (or a worse, a dependency) needs a newer Rust.

I think the Rust packaged in the distros is meant to be a build-dependency of software written in Rust (for example, ripgrep), not for Rust developers.

Re: Speed up your Python using Rust

#37

Post by jononor »

vog wrote: post #13

I like the article, but the following advice confused me, especially since this comes from RedHat i.e. Linux people: > Having Rust installed (recommended way is https://www.rustup.rs/ ). This essentially recommends unconditionally using the "curl | sh" anti-pattern. Shouldn't they recommend instead e.g. "apt-get install rustc" for Debian users? Since this doesn't make use of too recent Rust features, using Rust 1.14…

You can install rustup using your package manager and then use it to manage your Rust installs.

Re: Speed up your Python using Rust

#38

Post by forkerenok »

b0rsuk wrote: post #34

> Rust is a language that, because it has no runtime, can be used to integrate with any runtime; you can write a native extension in Rust that is called by a program node.js, or by a python program, or by a program in ruby, lua etc. and, however, you can script a program in Rust using these languages. — “Elias Gabriel Amaral da Silva” Can someone explain why is "having a runtime" problematic for writing extensions an…

If I understand the matter correctly, FFI-ing with a language that has runtime has more friction in extra overhead of initializing runtime, i.e. more state management in your app. Not that it is not doable.

EDIT: maybe indeed someone more knowledgeable will explain it or point to good condensed reads.

Re: Speed up your Python using Rust

#39

Post by dr_zoidberg »

emj wrote: post #28

About as fast as numpy.. More tools to create fast code is always great, but the tooling for Rust/C in Python needs to be easier, I just can't be bothered most of the time. This in numpy gets a better relative boost on my machine YMMV. import numpy def count_double_chars_np(val): ng=np.fromstring(val,dtype=np.byte) return np.sum(ng[:-1]==ng[1:]) def test_np(benchmark): benchmark(count_double_chars_np, val)

Good numpy implementation of the algorithm. If, for whatever reason, numpy isn't available, you can also pull it with a good comprehension:

    def count_doubles2(val):
        return sum(1 for c1, c2 in zip(val, val[1:]) if c1 == c2)
Which will also allow you to avoid a function call entirely, if it was useful in some way:

    In [56]: %timeit count_doubles(val)
    198 ms ± 13.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

    In [57]: %timeit count_doubles2(val)
    189 ms ± 21.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

    In [58]: %timeit sum(1 for c1, c2 in zip(val, val[1:]) if c1 == c2)
    135 ms ± 3.86 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

    In [59]: %timeit count_double_chars_np(val)
    6.95 ms ± 782 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
(Numpy still beats it, for long strings).

Re: Speed up your Python using Rust

#40

Post by cbcoutinho »

somenewacc wrote: post #36
vog wrote: post #13

I like the article, but the following advice confused me, especially since this comes from RedHat i.e. Linux people: > Having Rust installed (recommended way is https://www.rustup.rs/ ). This essentially recommends unconditionally using the "curl | sh" anti-pattern. Shouldn't they recommend instead e.g. "apt-get install rustc" for Debian users? Since this doesn't make use of too recent Rust features, using Rust 1.14…

I think distros package rustup too. Here on Arch it is pacman -S rustup (instead of the curl | sh thing) then proceed normally. Installing an outdated toolchain makes little sense because after this example people interested in Rust may want to do other things, and will encounter an artificial roadblock when they (or a worse, a dependency) needs a newer Rust. I think the Rust packaged in the distros is meant to be a…

This is true for the rust compiler included in the opensuse repository - its only there to build packages (of which the newly released Firefox is and has been since 54)
Post reply on HN