Live data from Hacker News

Speed up your Python using Rust

developers.redhat.com

41–50 of 102 posts

Re: Speed up your Python using Rust

#41
post #26

For comparison, I just implemented the same as C SWIG extension[1]. It's about 10% faster, but it's cheating by comparing bytes instead of utf-8 encoded characters. The more interesting part to me is the comparison of the amount of boilerplate code required. https://github.com/martinxyz/rust-python-example/commit/f8e3...

One thing though that gets very complicated about using SWIG is ownership semantics. With anything more complicated than passing scalar values, it is very easy to introduce a memory leak or double-free if you don't get the flags right. I wonder if Rust types naturally allow a much better inference of ownership semantics across the language boundary?

Re: Speed up your Python using Rust

#42
post #28

About as fast as numpy.. More tools to create fast code is always great, but the tooling for Rust/C in Python needs to be easier, I just can't be bothered most of the time. This in numpy gets a better relative boost on my machine YMMV. import numpy def count_double_chars_np(val): ng=np.fromstring(val,dtype=np.byte) return np.sum(ng[:-1]==ng[1:]) def test_np(benchmark): benchmark(count_double_chars_np, val)

Hi, can you send a Pull Request including your numpy implementation? https://github.com/rochacbruno/rust-python-example I would like to add it there just for the record and then I will update the article.

Re: Speed up your Python using Rust

#43
post #13

I like the article, but the following advice confused me, especially since this comes from RedHat i.e. Linux people: > Having Rust installed (recommended way is https://www.rustup.rs/ ). This essentially recommends unconditionally using the "curl | sh" anti-pattern. Shouldn't they recommend instead e.g. "apt-get install rustc" for Debian users? Since this doesn't make use of too recent Rust features, using Rust 1.14…

I edited the article including a reference to rust-toolset which is available on RHEL repositories `yum install rust-toolset-7` https://developers.redhat.com/blog/2017/11/01/getting-starte...

Re: Speed up your Python using Rust

#45
post #26

For comparison, I just implemented the same as C SWIG extension[1]. It's about 10% faster, but it's cheating by comparing bytes instead of utf-8 encoded characters. The more interesting part to me is the comparison of the amount of boilerplate code required. https://github.com/martinxyz/rust-python-example/commit/f8e3...

> It's about 10% faster, but it's cheating by comparing bytes instead of utf-8 encoded characters.

I'm really glad that you acknowledged this — I work with a lot of non-ASCII text and have run into that more than a few times in real code.

Re: Speed up your Python using Rust

#46
post #26

For comparison, I just implemented the same as C SWIG extension[1]. It's about 10% faster, but it's cheating by comparing bytes instead of utf-8 encoded characters. The more interesting part to me is the comparison of the amount of boilerplate code required. https://github.com/martinxyz/rust-python-example/commit/f8e3...

One thing though that gets very complicated about using SWIG is ownership semantics. With anything more complicated than passing scalar values, it is very easy to introduce a memory leak or double-free if you don't get the flags right. I wonder if Rust types naturally allow a much better inference of ownership semantics across the language boundary?

If you try to wrap any non-trivial type using SWIG typemaps you will quickly go insane. However to speed up an inner loop you can often get away with a few PyObject* arguments/returns. SWIG will pass those through and you can use the Python/C API directly, e.g. to return a numpy array. Allow SWIG to handle only simple types. The Python/C API is relatively sane, but you'll have to learn the reference counting conventions.

Re: Speed up your Python using Rust

#47
post #34

> Rust is a language that, because it has no runtime, can be used to integrate with any runtime; you can write a native extension in Rust that is called by a program node.js, or by a python program, or by a program in ruby, lua etc. and, however, you can script a program in Rust using these languages. — “Elias Gabriel Amaral da Silva” Can someone explain why is "having a runtime" problematic for writing extensions an…

"Can someone explain why is "having a runtime" problematic for writing extensions and calling them from Python ?"

Perhaps instead of saying "having a runtime" it would be better to examine the situation in terms of what the code assumes. Python assumes that it has the Python GC running on its code, that everything is a PyObject of one sort or another, that it has a Global Interpreter Lock that if taken will prevent anything from modifying anything it thinks it owns, and so on. Go assumes that it has the Go GC running (despite both "having GC", there's enough differences that it must be specified as a difference), that its objects are laid out in certain manners such that most field references are compiled down to static offsets rather than dynamic lookups, that it can run its core event loop and dispatch out work to its internal goroutines without asking anyone else, etc.

You could go on for quite a while; I don't intend those as complete lists. I just want to convey the flavor of conceptualizing the runtime in terms of assumptions that the code running in that runtime can make.

Once you look at it this way, it should be more clear why trying to jam two runtimes into one OS process gets to be tricky. I use the word "jam" quite carefully, because it always feels that way to me. The more differences between the assumptions of the two runtimes, the more translation the code is going to need. For instance, Python to anything else is going to involve unwrapping the data from the internal PyObject wrappers, and wrapping anything coming back from somewhere else back into PyObjects. Threading models have to be matched up. Memory layout has to be harmonized. Memory generally has to be kept strictly separated, because the two runtimes both expect to be able to manage memory, so you can't hand memory allocated by one of them to the other, which further implies that you're almost certainly copying everything across the boundary. Etc. etc.

I'd also separate out the way there can be differences in the affordances of the languages. For instance, Python doesn't have what Rust or Go would call "arrays". Rust and Go are fine with getting arrays of pointers, but the languages afford the use of memory-contiguous arrays without pointers, so especially if you're integrating with a third-party library, you have no choice but for some layer somewhere along the way to convert Python lists into the correct sort of array. The runtimes technically don't force this, but the structure of the libraries and code afforded by the other languages do. By contrast, if you were integrating with lisp, you might find many points where you need to turn things into singly-linked lists, again, not because Lisp can't handle arrays, but because you're likely to encounter pre-existing Lisp code that expects Lisp cons lists.

As another example, despite the fact Go and C generally see eye-to-eye on how to layout structs, the C support from Go is still extremely expensive due to the need to convert from how Go sees the concurrency world to how C sees the world. C, contrary to popular belief, actually does have a runtime, and that runtime tends to assume it has very deep control of the OS process it is running in. Go has to do a lot of work to isolate the running C code in an environment it is comfortable with, where it won't be pre-empted by the green thread code (on account of the fact that it can't be, C doesn't support that). There's also some tricksy code you may need to write to harmonize C's memory-management-via-malloc model with Go's "lifetimes determined via the GC" model. (If you listen carefully, you can hear the Go runtime go "klunk" every time it runs cgo code.)

Rust has a runtime too, but unlike a lot of languages, it has the ability to shut it off. You lose some services and capabilities, but on the upside, you significantly reduce the number of assumptions the Rust code is making, making it easier to integrate with other runtimes. (I say reduce because technically, it still doesn't make it to zero if you are precise enough in your thinking, but I'd expect that of all the current "cool" languages, Rust with the runtime off probably makes fewer assumptions than anything else.) That said, I'm not sure if this code is working in that mode. I see the rust code doesn't directly turn off the runtime, but I don't know what that "#[macro_use] extern crate cpython;" line fully expands to. It's possible that the full Rust runtime is still in play, which looks enough like C anyhow (by explicit design of the Rust team) that Python's existing C integration can just be reused. Either way Rust is still making many fewer assumptions that Go's relatively heavyweight (in terms of assumptions moreso than resources) runtime.

Re: Speed up your Python using Rust

#48
post #44
post #8

Earlier quoted context omitted.

Cython is unsafe.

Rust is disgusting language with too much commercial exploitation

Rust would have to have a significant and meaningful presence in the commercial space for this exploitation you mention to exist. It doesn't. Rust adoption right now is trivial and tiny compared to commercial programming at large.

Rust is a great language with some over-zealous fans who seem to not understand that frothing at the mouth about how C and C++ are literally weapons of murder and how Rust-safety is the most important thing any programmer should think about for any project is...not the best way to advocate for the language.

Re: Speed up your Python using Rust

#49
post #25
post #20

Earlier quoted context omitted.

Rust is a language in active development, continually getting improvements and new features. Using Rustup is the best way of managing up to date toolchains (and multiple toolchain versions if you have to). It's no harder than apt-get install, and sets the best practice early on so that someone doesn't get confused and have to switch later. "curl | sh" is only an anti-pattern in the sense that you have to trust the so…

I agree with your main point, but would like to add that apt does have additional signature verification with gpg, so it's a bit more secure than just https (e.g. anyone with access to a trusted CA and your network can mount an active attack against you).

HTTPS also doesn't guard you against someone replacing the binaries on the server (e.g. what happened to transmission). It also doesn't protect you from misconfigured corporate or state level MITM firewalls that don't check certificate validity.

HTTPS is intended for transport security. Using it for package authentication is generally a mistake. That's why most distributions accept the additional complexity of PGP instead of only relying on HTTPS.

Re: Speed up your Python using Rust

#50
post #13

I like the article, but the following advice confused me, especially since this comes from RedHat i.e. Linux people: > Having Rust installed (recommended way is https://www.rustup.rs/ ). This essentially recommends unconditionally using the "curl | sh" anti-pattern. Shouldn't they recommend instead e.g. "apt-get install rustc" for Debian users? Since this doesn't make use of too recent Rust features, using Rust 1.14…

You can download the rustup bootstrapper if you don’t like curl to bash. I would recommend against using the debian packages.

Debian packages integrate better with the system, are authenticated by gpg, and have at least one more critical pair of eyes on them.

If you're willing to stick with an older version of cargo and rustc, why not?

Post reply on HN