Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Speed up your Python using Rust

developers.redhat.com

Post reply on HN
Print view View original

51–60 of 102 posts

Re: Speed up your Python using Rust

#51

Post by radarsat1 »

Matumio wrote: post #46
radarsat1 wrote: post #41

Earlier quoted context omitted.

One thing though that gets very complicated about using SWIG is ownership semantics. With anything more complicated than passing scalar values, it is very easy to introduce a memory leak or double-free if you don't get the flags right. I wonder if Rust types naturally allow a much better inference of ownership semantics across the language boundary?

If you try to wrap any non-trivial type using SWIG typemaps you will quickly go insane. However to speed up an inner loop you can often get away with a few PyObject* arguments/returns. SWIG will pass those through and you can use the Python/C API directly, e.g. to return a numpy array. Allow SWIG to handle only simple types. The Python/C API is relatively sane, but you'll have to learn the reference counting conventi…

Agreed! On my current project (C++) I found that things get extremely complicated with shared_ptrs and directors. I even ended up contributing some solutions to SWIG.

It all appears to be due to a lack of semantics in the C header. SWIG depends on specifying this stuff in the interface file, but I've often wondered if it wouldn't be better to enhance the C-side, either by standard parameter name conventions or by some Doxygen-like standard comments to indicate ownership and other stuff.

SWIG has this nice potential to generate wrappers for (m)any language(s), but in practice as you said it's often just easier to use the Python API directly instead of trying to make it too general. Shame.

Re: Speed up your Python using Rust

#52

Post by conn01 »

sidlls wrote: post #48
conn01 wrote: post #44

Earlier quoted context omitted.

Rust is disgusting language with too much commercial exploitation

Rust would have to have a significant and meaningful presence in the commercial space for this exploitation you mention to exist. It doesn't. Rust adoption right now is trivial and tiny compared to commercial programming at large. Rust is a great language with some over-zealous fans who seem to not understand that frothing at the mouth about how C and C++ are literally weapons of murder and how Rust-safety is the mos…

Talk is cheap.

Re: Speed up your Python using Rust

#53

Post by makapuf »

Matumio wrote: post #26

For comparison, I just implemented the same as C SWIG extension[1]. It's about 10% faster, but it's cheating by comparing bytes instead of utf-8 encoded characters. The more interesting part to me is the comparison of the amount of boilerplate code required. https://github.com/martinxyz/rust-python-example/commit/f8e3...

Another possibility for this kind of examples that I used and has the minimum boiler pate is to just generate a .so file with your native function in C and call it from ctypes. Basically almost no boilerplate at all.

Re: Speed up your Python using Rust

#54

Post by emj »

rochacbruno wrote: post #42
emj wrote: post #28

About as fast as numpy.. More tools to create fast code is always great, but the tooling for Rust/C in Python needs to be easier, I just can't be bothered most of the time. This in numpy gets a better relative boost on my machine YMMV. import numpy def count_double_chars_np(val): ng=np.fromstring(val,dtype=np.byte) return np.sum(ng[:-1]==ng[1:]) def test_np(benchmark): benchmark(count_double_chars_np, val)

Hi, can you send a Pull Request including your numpy implementation? https://github.com/rochacbruno/rust-python-example I would like to add it there just for the record and then I will update the article.

Thanks would be nice, you should be able to just copy and paste that oneliner, but I'm not sure you blog post is better for it. The idea that Rust is easier to include in Python is important enough, and Numpy is a bit of an edge case imho.

Ideas are always CC-zero

Re: Speed up your Python using Rust

#55

Post by steveklabnik »

makapuf wrote: post #53
Matumio wrote: post #26

For comparison, I just implemented the same as C SWIG extension[1]. It's about 10% faster, but it's cheating by comparing bytes instead of utf-8 encoded characters. The more interesting part to me is the comparison of the amount of boilerplate code required. https://github.com/martinxyz/rust-python-example/commit/f8e3...

Another possibility for this kind of examples that I used and has the minimum boiler pate is to just generate a .so file with your native function in C and call it from ctypes. Basically almost no boilerplate at all.

> with your native function in C

Works well this way with Rust too.

Re: Speed up your Python using Rust

#56

Post by js2 »

See also “Fixing Python Performance With Rust” previously discussed here:

https://news.ycombinator.com/item?id=12748020

And “Evolving Our Rust With Milksnake”:

https://news.ycombinator.com/item?id=15697570

Both from Armin Ronacher at Sentry. About Milksnake:

Milksnake helps you compile and ship shared libraries that do not link against libpython either directly or indirectly. This means it generates a very specific type of Python wheel. Since the extension modules do not link against libpython they are completely Python version or implementation independent. The same wheel works for Python 2.7, 3.6 or PyPy. As such if you use milksnake you only need to build one wheel per platform and CPU architecture.

Re: Speed up your Python using Rust

#57

Post by steveklabnik »

jerf wrote: post #47
b0rsuk wrote: post #34

> Rust is a language that, because it has no runtime, can be used to integrate with any runtime; you can write a native extension in Rust that is called by a program node.js, or by a python program, or by a program in ruby, lua etc. and, however, you can script a program in Rust using these languages. — “Elias Gabriel Amaral da Silva” Can someone explain why is "having a runtime" problematic for writing extensions an…

"Can someone explain why is "having a runtime" problematic for writing extensions and calling them from Python ?" Perhaps instead of saying "having a runtime" it would be better to examine the situation in terms of what the code assumes. Python assumes that it has the Python GC running on its code, that everything is a PyObject of one sort or another, that it has a Global Interpreter Lock that if taken will prevent a…

Rust's runtime is basically the same weight as C, that is, crt: https://github.com/rust-lang/rust/blob/master/src/libstd/rt....

What you're talking about is more of dropping the standard library.

Re: Speed up your Python using Rust

#58

Post by zurn »

b0rsuk wrote: post #34

> Rust is a language that, because it has no runtime, can be used to integrate with any runtime; you can write a native extension in Rust that is called by a program node.js, or by a python program, or by a program in ruby, lua etc. and, however, you can script a program in Rust using these languages. — “Elias Gabriel Amaral da Silva” Can someone explain why is "having a runtime" problematic for writing extensions an…

Rust has a big standard library that is linked by default. Using it with no runtime is pretty rough going and is usually done only by people targeting bare metal microcontrollers or OS kernels, because it loses a lot of the power normally available in the language.

Re: Speed up your Python using Rust

#59

Post by steveklabnik »

conn01 wrote: post #44
fulafel wrote: post #8

Earlier quoted context omitted.

Cython is unsafe.

Rust is disgusting language with too much commercial exploitation

> too much commercial exploitation

Could you elaborate here? I'm interested.

I've mostly been saying "Rust has over 100 companies using it in production right now, but like, that's still 100 companies. It's good for where we're at, but it's not a massive number."

Re: Speed up your Python using Rust

#60

Post by steveklabnik »

zurn wrote: post #58
b0rsuk wrote: post #34

> Rust is a language that, because it has no runtime, can be used to integrate with any runtime; you can write a native extension in Rust that is called by a program node.js, or by a python program, or by a program in ruby, lua etc. and, however, you can script a program in Rust using these languages. — “Elias Gabriel Amaral da Silva” Can someone explain why is "having a runtime" problematic for writing extensions an…

Rust has a big standard library that is linked by default. Using it with no runtime is pretty rough going and is usually done only by people targeting bare metal microcontrollers or OS kernels, because it loses a lot of the power normally available in the language.

What power are you missing? I'm usually surprised by how much still exists without libstd. The biggest thing is collections.
Post reply on HN