Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Speed up your Python using Rust

developers.redhat.com

Post reply on HN
Print view View original

71–80 of 102 posts

Re: Speed up your Python using Rust

#71

Post by gpm »

blub wrote: post #69
simias wrote: post #19

Earlier quoted context omitted.

If you look at the way the rustup-init.sh script is written it's safe to be used with this "anti pattern". I see your objection though but unfortunately this ship has sailed, you might as well complain about websites that don't work without Javascript... The advantage of this method is that it will work on any linux distro (and even BSD, Darwin and mingw) and you'll get the latest stable version. I don't see the adva…

That makes no sense. If one curls to bash obviously they can't "look at the way the rustup-init.sh script is written". The ship hasn't sailed and neither has the sites without JS one. You have made a decision to favor covenience instead of security and you're trying to make it look like it's the normal state of affairs. Rust can be installed by downloading the appropriate package and checking it with gpg. Recommendin…

You're planning on downloading binaries and gpg keys from that site anyways. Either you trust it, in which case you might as well curl | bash, or you don't, in which case you shouldn't be running that script no matter how carefully you inspect it.

And of course you can inspect the bash script (not that it does you any good), curl > file; bash file. It's just that most people don't so that's not what is recommended .

Re: Speed up your Python using Rust

#72

Post by blub »

gpm wrote: post #71
blub wrote: post #69

Earlier quoted context omitted.

That makes no sense. If one curls to bash obviously they can't "look at the way the rustup-init.sh script is written". The ship hasn't sailed and neither has the sites without JS one. You have made a decision to favor covenience instead of security and you're trying to make it look like it's the normal state of affairs. Rust can be installed by downloading the appropriate package and checking it with gpg. Recommendin…

You're planning on downloading binaries and gpg keys from that site anyways. Either you trust it, in which case you might as well curl | bash, or you don't, in which case you shouldn't be running that script no matter how carefully you inspect it. And of course you can inspect the bash script (not that it does you any good), curl > file; bash file. It's just that most people don't so that's not what is recommended .

No, you're completely wrong. The key is available in multiple places and has been available for a while, so there is some verification that can be done.

The binary will be checked by gpg, it shouldn't matter where it's from.

Finally, if the recommmendation is to run curl foo | sh, the bash script can literally not be inspected.

Re: Speed up your Python using Rust

#73

Post by jerf »

TuringTest wrote: post #62
jerf wrote: post #47

Earlier quoted context omitted.

"Can someone explain why is "having a runtime" problematic for writing extensions and calling them from Python ?" Perhaps instead of saying "having a runtime" it would be better to examine the situation in terms of what the code assumes. Python assumes that it has the Python GC running on its code, that everything is a PyObject of one sort or another, that it has a Global Interpreter Lock that if taken will prevent a…

> C, contrary to popular belief, actually does have a runtime I've been left wondering what you meant by this. Are you referring to the stack and heap management? Or OS processes and threads? If not, could you please explain what you mean by C runtime, and how does Rust differs from it when it is shut down??

"could you please explain what you mean by C runtime,"

There's two components to the C runtime, what is specified by the C standard, and what is specified by POSIX and the operating systems. I am not sufficiently familiar with the C world to tell you exactly which thing is defined in which part. Fortunately, for this discussion of how integrating C code into another runtime goes, it doesn't really matter.

The C runtime includes the assumption that there is a malloc-compatible memory allocator available (note it's swappable), the process of linking programs when they start up and the whole surrounding "symbols" they can obtain. It has certain assumptions about what state needs to be saved when a function is called; for instance, it won't save the flags on the processor controlling IEEE FPU conformity. Function calls have a "stack" and there's a "heap", and the language itself distinguishes between them. C itself, IIRC, has no specification for threads whatsoever, but the OSes seem to have converged on a fairly similar model that could be fairly called part of the runtime now.

It's hard to "see" the C runtime because it has won so thoroughly that it just looks like "how computation is done", or is so deeply integrated into the operating system that it forces parts of the model on everything that runs on that OS. You kind of have to piece together what C does by looking at what it does that other languages do differently. Yes, most programs at some point will do some linking and symbol resolution, but once the interpreter has started up, dynamic languages have no concept of a static symbol table. Loading another Python module doesn't even remotely resemble loading a C library, either at startup or dynamically later. The language Go doesn't have a stack or a heap. The implementation does for practical reasons, but the language does not. Most other languages now will save the same things on the call stack as C, but that's not a requirement of computation; you could save a lot more of the processor's state, but it'll trash your function performance to do it. A "stack" and "heap" model is not necessary; Haskell for instance does not have a clear "stack" at all. (It does stack-like things, certainly, but it turns out getting what most people call "a stacktrace" from the runtime is actually fairly hard. I believe still not possible on GHC.) There are alternate methods for threading, including models that still use the C-style threads under the hood but include mandatory code to be run at startup and shutdown to be "part" of the runtime.

C is not as thin as it looks; it's just that history has made it appear to be the baseline. And as I know my internets, let me say that nothing in this post is criticism. Something has to be the baseline. While I think the C baseline is getting long in the tooth, it won for a reason, and I don't know that we could have gotten much better from the 1970s. (The other competition usually cited was either a performance non-starter (the Lisp of the time), or had it survived for 40+ years, we'd be able to write a very similar post about how it is getting long in the tooth too in 2017 (Pascal, for instance).)

Re: Speed up your Python using Rust

#74

Post by staticassertion »

Dowwie wrote: post #68

I never got around to talking about it, but as part of my "month of Rust", I ported permission-based authorization logic from Python to Rust and then ran performance benchmarks of the Rust implementation and a pypy-compiled version. The pypy-compiled python ran slightly faster. I've been told not to expect similar results in other implementations. These findings cannot be used to draw any conclusions about pypy. My r…

That's pretty interesting - JIT's can do a lot of great optimizations with runtime information, but I'm still surprised to hear that Pypy was faster. It would be cool to see the benchmarks, methodology, and Python code.

Personally, Pypy has never been an option due to the nature of the codebases I work on - or at least it wasn't. I was using pandas, numpy, scipy etc and I don't think it was compatible.

Re: Speed up your Python using Rust

#75

Post by k__ »

Dowwie wrote: post #68

I never got around to talking about it, but as part of my "month of Rust", I ported permission-based authorization logic from Python to Rust and then ran performance benchmarks of the Rust implementation and a pypy-compiled version. The pypy-compiled python ran slightly faster. I've been told not to expect similar results in other implementations. These findings cannot be used to draw any conclusions about pypy. My r…

I don't understand all the implications, but I often hear from JIT language people that their language could be as fast as a AOT language if it was used correctly.

What are the use-cases that lend itself to be faster in Rust, than JS/Ruby/Phython?

Re: Speed up your Python using Rust

#76

Post by stochastic_monk »

Matumio wrote: post #26

For comparison, I just implemented the same as C SWIG extension[1]. It's about 10% faster, but it's cheating by comparing bytes instead of utf-8 encoded characters. The more interesting part to me is the comparison of the amount of boilerplate code required. https://github.com/martinxyz/rust-python-example/commit/f8e3...

I find pybind11 [1] to be perfect for my C++ code. There's so little boilerplate, and I get RAII-guaranteed memory safety and all the speed my C++ development can bring.

For example, the binding of an accelerated HyperLogLog implementation only requires tiny amount of work, plus a line in my Makefile:

  PYBIND11_MODULE(_hll, m) {
      m.doc() = "pybind11-powered HyperLogLog"; // optional module docstring
      py::class_ (m, "hll")
          .def(py::init())
          .def("clear", &hll_t::clear, "Clear all entries.")
          .def("resize", &hll_t::resize, "Change old size to a new size.")
          .def("sum", &hll_t::sum, "Add up results.")
          .def("report", &hll_t::report, "Emit estimated cardinality. Performs sum if not performed, but sum must be recalculated if further entries are added.")
          .def("add", &hll_t::add, "Add a (hashed) value to the sketch.")
          .def("addh_", &hll_t::addh, "Hash an integer value and then add that to the sketch.");
  }
[1] https://github.com/pybind/pybind11

Re: Speed up your Python using Rust

#77

Post by Dowwie »

k__ wrote: post #75
Dowwie wrote: post #68

I never got around to talking about it, but as part of my "month of Rust", I ported permission-based authorization logic from Python to Rust and then ran performance benchmarks of the Rust implementation and a pypy-compiled version. The pypy-compiled python ran slightly faster. I've been told not to expect similar results in other implementations. These findings cannot be used to draw any conclusions about pypy. My r…

I don't understand all the implications, but I often hear from JIT language people that their language could be as fast as a AOT language if it was used correctly. What are the use-cases that lend itself to be faster in Rust, than JS/Ruby/Phython?

I would like to know this as well. This code is one instance of it.

Re: Speed up your Python using Rust

#78

Post by steveklabnik »

jerf wrote: post #73
TuringTest wrote: post #62

Earlier quoted context omitted.

> C, contrary to popular belief, actually does have a runtime I've been left wondering what you meant by this. Are you referring to the stack and heap management? Or OS processes and threads? If not, could you please explain what you mean by C runtime, and how does Rust differs from it when it is shut down??

"could you please explain what you mean by C runtime," There's two components to the C runtime, what is specified by the C standard, and what is specified by POSIX and the operating systems. I am not sufficiently familiar with the C world to tell you exactly which thing is defined in which part. Fortunately, for this discussion of how integrating C code into another runtime goes, it doesn't really matter. The C runti…

A great answer!

> I am not sufficiently familiar with the C world to tell you exactly which thing is defined in which part.

I've got some bits of knowledge here. I could be wrong, as it's not my expertise...

> Function calls have a "stack" and there's a "heap", and the language itself distinguishes between them.

I don't believe this is true or at least, not literally but the details are interesting! http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1548.pdf is what I usually go by when talking about C11. Malloc is defined in 7.22.3.4, and says:

> The malloc function allocates space for an object whose size is specified by size and whose value is indeterminate.

In 7.22.3, the overview for all the memory functions, it says stuff like

> The lifetime of an allocated object extends from the allocation until the deallocation.

which restricts how you can implement it, of course, but it doesn't use the words "heap" and "stack" at all; "stack" is never mentioned in the document. 6.2.4 talks about storage durations, this is usually what we think about when we talk about "stack" and "heap" and such. "stack allocated" is more properly termed "automatic storage duration" and "heap allocation" is "allocated storage duration."

This is a side effect of the fact that C itself is defined in terms of a virtual machine! They call it the "abstract machine".

Anyway, all of this is in service of your point about history and such. Many people just assume all of this is how it has to be, rather than something that came to be thanks to history. It's all very interesting!

> C itself, IIRC, has no specification for threads whatsoever, IIRC

C11 added this, actually, but before that, you're 100% right.

Re: Speed up your Python using Rust

#80

Post by the_mitsuhiko »

gatmne wrote: post #50
the_mitsuhiko wrote: post #21

Earlier quoted context omitted.

You can download the rustup bootstrapper if you don’t like curl to bash. I would recommend against using the debian packages.

Debian packages integrate better with the system, are authenticated by gpg, and have at least one more critical pair of eyes on them. If you're willing to stick with an older version of cargo and rustc, why not?

Because the ecosystem is not. A lot of what’s on crates.io needs the latest and greatest version.
Post reply on HN