Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pythran as a bridge between fast prototyping and code deployment

serge-sans-paille.github.io

Post reply on HN
Print view View original

11–20 of 27 posts

Re: Pythran as a bridge between fast prototyping and code deployment

#11

Post by keldaris »

tomp wrote: post #7
rademacher wrote: post #2

Looks pretty nice. Unfortunately, I never run into this issue because I only get to write research code. The Julia language was designed to target the two language problem and at least from these benchmarks it looks pretty competitive [1]. I imagine over time, pythran may fix some limitations and beat Julia in most benchmarks. [1] https://github.com/fluiddyn/BenchmarksPythonJuliaAndCo/tree/...

Note that the blog post is also about deployment , not just about performance. Does Julia support statically compiled executables without dependencies or GC?

Unfortunately, not really. There is some community work in that regard [1], but it doesn't seem to get as much attention as one would like. Some people have gotten it to work, but official support (guaranteeing maintenance and decent documentation) for static compilation and easy deployment would make a huge difference.

[1] https://github.com/JuliaLang/PackageCompiler.jl

Re: Pythran as a bridge between fast prototyping and code deployment

#12

Post by serge-ss-paille »

ktta wrote: post #10
ktta wrote: post #9

Was Cython given a consideration for this project? I see that you are involved with the Pythran project, so could you tell us the shortcomings of Cython? As I understand it, before Pythran didn't support Python 3, but seems like that has changed

Can't edit on my app, but the first question was for the blog post writer and my second to the submitter

> could you tell us the shortcomings of Cython

In order to achieve top performance, in the context of numerical simulations, you generally end up explicity writing the loops are implicit in high-level numpy (less abstraction).

Cython does not perform any high-level optimisation on the code, while Pythran does. For instance Pytrhan computes whether an array index may be negative or not, and generates wraparound only when needed. On the otherhand Cython requires a compiler directive to do so.

That being said, Cython can do plenty of stuff Pythran cannot: import native libraries, wrap classes, mixed Python/native mode etc. It has a much stronger codebase (more tested/validated) and a larger community.

Re: Pythran as a bridge between fast prototyping and code deployment

#13

Post by cultus »

svantana wrote: post #6
rademacher wrote: post #4

Earlier quoted context omitted.

I used to only use MATLAB, which for a lot of research code applications is actually nice for getting a prototype running quickly. Now that I have the freedom to choose, I typically use Julia as I'm trying to gain skills in opensource languages that are actually valuable in the job market. The choice of Julia over python is probably due to my nature of going against the grain, which flies in the face of my previous p…

One major issue with Julia is that it only recently reached 1.0, with a lot of breaking changes, that make a lot of libraries incompatible. Another issue is that it's not always that fast, for a recent project I never managed to exceed 100 MFLOPS, at which point I switched to C++ and got 3 GFLOPS. But the python version stalled out at 4 MFLOPS though...

Did you write type-unstable code? That brings Julia performance down and memory usage up, since things often get inferred to be Any. So, a vector of what you think are doubles could be turned into a boxed vector of Any. It will slow things down to the speed of Python. Fortunately, it's usually pretty easy to avoid this if you are aware of it.

Well-written Julia should always be within a factor of 2-3 of C, often less. Huge problems are done in pure Julia now. Pure Julia code has been run on HPCs to over a petaflop, something that only C/C++ and Fortran have done. 100 MFLOP is not a problem.

Re: Pythran as a bridge between fast prototyping and code deployment

#14

Post by [deleted] »

tomp wrote: post #7
rademacher wrote: post #2

Looks pretty nice. Unfortunately, I never run into this issue because I only get to write research code. The Julia language was designed to target the two language problem and at least from these benchmarks it looks pretty competitive [1]. I imagine over time, pythran may fix some limitations and beat Julia in most benchmarks. [1] https://github.com/fluiddyn/BenchmarksPythonJuliaAndCo/tree/...

Note that the blog post is also about deployment , not just about performance. Does Julia support statically compiled executables without dependencies or GC?

[deleted]

Re: Pythran as a bridge between fast prototyping and code deployment

#15

Post by rademacher »

cultus wrote: post #13
svantana wrote: post #6

Earlier quoted context omitted.

One major issue with Julia is that it only recently reached 1.0, with a lot of breaking changes, that make a lot of libraries incompatible. Another issue is that it's not always that fast, for a recent project I never managed to exceed 100 MFLOPS, at which point I switched to C++ and got 3 GFLOPS. But the python version stalled out at 4 MFLOPS though...

Did you write type-unstable code? That brings Julia performance down and memory usage up, since things often get inferred to be Any. So, a vector of what you think are doubles could be turned into a boxed vector of Any. It will slow things down to the speed of Python. Fortunately, it's usually pretty easy to avoid this if you are aware of it. Well-written Julia should always be within a factor of 2-3 of C, often less…

Here is a reference for the comment above and a brief excerpt,

"Written in the productivity language Julia, the Celeste project—which aims to catalogue all of the telescope data for the stars and galaxies in in the visible universe—demonstrated the first Julia application to exceed 1 PF/s of double-precision floating-point performance (specifically 1.54 PF/s)." [1]

[1] https://www.nextplatform.com/2017/11/28/julia-language-deliv...

Re: Pythran as a bridge between fast prototyping and code deployment

#17

Post by byt143 »

tomp wrote: post #7
rademacher wrote: post #2

Looks pretty nice. Unfortunately, I never run into this issue because I only get to write research code. The Julia language was designed to target the two language problem and at least from these benchmarks it looks pretty competitive [1]. I imagine over time, pythran may fix some limitations and beat Julia in most benchmarks. [1] https://github.com/fluiddyn/BenchmarksPythonJuliaAndCo/tree/...

Note that the blog post is also about deployment , not just about performance. Does Julia support statically compiled executables without dependencies or GC?

Better support for static compilation Is on the roadmap

Re: Pythran as a bridge between fast prototyping and code deployment

#18

Post by kristofferc »

I feel that a comparison to the handwritten C++ version would make the claims a lot stronger. Making something 10x faster is not very hard if it is incredibly slow to begin with and is, on its own, fairly uninteresting. On the other hand, if the results here approcahed the speed of optimized C++ code, then this workflow makes a lot of sense.

Re: Pythran as a bridge between fast prototyping and code deployment

#19

Post by jeanl »

serge-ss-paille wrote: post #12
ktta wrote: post #10

Earlier quoted context omitted.

Can't edit on my app, but the first question was for the blog post writer and my second to the submitter

> could you tell us the shortcomings of Cython In order to achieve top performance, in the context of numerical simulations, you generally end up explicity writing the loops are implicit in high-level numpy (less abstraction). Cython does not perform any high-level optimisation on the code, while Pythran does. For instance Pytrhan computes whether an array index may be negative or not, and generates wraparound only w…

For me, the biggest shortcoming: Cython does not create independent C++ code (independent of the python interpreter that is) that can be used in a separate C++ code base. My main point is that pythran makes it possible to deploy python/numpy code as C++ code.

Re: Pythran as a bridge between fast prototyping and code deployment

#20

Post by jeanl »

kristofferc wrote: post #18

I feel that a comparison to the handwritten C++ version would make the claims a lot stronger. Making something 10x faster is not very hard if it is incredibly slow to begin with and is, on its own, fairly uninteresting. On the other hand, if the results here approcahed the speed of optimized C++ code, then this workflow makes a lot of sense.

That's a fair point and I have to admit I didn't have the time/courage to port the entire algo to C++. But I can see why that would make the results a lot more convincing.
Post reply on HN