Live data from Hacker News

JNumPy: Writing high-performance C extensions for Python in minutes

github.com

51–60 of 66 posts

Re: JNumPy: Writing high-performance C extensions for Python in minutes

#51
post #28

Coincidentally I used PyJulia last week to call some numerical code I translated from Python/Numpy to Julia. The actual translation was straightforward and done in under one hour, but I needed an additional day for fine tuning and profiling to actually make it fast. Does anyone know, how JNumPy compares to PyJulia? Calling my Julia code via PyJulia seems to increase the startup time of Julia even more and it would be…

Are you aware of the juliacall package? [1]

The announcement post [2] says "Calling Julia from Python, juliacall seems to have smaller overhead (in time) than julia". The benchmarks below that show the overhead on calling `identity` with `juliacall` being less than half of what it is with PyJulia.

[1] https://docs.juliahub.com/PythonCall/WdXsa/0.9.4/juliacall/ [2] https://discourse.julialang.org/t/ann-pythoncall-and-juliaca...

Re: JNumPy: Writing high-performance C extensions for Python in minutes

#52

Earlier quoted context omitted.

Do you have any source on well written julia being any slower than well written C?

I did a comparison of Julia vs numpy, cython and pythran [1] some time ago, for a typical dsp routine we use in our work, and Julia was quite a bit slower than the alternatives. Now I'm by no means a Julia expert so I might have missed an optimisation opportunity (although I posted this and nobody could point to something obvious) , however the whole advertisement behind Julia is that one gets essentially C speed wit…

> the whole advertisement behind Julia is that one gets essentially C speed with the ease of Python. That's actually my main gripe with Julia, the community advertises it as something that it isn't.

I partially agree with this - that statement is true some of the time with some code, but in general there's a lot of nuance to that. (Though, I haven't seen that as a general claim much apart from in the initial "Why we created Julia" post describing what they were hoping to achieve; I more often see specific claims where people were able to port their scientific code into Julia that looks like Python and runs like C. But that may just be us having different bubbles online.)

My partial disagreement is from the fact that oftentimes, the "ease of Python" seems to get interpreted by Python coders as "as easy as Python is to me today" rather than "as easy as it was when I was a beginner" i.e. they expect to be able to port their Python knowledge directly somehow. Python has its own quirks and pitfalls, and over the first few months of people learn to work with/around them and write idiomatic Python code (and then it becomes second nature, and we often even forget that we had to learn to do this). Given a similar on-ramp, people can easily write Julia code that's at least comparable to C in terms of speed, even if it doesn't always match it.

But when you do need to match or exceed C, Julia starts asking for more in-depth knowledge of allocations, types and their performance characteristics, and the code starts deviating away from looking quite as readable as Python. Hopefully, you only need that in some deep internal functions or packed away in a library, but it _is_ a big caveat to that statement.

Re: JNumPy: Writing high-performance C extensions for Python in minutes

#53
post #14

Python itself has to become faster. For the love of god. Make it as fast as PHP. My biggest pain point in computer science is that for every project I have to decide to either cope with PHP's butt-ugly namespace system or with Python's masochistic slowness. Please, Python developers. Take a closer look at how PHP achieves its speed, hold all other development and copy whatever they do! The result would be heaven.

> ... for every project I have to decide to either cope with PHP's butt-ugly namespace system or with Python's masochistic slowness.

Good news: there are more than two programming languages!

But seriously, what situation are you in where your only choices are Python and PHP?

Re: JNumPy: Writing high-performance C extensions for Python in minutes

#54
post #8

Earlier quoted context omitted.

Do you have any source on well written julia being any slower than well written C?

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

It says right there: "Always look at the source code"

fannkuch-redux, n-body, spectral-norm, reverse-complement: the fastest C wins by a large margin because it's an unreadable mess of vector intrinsics. If you really wanted to, you can do that in Julia too (and it might read better). Julia looks pretty even with C in the ones that were faithfully translated.

Re: JNumPy: Writing high-performance C extensions for Python in minutes

#55
> import julia_numpy as np

I don't get it. Does this override python's numpy library with a julia native? If so does it implement all of numpy functionality? Or does this piggyback on normal numpy, redirecting accordingly, but also allows plugging extensions written in julia? The documentation doesn't shed any light at all. Neither do the demos.

Also, is this simply a python julia interface and the "high-performance" part of the description is just part of the usual juliaspeak? Or is there an actual "high-performance" component there that makes this more than a simple python julia interface?

Re: JNumPy: Writing high-performance C extensions for Python in minutes

#56
post #50

Earlier quoted context omitted.

I wish they would stop overselling too. There are so many problems with trying to adopt it for real projects it isn't even funny... Good for researchers sometimes, but even then... It's got issues. Pretty toxic community too.

> There are so many problems with trying to adopt it for real projects it isn't even funny... Good for researchers sometimes, but even then... It's got issues. Can you expand on that, so that it's possible to improve on them?

It seems to use a lot of memory which can be unacceptable for many people and can become impractical if you plan on running several programs simultaneously. For several of the problems at https://benchmarksgame-team.pages.debian.net/benchmarksgame/... on the Computer Language Benchmarks Games, it shows the Julia programs as using several times more memory than similar Java programs. It's even worse when comparing to C, C++, Rust, etc...

Re: JNumPy: Writing high-performance C extensions for Python in minutes

#57
post #56
post #50

Earlier quoted context omitted.

> There are so many problems with trying to adopt it for real projects it isn't even funny... Good for researchers sometimes, but even then... It's got issues. Can you expand on that, so that it's possible to improve on them?

It seems to use a lot of memory which can be unacceptable for many people and can become impractical if you plan on running several programs simultaneously. For several of the problems at https://benchmarksgame-team.pages.debian.net/benchmarksgame/... on the Computer Language Benchmarks Games, it shows the Julia programs as using several times more memory than similar Java programs. It's even worse when comparing to…

Remember that for those benchmarks, C, C++, and Rust compile the binaries and then run the benchmark, while Julia compiles as part of the benchmark. The memory usage is dramatically decreased if you do a workflow similar to the other languages using something like PackageCompiler.jl to build a binary that is benchmarked. If you treat the other languages as "JIT" and include those factors in the benchmarking process, it's a lot closer.

Re: JNumPy: Writing high-performance C extensions for Python in minutes

#58
post #56

Earlier quoted context omitted.

It seems to use a lot of memory which can be unacceptable for many people and can become impractical if you plan on running several programs simultaneously. For several of the problems at https://benchmarksgame-team.pages.debian.net/benchmarksgame/... on the Computer Language Benchmarks Games, it shows the Julia programs as using several times more memory than similar Java programs. It's even worse when comparing to…

Remember that for those benchmarks, C, C++, and Rust compile the binaries and then run the benchmark, while Julia compiles as part of the benchmark. The memory usage is dramatically decreased if you do a workflow similar to the other languages using something like PackageCompiler.jl to build a binary that is benchmarked. If you treat the other languages as "JIT" and include those factors in the benchmarking process,…

While this is true, it's also true that Julia currently brings more into ram than it should (and brings stuff in earlier than it should). The best example of this is that we currently allocate buffers ahead of time for matrix multiplication temporaries (even if matrix multiplication is never performed in the program). We do need to fix that type of thing eventually.

Re: JNumPy: Writing high-performance C extensions for Python in minutes

#59
post #56

Earlier quoted context omitted.

It seems to use a lot of memory which can be unacceptable for many people and can become impractical if you plan on running several programs simultaneously. For several of the problems at https://benchmarksgame-team.pages.debian.net/benchmarksgame/... on the Computer Language Benchmarks Games, it shows the Julia programs as using several times more memory than similar Java programs. It's even worse when comparing to…

Remember that for those benchmarks, C, C++, and Rust compile the binaries and then run the benchmark, while Julia compiles as part of the benchmark. The memory usage is dramatically decreased if you do a workflow similar to the other languages using something like PackageCompiler.jl to build a binary that is benchmarked. If you treat the other languages as "JIT" and include those factors in the benchmarking process,…

Thanks, I hadn't noticed that Julia was also doing compilation during the benchmarking and I'm curious if it would be hard to get those benchmarks to be done with PackageCompiler.jl. With that being said, the memory usage still seems very high. For many of the problems, Julia is often using the second most memory of all the programming language implementations being benchmarked and several of those other programming languages implementations (Node.js, OpenJDK, .NET, etc...) also are doing some sort of compilation during the benchmarking as well. I also tried compiling several of the C programs with GCC and GCC usually only used ~30 MB of additional memory so even if you add that to the the memory usage of the C programs, it's still much less than the Julia programs. Using PackageCompiler.jl does look promising but that does add another step which looks to be slightly more involved (looks a bit comparable to using profile-guided optimization) than compilation steps for other programming languages and some could see that as another problem for using Julia in real projects.

Re: JNumPy: Writing high-performance C extensions for Python in minutes

#60
post #59

Earlier quoted context omitted.

Remember that for those benchmarks, C, C++, and Rust compile the binaries and then run the benchmark, while Julia compiles as part of the benchmark. The memory usage is dramatically decreased if you do a workflow similar to the other languages using something like PackageCompiler.jl to build a binary that is benchmarked. If you treat the other languages as "JIT" and include those factors in the benchmarking process,…

Thanks, I hadn't noticed that Julia was also doing compilation during the benchmarking and I'm curious if it would be hard to get those benchmarks to be done with PackageCompiler.jl. With that being said, the memory usage still seems very high. For many of the problems, Julia is often using the second most memory of all the programming language implementations being benchmarked and several of those other programming…

In the past the benchmarkgame people haven't let us use PackageCompiler. The big memory gain would be that you could skip loading things like LinearAlgebra and the other standard libraries that aren't being used (but which you are still loading code for).
Post reply on HN