Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Numba: A High Performance Python Compiler

numba.pydata.org

Post reply on HN
Print view View original

31–40 of 63 posts

Re: Numba: A High Performance Python Compiler

#31

Post by samsquire »

csdvrx wrote: post #29
samsquire wrote: post #24

Earlier quoted context omitted.

Thank you for your reply. I'm thinking the code for your average CRUD or even desktop compositor. A compositor copies pixels from multiple places into one place. Surely that can be defined with a simple loop? But no there's hundreds of APIs in the way. Add Wayland and X11 and you have something that is opaque and understood by very few people. The motivation behind my comment was that most of programming computers is…

For a compositor, I'd think of the set pixels being changed (an "invalidation") a good example: the constraint would be to update it on the screen. Unchanged? Don't bother, leave it as-is. I think that's how Intel power saving works. Now think about the MVC model: some changes in the data could result in a change in the view if the data currently shown on screen is what has changed - like triggers in SQL. I wonder if…

You're right, and thank you for bringing async up.

And thankyou for bringing up constraint propagation.

One of my ideas is the definition of formulas that act as materialized views over other materialised views. So we can layer materialized views over other materialized views and then work out a derived formula that is potentially nearer to what we want and potentially summarise the formula without needing to calculate the underlying views, we can compute the formula directly.

Is this differential dataflow?

I think it's an application of algebra and JIT compilers could do it to expressions if we fed symbolic expressions of programming languages into sympy or machine algebra.

In react, react does diffing between virtual DOM nodes to see if there are changed. There is also dirty region checking in old games and damage regions. These problems are mathematically defined.

Here's my writings on the idea https://github.com/samsquire/ideas4#31-algebraic-materialise...

Re: Numba: A High Performance Python Compiler

#32

Post by stared »

kylebarron wrote: post #25
stared wrote: post #11

As a side note, now it is easy to write Rust code, which can be directly used in Python - https://github.com/PyO3/pyo3 . It cannot use NumPy and other libraries (since it is Rust), but at the same time, I see its potential in creating high-performance code to be used in Python numerical environment.

On the contrary, it can use and interface with numpy quite easily: https://github.com/PyO3/rust-numpy

Good to know!

Re: Numba: A High Performance Python Compiler

#33

Post by Alifatisk »

m_c_g wrote: post #19
chestertn wrote: post #16

I will save you the pain: switch to Julia.

Indeed! Converting one's entire code base to a different language ecosystem, finding equivalents to each of your third-party dependencies, is less painful than employing a library to selectively compile a few performance bottlenecks in your code. (Modules like PyJulia facilitate a more incremental approach.)

/s

Re: Numba: A High Performance Python Compiler

#34

Post by melony »

grej wrote: post #12

We were very heavy numba users at my former company. I would even go so far as to say numba was probably the biggest computational enabler for the product. I’ve also made a small contribution to the library. It’s a phenomenal library for developing novel computationally intensive algorithms on numpy arrays. It’s also more versatile than Jax. In presentations, I’ve heard Leland McInnes credits numba often when he spea…

These days I have switched to

https://www.taichi-lang.org/

Re: Numba: A High Performance Python Compiler

#35

Post by netjiro »

fwilliams wrote: post #13
gjvc wrote: post #4

Earlier quoted context omitted.

What if I'm (in Python) doing non-numerical stuff like parsing text and generating code? What JIT / AOT tooling (if any) is suitable?

I have personally gotten a lot of mileage from just writing the compute heavy parts of my code in C++ and exposing it to Python with a tool like PyBind11 [1] or NumpyEigen [2]. I find tools like numba and cython to be more trouble than they're worth. [1] https://github.com/pybind/pybind11 [2] https://github.com/fwilliams/numpyeigen

I prototype in python or whatever, then, if the project survives into market and has legs I either buy more hardware or rewrite the expensive parts in C++.

Reduces calendar time, risk, cost. And I'm likely to make better decisions once the code and market is better understood after the prototype is tested under real world conditions and the requirements have changed (like they always seem to do).

Re: Numba: A High Performance Python Compiler

#36

Post by singhrac »

gjvc wrote: post #4
chazeon wrote: post #2

Software from our group (cij[1], qha[2]) were developed when numba seems to be the best option for JIT. It generates more pain in the hindsight. It generates a lot of depreciated warning due to unstable API, locked numpy to a certain version (i remember 1.21) due to compatibility issues, and when M1 Mac comes out, there were for a long time lack of llvmlite porting to the new platform, so cannot run on these new Macs…

What if I'm (in Python) doing non-numerical stuff like parsing text and generating code? What JIT / AOT tooling (if any) is suitable?

As a slight contrast to the other responses, I found setting up maturin (Rust + Python) very straightforward since the documentation is recent, and I find it's easy to write parsers in Rust because the ADT syntax is very terse.

Re: Numba: A High Performance Python Compiler

#37

Post by PheonixPharts »

grej wrote: post #12

We were very heavy numba users at my former company. I would even go so far as to say numba was probably the biggest computational enabler for the product. I’ve also made a small contribution to the library. It’s a phenomenal library for developing novel computationally intensive algorithms on numpy arrays. It’s also more versatile than Jax. In presentations, I’ve heard Leland McInnes credits numba often when he spea…

> It’s also more versatile than Jax

Does numba do automatic differentiation?

I view JAX as primarily an automatic differentiation tool with the bonus that it makes great use of XLA and can easy make use of GPU/TPUs.

I don’t usually see numba and JAX as solving the same problem, but would be excited to be wrong

Re: Numba: A High Performance Python Compiler

#38

Post by IceHegel »

Are there a standard set of benchmarks these python JIT projects use?

I’m very interested in adding something like this to some projects but it needs to be 10-100x faster to be worth the hassle. Otherwise, for our applications, it’s a better time investment to rewrite in Go and get the speed and pro tooling than to further optimize python.

Re: Numba: A High Performance Python Compiler

#39

Post by doliveira »

IceHegel wrote: post #38

Are there a standard set of benchmarks these python JIT projects use? I’m very interested in adding something like this to some projects but it needs to be 10-100x faster to be worth the hassle. Otherwise, for our applications, it’s a better time investment to rewrite in Go and get the speed and pro tooling than to further optimize python.

Lol, matrix arithmetics and scientific programming in Go

Re: Numba: A High Performance Python Compiler

#40

Post by hedgehog »

IceHegel wrote: post #38

Are there a standard set of benchmarks these python JIT projects use? I’m very interested in adding something like this to some projects but it needs to be 10-100x faster to be worth the hassle. Otherwise, for our applications, it’s a better time investment to rewrite in Go and get the speed and pro tooling than to further optimize python.

If you have numeric code that's too slow in Numba your next stop will likely involve a big multi-language effort and GPU specialists and none of that would be in Go except maybe a wrapper for your apps.
Post reply on HN