Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cython is 20

blog.behnel.de

Post reply on HN
Print view View original

21–30 of 71 posts

Re: Cython is 20

#21

Post by Galanwe »

jokoon wrote: post #16

Question: I found python "bindings" for SFML, written in cython, and patched them a bit. I guess Cython is not really made to write bindings, but is it easier to write bindings with cython or cpython?

As someone who has been writing python bindings regularly for 10 years:

Writing bindings in Cython is much, much faster in terms of development time. It fits nicely and unintrusively in an already python packaged library. You can gently add some C functions or call C libraries in minutes.

You won't have a full control of what's happening though. Just have a look at the generated code and you'll see the mess of indirections that are generated.

Cython bindings become limited when you have to build more complex stuff though, going deeper than just calling some C functions. The typical case is when you have to actually handle the lifetime and borrowing of C native objects.

At that point, CPython will be the way to go, but it's much more code, and very error prone: you have to manually keep track of reference counting.

Re: Cython is 20

#22

Post by olau »

fname11 wrote: post #11
pjmlp wrote: post #7

While it is nice that this option is available, it would be much better if Python itself would embrance the necessary runtime capabilties to not have to rely on it.

This is not going to happen. GvR has successfully ignored Cython and PyPy for decades and has attached himself to a JIT project at Microsoft (has anything emerged?). CPython is in the hands of not really productive bigcorp representatives who care about large legacy code bases. My guess is that CPython will be largely the same in 10 years, with the usual widely hyped initiatives that go nowhere ("need for speed etc."…

While it's true speed was not a priority, I think most of those initiatives didn't try hard to work with upstream.

The Microsoft funded project is different, they're merging things. I don't think they've started on a JIT translator yet, though, last time I looked they were busy picking lower-hanging fruit. From watching their communications, I think they might get there at some point.

It's not as simple as just emitting machine code, though. To get something in the same magnitude of typical C code, you need to deduce types and peel away the boxing and unboxing layers.

Re: Cython is 20

#23

Post by physicsguy »

jokoon wrote: post #19
physicsguy wrote: post #2

I love Cython. I really feel like it's the right balance of usability and allowing you to do what you want/need. Want to make your code a bit faster? Write Python with type annotations. Want to call a C library? Just import the header, and then use it from a function. Pybind11 is also great, but quite different in aims - I feel like it's more like a project for C++ programmers wanting to expose functionality to Pytho…

Any benchmarks for type annotations? I already wrote a few patch for pysfml, which is written in cython, it was a bit awkward, and now I'm asking myself if cython is really the right tool to write bindings, compared to cpython, for example.

Gonna depend a huge amount on what you're doing to be honest. I used it for physics modelling codes and it made a bit of a difference (comparable to Numba) but dropping to C for the main computation routines was what we ended up doing, and that worked very well for us.

It's very fast to write for, that's the main benefit. Use it together with profiling and just pick off the slowest part first.

Re: Cython is 20

#24

Post by physicsguy »

chaxor wrote: post #17
pjmlp wrote: post #7

While it is nice that this option is available, it would be much better if Python itself would embrance the necessary runtime capabilties to not have to rely on it.

Don't count on it - switch to Julia instead.

Julia is great if you can afford to spend 5 minutes sitting around for your session to load, but most people have things to do.

Re: Cython is 20

#25

Post by hansor »

kubb wrote: post #3

and i still have no idea what i could use it for...

We had to parse dozeon of 20GB files daily with super complex structure and not in linear structure. With Cython (finally we migrated to Pypy) we gained around 20-60x speedup.

Re: Cython is 20

#26

Post by dagw »

f311a wrote: post #20
pjmlp wrote: post #7

While it is nice that this option is available, it would be much better if Python itself would embrance the necessary runtime capabilties to not have to rely on it.

To write fast Cython, you basically need to write in C and control everything including Python API calls. No runtime will help with this.

That is really only true if you want to squeeze every drop of performance out of Cython. For the first 80% of performance gains you don't have to go that deep.

That is another thing that is nice about Cython, you don't have to learn all of Cython to be productive. Take your existing python function and just add some type annotations and you'll see real performance gains. Then you can profile your code and see what the next bottle neck is and fix that and so on.

So, yes, Cython gives you the power to manually control the GIL and the Python API calls and manage your own memory management and layout for those corner cases where that is what you need. Most of the time you can happily ignore all of that and get almost all of the speedup available.

Re: Cython is 20

#28

Post by erwincoumans »

I would recommend considering using NanoBind, the follow up of PyBind11 by the same author (Wensel Jakob), and move as much performance critical code to C or C++. https://github.com/wjakob/nanobind

If you really care about performance called from Python, consider something like NVIDIA Warp (Preview). Warp jits and runs your code on CUDA or CPU. Although Warp targets physics simulation, geometry processing, and procedural animation, it can be used for other tasks as well. https://github.com/NVIDIA/warp

Google Jax is another option, jitting and vectorizing code for TPU, GPU or CPU. https://github.com/google/jax

Re: Cython is 20

#29

Post by amkkma »

physicsguy wrote: post #24
chaxor wrote: post #17

Earlier quoted context omitted.

Don't count on it - switch to Julia instead.

Julia is great if you can afford to spend 5 minutes sitting around for your session to load, but most people have things to do.

When's the last time you've tried it? Load times are way better. New improvements for code caching have been merged in 1.8, and native code caching is coming in 1.9. It's a complex problem due to Julia's aggressive specialization and composability, so you can't just expect Julia to do it like other compiled languages.

Check out this recent writeup from a core dev: https://discourse.julialang.org/t/precompile-why/78770/8

Re: Cython is 20

#30

Post by ninjin »

physicsguy wrote: post #24
chaxor wrote: post #17

Earlier quoted context omitted.

Don't count on it - switch to Julia instead.

Julia is great if you can afford to spend 5 minutes sitting around for your session to load, but most people have things to do.

Julia has its own heap of issues, but five minutes is a load of bull:

    > time julia -e 'using Plots; plot(rand(10, 5), rand(10))'
    
    ________________________________________________________
    Executed in    5.77 secs    fish           external
       usr time    5.74 secs  214.00 micros    5.74 secs
       sys time    0.57 secs    0.00 micros    0.57 secs
This is also on a fairly old version at that:

    > julia -v
    julia version 1.6.3
Regardless, this conversation was to be about Cython; it which deserves a lot of praise in its own right as a tool in your toolbox to make the blasted snake run faster.
Post reply on HN