Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Test for lists in Cython

github.com

Post reply on HN
Print view View original

51–60 of 147 posts

Re: Test for lists in Cython

#51

Post by cfcf14 »

PartiallyTyped wrote: post #46
jeremiecoullon wrote: post #44

Earlier quoted context omitted.

I've been using JAX ( https://jax.readthedocs.io/en/latest/ ) for scientific computing in general (in particular MCMC algorithms), as it's really fast. Even on a CPU you get massive speedups compared to numpy (can be up to 2 or 3 orders of magnitude faster in some cases). The main selling point of the library is automatic differential and compilation to XLA, but I've been using it even when I don't need gradients, as…

Have you tried numba+numpy? In my experience, it is much faster than Jax and can compile to cuda. It's not caveat free, but it also removes the hustle of labeling arrays as donated in Jax. You may find this interesting https://github.com/scikit-hep/iminuit/blob/develop/tutorial/...

Have you been successful in implementing non-trivial computational code in numba/numpy? I've always found it starts to really break for anything which isn't really trivial, and the errors are mostly non-prescriptive and highly verbose.

Re: Test for lists in Cython

#53

Post by jakobnissen »

galangalalgol wrote: post #49
jakobnissen wrote: post #45

I'm a massive Julia fanboy, but I would not extend Python with Julia if I could choose not to. Julia has a _massive_ runtime with a hello-world script consuming 150 MB of RAM, not to mention the dreaded startup-time. It's better and easier to use Julia as your main top-level "glue" language and call Python/Rust/C from Julia. Julia is in many ways a better glue language than Python - better multithreading, easier call…

You seem to understand Julia well. Is there a reason there is no nexus plugin or way to mirror the julia repo for dev networks that dont have unrestricted access to the internet, or are even airgapped. I see multiple people asking this online, so it is a common enough problem, but it seems like people are saying the Julia approach makes it hard to support.

I'm sorry, I have no idea, but I'm sure someone else has insight.

Why would you want to mirror the Julia repo instead of just downloading already-compiled binaries? Do you want to maintain your own fork with your own set of patches?

Re: Test for lists in Cython

#54

Post by efaref »

iExploder wrote: post #4
ur-whale wrote: post #3

From the benchmarks: > Rust is not that fast because it needs to copy data; I'm surprised. Don't know much about Rust, but isn't it hailed as being competitive with C/C++ ?

rust has concept of references, maybe it was referring to its particular implementation of the problem

The implementations for Rust and Julia are different in a way that means the comparison is not really apples-to-apples.

The Julia implementation replaces the input list with a Julia object, which is of course trivial for Julia to iterate over.

The Rust implementation is asked to convert the list-of-list-of-floats back to Python types, and then back into Rust types. Of course converting 100M floats back and forth is going to take a non-trivial amount of time.

The equivalent in Rust would be to wrap the return value in a PyObject that contains the Rust `Vec>`, and the Rust code could then just iterate that directly. It would even be made safe without the GIL as you'd implement your own mutex on the Rust side if you wanted it to be mutable.

If you do this then it runs in about 1.2s on my laptop.

Re: Test for lists in Cython

#55

Post by craftinator »

cfcf14 wrote: post #51
PartiallyTyped wrote: post #46

Earlier quoted context omitted.

Have you tried numba+numpy? In my experience, it is much faster than Jax and can compile to cuda. It's not caveat free, but it also removes the hustle of labeling arrays as donated in Jax. You may find this interesting https://github.com/scikit-hep/iminuit/blob/develop/tutorial/...

Have you been successful in implementing non-trivial computational code in numba/numpy? I've always found it starts to really break for anything which isn't really trivial, and the errors are mostly non-prescriptive and highly verbose.

I had the same experience about 2 years ago. Maybe it's changed since then? It was nice when you had a pure math function to write, but otherwise seemed to be unreliable, especially in multithreaded and multiprocess situations.

Re: Test for lists in Cython

#56

Post by ZeroCool2u »

galangalalgol wrote: post #49
jakobnissen wrote: post #45

I'm a massive Julia fanboy, but I would not extend Python with Julia if I could choose not to. Julia has a _massive_ runtime with a hello-world script consuming 150 MB of RAM, not to mention the dreaded startup-time. It's better and easier to use Julia as your main top-level "glue" language and call Python/Rust/C from Julia. Julia is in many ways a better glue language than Python - better multithreading, easier call…

You seem to understand Julia well. Is there a reason there is no nexus plugin or way to mirror the julia repo for dev networks that dont have unrestricted access to the internet, or are even airgapped. I see multiple people asking this online, so it is a common enough problem, but it seems like people are saying the Julia approach makes it hard to support.

Frankly, Julia is a young language, so these things have yet to materialize. Also, for those that really do want it, the Julia package mirror is one of the few products that the company behind Julia sells as part of JuliaHub. It is inconvenient though that it doesn't fit in with the rest of the languages in Nexus.

Re: Test for lists in Cython

#57

Post by PartiallyTyped »

cfcf14 wrote: post #51
PartiallyTyped wrote: post #46

Earlier quoted context omitted.

Have you tried numba+numpy? In my experience, it is much faster than Jax and can compile to cuda. It's not caveat free, but it also removes the hustle of labeling arrays as donated in Jax. You may find this interesting https://github.com/scikit-hep/iminuit/blob/develop/tutorial/...

Have you been successful in implementing non-trivial computational code in numba/numpy? I've always found it starts to really break for anything which isn't really trivial, and the errors are mostly non-prescriptive and highly verbose.

I still have some issues in nojit mode, but jit is most of the time fine. It's still a bit iffy with lists, but most of the time I can use numpy arrays.

TBF I am mainly using it for mostly pure path functions.

Re: Test for lists in Cython

#58

Post by adwn »

mhh__ wrote: post #23
volta83 wrote: post #16

Earlier quoted context omitted.

To add, it is trivial to guarantee that Rust code is "zero-copy", so if this is something you care about, Rust allows your program to fail to compile if it tries to make a copy.

Godbolt example?

Here you go:

https://play.rust-lang.org/?version=stable&mode=debug&editio...

Because the data structure LotsOfData doesn't implement the Clone trait, attempting to create a copy will fail at compile-time.

Re: Test for lists in Cython

#59

Post by Gravityloss »

I was a huge fan of Matlab way back. I wrote a hundred small Matlab programs for usage in the research department of the company I worked in. Doing data operations in Matlab was way more elegant than in say, Numpy which I tried later. Development was fast and ergonomics were good.

After using Ruby for years, returning to Matlab style code in Julia felt somewhat awkward. Instead of my_array.length you have length(my_array). In my personal preference the method call is just a nicer way of doing the same thing. The single-instruction-multiple-data or dot notation sometimes worked and sometimes didn't, so you had to resort to loops anyway.

    % Julia
    a=[1 2 3 4]
    a.^2
    log.(a)

    # Ruby
    a=[1,2,3,4]
    a.map{|element| element.pow(2)}
    a.map{|element| Math.log(element)}
Ruby and map or each have their verbosity but overall it feels a more robust "hammer" for general programming tasks. On the other hand, Julia can be really dense and still easy to understand.

Other languages of course take some of these things even further. Maybe some day I will find an ergonomics and nicety first successor to both.

Plotting with two Y axes or generating histograms in Julia was also way harder than I remember it being in Matlab. Also manually having to load the file before every run to see the changes in action added a lot of overhead to the workflow.

The workflow I was used to in Matlab involved very frequent making changes to code, running of the code that usually made some plot. Which is one command in Matlab, and plotting was fast, in 2001 already, on Windows NT 4. In Julia, you have to first load the modified file, then run it. Plotting takes a long time. One just can't get nearly as productive with it in 2021, compared to Matlab of 2001 vintage.

What language would I pick if I had to do some quick analysis from some tables downloaded from the internet? Probably Julia still. If I had free access to Matlab, I would probably use it though.

Re: Test for lists in Cython

#60

Post by jhgb »

Gravityloss wrote: post #59

I was a huge fan of Matlab way back. I wrote a hundred small Matlab programs for usage in the research department of the company I worked in. Doing data operations in Matlab was way more elegant than in say, Numpy which I tried later. Development was fast and ergonomics were good. After using Ruby for years, returning to Matlab style code in Julia felt somewhat awkward. Instead of my_array.length you have length(my_a…

> Instead of my_array.length you have length(my_array). In my personal preference the method call is just a nicer way of doing the same thing.

Well...they're both method calls, aren't they? (So they're both the nicer way?)

Post reply on HN