Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Test for lists in Cython

github.com

Post reply on HN
Print view View original

141–147 of 147 posts

Re: Test for lists in Cython

#141

Post by N1H1L »

ZuLuuuuuu wrote: post #8

I feel like the #1 downside of Python for the last few years is that you cannot take advantage of multiple cores of a CPU easily. Especially when you think it is heavily used in data analysis. We use Python for data analysis as well, and for 95% of operations we are doing, numpy is fast enough that we don't have any complaints. But sometimes, we do wish to be able to take advantage of all the cores in our CPUs, espec…

Try dask

Distribute your data and run everything as dask.delayed and then compute only at the end.

Also check out legate.numpy from Nvidia which promises to be a drop in numpy replacement that will use all your CPU cores without any tweaks on your part.

https://github.com/nv-legate/legate.numpy

Re: Test for lists in Cython

#142

Post by N1H1L »

cfcf14 wrote: post #51
PartiallyTyped wrote: post #46

Earlier quoted context omitted.

Have you tried numba+numpy? In my experience, it is much faster than Jax and can compile to cuda. It's not caveat free, but it also removes the hustle of labeling arrays as donated in Jax. You may find this interesting https://github.com/scikit-hep/iminuit/blob/develop/tutorial/...

Have you been successful in implementing non-trivial computational code in numba/numpy? I've always found it starts to really break for anything which isn't really trivial, and the errors are mostly non-prescriptive and highly verbose.

I have a 25,000+ loc scientific codebase in Python, with lot of use in my specialized domain. I gave up on numba, because it's speedups vanish when transitioning to real problems. And secondly, it's bugs are not that intuitive.

Re: Test for lists in Cython

#143

Post by yummypaint »

ZuLuuuuuu wrote: post #8

I feel like the #1 downside of Python for the last few years is that you cannot take advantage of multiple cores of a CPU easily. Especially when you think it is heavily used in data analysis. We use Python for data analysis as well, and for 95% of operations we are doing, numpy is fast enough that we don't have any complaints. But sometimes, we do wish to be able to take advantage of all the cores in our CPUs, espec…

What I do is write single-threaded programs that process only a section of data, and manage them with HTCondor. It's some extra overhead and complexity but this way you can easily scale to hundreds of machines without changing anything. You also get mature queuing and job management tools that work independently of your program. If your application is easy to paralellize and you might need more than one machine i highly reccommend this route.

Re: Test for lists in Cython

#144

Post by nuisance-bear »

dr_zoidberg wrote: post #128
optimalsolver wrote: post #126

Earlier quoted context omitted.

Thanks for this. Will Julia advocates every use honest benchmarks to make their language look good? I doubt it.

OP has an open pull request on his repo[0] where someone made basically this same change (different names for the variables, but same idea). According to OP, he tried this but it resulted in slower execution for him. I'm not sure he really followed what that PR says. In the case of the submitter, it says on their machine it gave a 2x speedup (smaller than my ~7x, but still significant). [0] https://github.com/00sapo/…

If you carefully read OP's response in the PR, he says he tried static typing make_list in addition to iterate_list. Your comment elides the resource allocation step.

Like OP, I observed that static typing make_list yields little benefit. It's runtime is 3x the runtime of iterate_list. And that makes sense. I'm not sure why we'd expect Cython to speed up the allocation of large numbers of Python objects.

Re: Test for lists in Cython

#145

Post by squarefocus »

galangalalgol wrote: post #49
jakobnissen wrote: post #45

I'm a massive Julia fanboy, but I would not extend Python with Julia if I could choose not to. Julia has a _massive_ runtime with a hello-world script consuming 150 MB of RAM, not to mention the dreaded startup-time. It's better and easier to use Julia as your main top-level "glue" language and call Python/Rust/C from Julia. Julia is in many ways a better glue language than Python - better multithreading, easier call…

You seem to understand Julia well. Is there a reason there is no nexus plugin or way to mirror the julia repo for dev networks that dont have unrestricted access to the internet, or are even airgapped. I see multiple people asking this online, so it is a common enough problem, but it seems like people are saying the Julia approach makes it hard to support.

It is possible to write a script that mirrors the registry, packages, and artifacts and set up a dev image that respects it. The script basically just needs to use Pkg and Base. However, artifacts end up taking too much space and then you have to decide how to host only a subset of package versions intelligently and propagate that change through at least the registry and artifact mirrors.

I went far down this path before infosec agreed to give github / highly respected cdn access.

Re: Test for lists in Cython

#146

Post by dr_zoidberg »

nuisance-bear wrote: post #144
dr_zoidberg wrote: post #128

Earlier quoted context omitted.

OP has an open pull request on his repo[0] where someone made basically this same change (different names for the variables, but same idea). According to OP, he tried this but it resulted in slower execution for him. I'm not sure he really followed what that PR says. In the case of the submitter, it says on their machine it gave a 2x speedup (smaller than my ~7x, but still significant). [0] https://github.com/00sapo/…

If you carefully read OP's response in the PR, he says he tried static typing make_list in addition to iterate_list . Your comment elides the resource allocation step. Like OP, I observed that static typing make_list yields little benefit. It's runtime is 3x the runtime of iterate_list . And that makes sense. I'm not sure why we'd expect Cython to speed up the allocation of large numbers of Python objects.

But OP didn't type iterate_list either. Perhaps he saw little benefit on one and assumed the same would happen in the other. Cython's annotation mode (cython -a) doesn't really help either here, showing both OPs and optimized versions with a strong yellow color, indicating that they would be about the same, but compiling and trying both definitely shows the improvement.

Another funny thing: making a pure python version that uses the sum() built-in results in code that runs 80% as fast as the cython version, but also gives lower error (e-8 versus e-4 error that gets accumulated when the numbers are summed up one by one).

Re: Test for lists in Cython

#147

Post by ZuLuuuuuu »

captain_price7 wrote: post #138
ZuLuuuuuu wrote: post #80

Earlier quoted context omitted.

I didn't know that. So let's say I have 4 regular Python threads calling the same function, and in this function, let's say I call numpy.add on the same array (but different parts of the array), then will it actually use different cores for these 4 different threads? I will try it out, if it works, then that's actually great and would be super easy.

The answer is No, isn't it?

I wrote a small code to test it. Unfortunately it didn't work, but I don't know if the problem is me, maybe I made a mistake...
Post reply on HN