Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Test for lists in Cython

github.com

Post reply on HN
Print view View original

81–90 of 147 posts

Re: Test for lists in Cython

#81

Post by nostoc »

leephillips wrote: post #67
jhgb wrote: post #60

Earlier quoted context omitted.

> Instead of my_array.length you have length(my_array). In my personal preference the method call is just a nicer way of doing the same thing. Well...they're both method calls, aren't they? (So they're both the nicer way?)

Those who are attached to the class-based OOP model used, for example, in Python will find the dot notation more natural. But Julia’s multiple dispatch is a superset of this, and unarguably more powerful and flexible. Python OOP leads to monstrosities like >>> ', '.join(['1', '2', '3']) '1, 2, 3' >>> '1, 2, 3'.split(', ') ['1', '2', '3']

That's not a failure of OOP, but a failure of design.

This could have easily been avoided if `join` would have been a method of the list object, not the string object. But we're stuck with this now.

Re: Test for lists in Cython

#82

Post by pletnes »

ZuLuuuuuu wrote: post #80
pletnes wrote: post #37

Earlier quoted context omitted.

Numpy operations release the GIL (usually at least) so you can use a threadpool and, indeed, share memory. Just try it and you may be pleasantly surprised. Dask is great if you’re processing large amounts of data, and it recommends and supports threads for this reason.

I didn't know that. So let's say I have 4 regular Python threads calling the same function, and in this function, let's say I call numpy.add on the same array (but different parts of the array), then will it actually use different cores for these 4 different threads? I will try it out, if it works, then that's actually great and would be super easy.

Great, then my comment was worth it!

Not sure there’s anything to prevent you from foot-shooting if you’re operating on the same array. I’d expect that this is only safe to do if, as you say, arrays are not overlapping. Let me know what you find.

Re: Test for lists in Cython

#83

Post by leephillips »

jhgb wrote: post #79
leephillips wrote: post #67

Earlier quoted context omitted.

Those who are attached to the class-based OOP model used, for example, in Python will find the dot notation more natural. But Julia’s multiple dispatch is a superset of this, and unarguably more powerful and flexible. Python OOP leads to monstrosities like >>> ', '.join(['1', '2', '3']) '1, 2, 3' >>> '1, 2, 3'.split(', ') ['1', '2', '3']

Multiple dispatch is usually class-based as well.

Not in Julia.

Re: Test for lists in Cython

#84

Post by PartiallyTyped »

jeremiecoullon wrote: post #71
PartiallyTyped wrote: post #46

Earlier quoted context omitted.

Have you tried numba+numpy? In my experience, it is much faster than Jax and can compile to cuda. It's not caveat free, but it also removes the hustle of labeling arrays as donated in Jax. You may find this interesting https://github.com/scikit-hep/iminuit/blob/develop/tutorial/...

I haven't tried numba but I've heard good things about it! Nice linked tutorial. If I understand correctly, you pass in jitted functions (using numba, and jax) into iminuit which does the optimisation? With Jax you can write native for loops that can also be jitted (I imagine you can also do this in numba?); this can then be really fast. Though in that case you would have to write the optimisation algorithm yourself…

> I haven't tried numba but I've heard good things about it! Nice linked tutorial. If I understand correctly, you pass in jitted functions (using numba, and jax) into iminuit which does the optimisation?

Yes, you can just pass the function and it runs the optimization, in severe cases, you can start by doing a grid `.scan` or `.simplex` (Nelder-Mead simplex method), then `migrad` to minimize, and `.hesse` for 1 sigma bound.

You can also provide a gradient function that, well computes the gradient instead of computing it numerically.

> I imagine you can also do this in numba

Yes! It compiles to native code.

> Another big speedup in Jax is due to vmap/pmap, which allow to vectorise/parallelise computation.

It's possible to compile some vmapped functions with numba too, if the link is any indication, you may see even greater speedup than just jax.grad

However, I do concur that jax's vmap is absolutely fantastic and I found it very useful on many occasions.

Re: Test for lists in Cython

#85

Post by jakobnissen »

syntonym2 wrote: post #65

I use python in a scientific context, but have so far not written much extensions for python in any of the languages tested. I'm interested in some guidance which language a) is easy to integrate with python and b) has some good performance, but this benchmark lacks the details to come to any conclusion. I tried to run the benchmark on my own computer, but the setup documentation was not enough for me to get the juli…

About two years ago (before I switched from Python to Julia), I was in the same boat as you. What I concluded was:

1) Calling into an actual static language like C or Rust is the best option. You get maximal performance and all the benefits of the static language. The downside is that you need to learn another language, and manage both languages in your project, including setup and compilation of the static language etc.

2) Cython is easiest for small-scale projects, since it integrates very well with Python, and you can learn it incrementally. But I found it annoying to work with - it felt like half a language that fell between Python and a proper static language. I ended up using Cython in the end, but I wasn't happy with it.

3) Numba looks interesting and promising. At least 2 years ago, it was too brittle and had too many situations where it didn't work or didn't give noticable speedups. I'm sure they improved it since then. I would definitely take a look.

You can always just learn Julia of course and have this entire problem of "my high-level language is too slow" completely disappear ;)

Re: Test for lists in Cython

#86

Post by jeremiecoullon »

PartiallyTyped wrote: post #84
jeremiecoullon wrote: post #71

Earlier quoted context omitted.

I haven't tried numba but I've heard good things about it! Nice linked tutorial. If I understand correctly, you pass in jitted functions (using numba, and jax) into iminuit which does the optimisation? With Jax you can write native for loops that can also be jitted (I imagine you can also do this in numba?); this can then be really fast. Though in that case you would have to write the optimisation algorithm yourself…

> I haven't tried numba but I've heard good things about it! Nice linked tutorial. If I understand correctly, you pass in jitted functions (using numba, and jax) into iminuit which does the optimisation? Yes, you can just pass the function and it runs the optimization, in severe cases, you can start by doing a grid `.scan` or `.simplex` (Nelder-Mead simplex method), then `migrad` to minimize, and `.hesse` for 1 sigma…

Nice to know that numba has these features; I'll have to check it out at some point!

Re: Test for lists in Cython

#87

Post by leephillips »

deetsb wrote: post #77
leephillips wrote: post #67

Earlier quoted context omitted.

Those who are attached to the class-based OOP model used, for example, in Python will find the dot notation more natural. But Julia’s multiple dispatch is a superset of this, and unarguably more powerful and flexible. Python OOP leads to monstrosities like >>> ', '.join(['1', '2', '3']) '1, 2, 3' >>> '1, 2, 3'.split(', ') ['1', '2', '3']

As I’m clearly missing the bigger picture, what do you find to be super offensive about this? I read the first as the concat symbol applied to an iterable leads to string concated by the concat symbol. I read the second as an iterable broken by split symbol leads to an iterable of the chunks.

It’s hard to remember what order to put things in, because the two methods follow opposite conventions. And that’s because this type of OO design has no obvious method of organization.

In the first case, I want to do something to a list: join it into a string. So, clearly, I need a list method? But no, I need to engage in some form of indirection; for some reason, I need to reach for a string method. Even if there is no string that I want to use as a delimiter. In that case, I need to use a string method on an empty string.

OK, I’ll play along. Now I want to take the string and split it into an array. Now that I’ve been educated, I know better than to try the sensible thing. Pre-enlightenment, I would have reached for a method applied to the thing that I wanted to transform. But now I know I should think backwards, and use a method applied to the delimiter. OOPs.

Re: Test for lists in Cython

#88

Post by rsfern »

Gravityloss wrote: post #75
leephillips wrote: post #63

Earlier quoted context omitted.

The standard way to make a histogram in Julia is histogram(data) Using the latest version (1.6 - although 1.6.1 just came out) the time to first plot is just a few seconds. After that, plotting in the REPL is instantaneous. I probably don’t understand what you’re getting at when you speak of making frequent changes to code. REPL-based development in Julia is excellent, and there are Pluto notebooks as well.

The way I would like to work is to have the repl open on the left hand side of the screen and code editor (like Sublime text, I'm sure many use Vim) on the right hand side. I would run the code in repl (just using up-arrow and enter), get some plot, modify the code in the editor, save it, and rerun it repl. Repl is used for connecting inputs to the program, not for editing. Often times you want to develop some small…

Check out Revise.jl, it enables this workflow, and it’s awesome.

https://docs.julialang.org/en/v1/manual/workflow-tips/#Revis...

Re: Test for lists in Cython

#89

Post by leephillips »

leephillips wrote: post #87
deetsb wrote: post #77

Earlier quoted context omitted.

As I’m clearly missing the bigger picture, what do you find to be super offensive about this? I read the first as the concat symbol applied to an iterable leads to string concated by the concat symbol. I read the second as an iterable broken by split symbol leads to an iterable of the chunks.

It’s hard to remember what order to put things in, because the two methods follow opposite conventions. And that’s because this type of OO design has no obvious method of organization. In the first case, I want to do something to a list: join it into a string. So, clearly, I need a list method? But no, I need to engage in some form of indirection; for some reason, I need to reach for a string method. Even if there is…

Here it is in Julia:

     julia> join([1 2 3], ", ")
     "1, 2, 3"


     julia> split("1, 2, 3", ", ")
     3-element Vector{SubString{String}}:
      "1"
      "2"
      "3"
Note how they are both functions. The data that they operate on is the first argument, in both cases. The optional second argument is the obvious next most important thing, the delimiter. Other optional arguments come after that. There is nothing to remember, because it makes sense.

Re: Test for lists in Cython

#90

Post by leephillips »

rsfern wrote: post #88
Gravityloss wrote: post #75

Earlier quoted context omitted.

The way I would like to work is to have the repl open on the left hand side of the screen and code editor (like Sublime text, I'm sure many use Vim) on the right hand side. I would run the code in repl (just using up-arrow and enter), get some plot, modify the code in the editor, save it, and rerun it repl. Repl is used for connecting inputs to the program, not for editing. Often times you want to develop some small…

Check out Revise.jl, it enables this workflow, and it’s awesome. https://docs.julialang.org/en/v1/manual/workflow-tips/#Revis...

Also, my Julia REPL is in a term buffer in vim, and I communicate with it (in both directions) from my editing buffer using the vim-sendtowindow plugin.
Post reply on HN