Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Julia 1.0

julialang.org

Post reply on HN
Print view View original

401–410 of 446 posts

Re: Julia 1.0

#401

Post by mlthoughts2018 »

Certhas wrote: post #384
marmaduke wrote: post #378

Earlier quoted context omitted.

I’m not insisting I have the only use case, but apart from the examples of traversing language boundaries, I haven’t see a good example of what’s so painful in Numba. What is so challenging that is requires code generation?

I have a data structure based on which I generate a dynamical behaviour that I want to integrate. So I construct a rhs. I further want the user of the library be able to pass it new functions that can be integrated into overall dynamical behaviour. There are different ways to achieve this, the simplest version is with closures. Pass a list of functions, and some parameters and I construct a right hand side function f…

This just sounds like bad software designto me. You are miswanting something overly generic that’s super not needed, and regardless of implementing in any given language, it sounds like it would benefit hugely from taking a more YAGNI approach to it, restricting its genericity based on likely usage (not intended or imagined usage), and either just manually writing stuff for an exhaustive set of use cases, or code genning just those cases and not allowing or encouraging arbitrary code gen of possible other cases.

I love it when libraries limit what can be done with them and document an extremely specific scope they apply to.

When libraries try to be all things to all people, it’s bad. A sophisticatedcode gen tool that enables library authors to choose to do that is a bad thing, not a good thing.

Re: Julia 1.0

#402

Post by ffriend »

marmaduke wrote: post #223
jernfrost wrote: post #217

Earlier quoted context omitted.

The point is that Python presents a very complex environment where you have to deal with different languages and technologies. For package developers it is a lot eaiser to use Julia. Saying one should focus more on Python and we would not have these problems is missing the point. Enormous resources by countless companies has been poured in to solve the performance problems of Python. It is almost impossible to do due…

I’m not missing the point: I’ve tried doing the same thing in Julia before as I do in Python, and it’s not a 10x difference. No language (relevant to this discussion) is ever going to cut down on lines of code required to do error checking, code gen, plotting etc. Sure, Julia is great as a high-level interface to LLVM, but so is Numba. Python’s design leaves something to be desired performance-wise for those coming f…

I have quite limited experience with Cython and tried Numba just a couple of times, but I'm curious how much would it take to rewrite one of my Julia libraries to them.

The library is for reverse-mode automatic differentiation, but let's put AD itself aside and talk about code generation. As an input to code generator, I have a computational graph (or "tape") - a list of functions connecting input and intermediate variables. As an output I want a compiled function for CPU/GPU. (Note: Theano used to do exactly this, but that's a separate huge system not relevant no Numba or Cython).

In Julia I follow the following steps:

1. Convert all operations on the tape to expressions (~approx 1 line per operation type).

2. Eliminate common subexpressions.

3. Fuse broadcasting, e.g. rewrite:

    c .= a .* b
    e .= c .+ d
into

    e .= a .* b .+ d
Dot near operations means that they are applied elementwise without creating intermediate arrays. On CPU, Julia compiler / LLVM then generates code that reads and writes to memory exactly once (unlike e.g. what you would get with several separate operations on numpy arrays). On GPU, CUDAnative generates a single CUDA kernel which on my tests is ~1.5 times faster then several separate kernels. Note that `.=` also means that the result of operation is directly written to a (buffered) destination, so it no memory is allocated in the hot loop.

4. Rewrite everything I can into in-place operations. Notably, matrix multiplication `A * B` is replaced with BLAS/CUBLAS alternative.

5. Add to the expression function header, buffers and JIT-compile the result.

In Python, I imagine using `ast` module for code parsing and transformations like common expression elimination (how hard it would be?). Perhaps, Numba can be used to compile Python code to fast CPU and GPU code, but does it fit with AST? Also, do Numba or Cython do optimizations like broadcasting and kernel fusion? I'd love to see side-by-side comparison of capabilities in such a scenario!

Re: Julia 1.0

#403

Post by jacobolus »

Certhas wrote: post #400
jacobolus wrote: post #393

Earlier quoted context omitted.

Yes, it would be interesting to know the history. Whoever designed the European convention for labeling building floors was numerically competent. Too bad medieval European mathematicians and the designers of Fortran weren’t. ;-)

The base level doesn't need to have a floor, it's just ground. Once you add a floor you are on the first floor above ground. Really your condescending tone as if all the mathematicians that prefer to work with 1 indexing are just incompetent is grating. I'm happy to be writing ``` for i in 1:n func!(a[i]) end ``` to iterate over an object of length n. Or split an array as a[1:m], b[m+1:n]. Slicing semantics which are…

Sorry, that last bit of my comment was gratuitous.

I am legitimately (mildly) curious about the history of the different naming conventions for floors of buildings though.

> The base level doesn't need to have a floor, it's just ground. Once you add a floor you are on the first floor above ground.

Yes, my point is this is an example where the European 0-based indexing system makes more sense (in my opinion) than the American 1-based indexing system. I speculate that whoever started calling the ground floor the “first floor” hadn’t really put much thought into how well that would generalize to large buildings with many floors including some underground.

Similarly, whoever decided the calendar should start at year 1 AD with the prior year as 1 BC hadn’t really considered that it might be nice to do arithmetic with intervals of years across the boundary.

There are many standard mathematical formulas which are clarified by indexing from 0. But nobody can switch because the 1-indexed version is culturally fixed. Most of the rest of the time the 0-indexed vs. 1-indexed versions makes basically no difference. It is rare that the 1-indexed version is notably nicer.

> Or split an array as a[1:m], a[m+1:n]

Yes, I find it substantially clearer to write this split as a[:m], a[m:]. Particularly when dealing with e.g. parsing serialized structured data. But also when writing numerical code. Carrying the extra 1s around just adds clutter, and forces me to add and subtract 1s all over the place; reasoning about it adds mental overhead, and extra bugs sneak in. (At least when writing Matlab code; I haven’t spent much time with Julia.)

Re: Julia 1.0

#404

Post by marmaduke »

ffriend wrote: post #402
marmaduke wrote: post #223

Earlier quoted context omitted.

I’m not missing the point: I’ve tried doing the same thing in Julia before as I do in Python, and it’s not a 10x difference. No language (relevant to this discussion) is ever going to cut down on lines of code required to do error checking, code gen, plotting etc. Sure, Julia is great as a high-level interface to LLVM, but so is Numba. Python’s design leaves something to be desired performance-wise for those coming f…

I have quite limited experience with Cython and tried Numba just a couple of times, but I'm curious how much would it take to rewrite one of my Julia libraries to them. The library is for reverse-mode automatic differentiation, but let's put AD itself aside and talk about code generation. As an input to code generator, I have a computational graph (or "tape") - a list of functions connecting input and intermediate va…

> In Julia I follow the following steps: … does it fit with AST?

I'm fairly certain the steps you've listed can be accomplished through AST manipulations, and would go something like

    def manip_ast(fn):
        import ast, inspect
        fn_ast = ast.parse(inspect.getsource(fn))
        new_fn_ast = …
        return compile(new_fn_ast, …)

    def rewrite(fn):
        fn = manip_ast(fn)
        fn = numba.jit(fn)

    @rewrite
    def func(*args):
        …
there's nothing in the language that prevents this from working with the autograd package, except no one's taken the time to implement it (https://github.com/HIPS/autograd/issues/47). That said, for many tasks with wide vector data, a DL framework is going to do ok, e.g. PyTorch.

> Julia compiler / LLVM then generates code that reads and writes to memory exactly once (unlike e.g. what you would get with several separate operations on numpy arrays)

Numba's gufuncs address exactly this + broadcasting over arbitrary input shapes. I've used this extensively. That said, I don't find fusing broadcasting is always a win, especially when arrays exceed cache size. Numba's CUDA support will also fuse jit functions into a single kernel, or generate device functions.

Sometimes you want manual control over kernel fusion, and I've found the Loopy (https://documen.tician.de/loopy/) to be fairly flexible in this regard, but it's a completely different approach compared to Numba/Julia.

I'd be interested in a side by side comparison as well, and I was thinking that the main difficulty would be that I couldn't write good Julia code, but maybe we can pair up, if that'd be interesting, to address several common topics that come up (fusion, broadcasting, generics but specialization, etc).

Re: Julia 1.0

#405

Post by lukka5 »

ChrisRackauckas wrote: post #153
athenot wrote: post #147

As an outsider, I'd like to see somewhere near the home page a few short snippets of code to get a feel for Julia and hopefully show the kind of uses for which it is a natural choice. Nim's home page¹ shows a piece of sample code right at the top. Perl6's page² has a few tabs quickly showing some patterns it's good at. Golang³ has a dynamic interpreter prepopulated with a Hello World. Julia's home page shows a nice f…

For scientific computing, showing the package ecosystem is the most important thing. When you look at this thread, people are asking about dataframes and differential equations. Julia's site reflects this: yes there are things like Pandas, and for plotting, etc.

I agree but and I think is better to show both. In a first section the package ecosystem is presented mostly targeting scientific computing and on a second section some code examples for general-purpose computing.

Re: Julia 1.0

#406

Post by marmaduke »

Certhas wrote: post #384
marmaduke wrote: post #378

Earlier quoted context omitted.

I’m not insisting I have the only use case, but apart from the examples of traversing language boundaries, I haven’t see a good example of what’s so painful in Numba. What is so challenging that is requires code generation?

I have a data structure based on which I generate a dynamical behaviour that I want to integrate. So I construct a rhs. I further want the user of the library be able to pass it new functions that can be integrated into overall dynamical behaviour. There are different ways to achieve this, the simplest version is with closures. Pass a list of functions, and some parameters and I construct a right hand side function f…

> With DifferentialEquations.jl I also could just

yep... I took a look at the DE packages in Julia today, and quite frankly they're much better than the situation in Python, perhaps because of one or more prolific applied mathematicians are making a concerted effort, which is lacking Python? I dunno, but I did recommend my colleagues look at Julia for DEs, for this reason.

That said,

> Pass a list of functions, and some parameters and I construct a right hand side function from it. Unfortunately this does not work with numba.

I'm pretty sure I've done this before with numba, so maybe getting concrete would help, e.g. an Euler step

    def euler(f, dt, nopython=False):
        @numba.jit(nopython=nopython)
        def step(x):
            return x + dt * f(x)
where user can provide regular Python function or a @numba.jit'd function. If a @numba.jit'd function is provided, and nopython=True, this should result in fused machine code. This sort of code gen through nest functions can be done repeatedly for e.g. the time stepping loop.

I've done this for CPU & GPU code for a complex model space (10 neural mass models, 8 coupling functions, 2 integration schemes, N output functions, ...) which, by the above pattern, results in flexible but fast code.

Is this a pattern that captures your use case or not yet?

> implement sparse matrix algorithms by hand because I can't call scipy.sparse.

agreed, this is a surprising omission, which I attribute to not much of the numerical Python community making use of Numba, but could be fixed rapidly.

> constrained my users to use numba compatible code in their right hand side functions

what did you run into that was problematic?

> I had to implement a non-linear solver from scratch in numba rather than being able to use scipys excellent set of solvers

I didn't follow; passing @numba.jit'd functions to scipy is in the Numba guide, so what exactly didn't work?

Re: Julia 1.0

#407

Post by marmaduke »

Certhas wrote: post #380
marmaduke wrote: post #376

Earlier quoted context omitted.

I’m aware of that, but high performance, fat vector AD would be done with eg Theano or PyTorch, not autograd.

That's the problem: You're locked into one particular library. You can not combine libraries without sacrificing massive performance. There is just no way around that. The numba story in that github issue mirrors my own experience: Excitement! This works! It's fast! Ok here are some limitations that I can work around. Hmm I would really like to use this library, in principle it should be possible to JIT its output/ma…

> That's the problem: You're locked into one particular library.

But that's not a issue resolved by any particular language; Julia appears to be free of lock-in because it hasn't had time to develop multiple, exclusive approaches to the same problems. Perhaps Julia builds into the language the ultimate performance solutions, so ok, then, for example, wait until there are N different web frameworks, and there you will find your silos. Python has many approaches to making things fast, which is why there are silos. /shrug

> I switched already

I probably would too if I was still a grad student.

> why you hang out in a thread about Julia

I was perusing whilst waiting for my Python code to complete, when I saw someone suggesting Python is already quite OK, catching some hate. I'm more than happy for the Julia community, but I think it's helpful to get exchanges accurate and critical.

Re: Julia 1.0

#408

Post by marmaduke »

ChrisRackauckas wrote: post #360
marmaduke wrote: post #338

Earlier quoted context omitted.

SciPy solvers are mostly interfaces to existing established solvers, and I’ve not had any problems with them. We’ve also used PyDSTool without problem, and it appears to support Python 3, https://github.com/robclewley/pydstool/blob/master/README.rs... If you think these are poorly maintained you should see XPPAUT, a tool still quite widely used.

SciPy's solvers cannot handle events which are nearby, most return codes aren't documented, you cannot run the wrapped solvers in stepping control mode, you cannot give it a linear solver routine, etc. So it wraps established solvers but still only gives the very basic solving feature of it, and most of the details that made the methods famous are not actually available from SciPy's interface. And it wasn't Python 3…

You mentioned Python 3, not me. Btw, I did look through your DE packages, and they are definitely an amazing contribution not seen in Python; I've recommend to colleagues.

Re: Julia 1.0

#409

Post by ScottBurson »

simonbyrne wrote: post #328
jacobolus wrote: post #324

Earlier quoted context omitted.

This depends on what the behavior is of the division operator. People should use floor division, then implementing modulo is easy. I don’t know what is available as chip instructions, I could certainly believe hardware designers made the wrong choice.

I take the opinion that people should have the option to round division (and compute remainders) however they want: down, up, to zero, to nearest (with different options for breaking ties). https://github.com/JuliaLang/julia/issues/9283

Common Lisp has all four division operators ('floor', 'ceiling', 'truncate' (equivalent to what most languages consider division), and 'round') [0], and both 'mod' and 'rem' [1].

[0] http://www.lispworks.com/documentation/lw50/CLHS/Body/f_floo...

[1] http://www.lispworks.com/documentation/lw50/CLHS/Body/f_mod_...

Re: Julia 1.0

#410

Post by ptx »

kgwgk wrote: post #73
peatmoss wrote: post #62

Earlier quoted context omitted.

> 1-based indexing now seems like a poor choice since Julia has become something more than the original mission of a better MATLAB or Octave. It’s a, admittedly, minor tragedy of Julia’s success. I’m curious as to why this is a problem outside numerical computing. From my perspective, this is consistent with a long history of mathematics dealing with matrices that predates electronic computers. 0-based arrays are pop…

> the technical reason we started counting arrays at zero is that in the mid-1960’s, you could shave a few cycles off of a program’s compilation time on an IBM 7094. The social reason is that we had to save every cycle we could, because if the job didn’t finish fast it might not finish at all and you never know when you’re getting bumped off the hardware because the President of IBM just called and fuck your thesis,…

Very strange article. The first part builds suspense for the great truth that's going to be revealed by the author's extensive research, insisting that it's not something as simple as pointer arithmetic.

Then the second parts quotes the creator of BCPL, revealing – a-HA! – what we already knew, that the array is a pointer to the first element and the index is the offset from the pointer, and that's why it's zero.

(And after that it veers of into complaining about how the papers documenting this history cost too much money, so they didn't actually read them.)

Post reply on HN