Live data from Hacker News

Statistics with Julia [pdf]

people.smp.uq.edu.au

81–90 of 136 posts

Re: Statistics with Julia [pdf]

#81
post #80

I'd really recommend anyone doing mildly numerical / data-ey work in python to give Julia a patient and fair try. I think the language is really solidly designed, and gives you ridiculously more power AND productivity than python for a whole range of workloads. There are of course issues, but even in the short time I've been following & using the language these are being rapidly addressed. In particular: generally le…

Are all the plotting/visualization options still half baked?

Yes, they are. Slow and hardly as expressive or rich as python/r counterparts.

Re: Statistics with Julia [pdf]

#82

Earlier quoted context omitted.

It's a bit more nuanced than that. It's "as fast" without having to write any C. I tried to recreate something like AlphaGo in Python using Keras, I never got the learning to work (probably because I was impatient and training on a laptop CPU), but a lot of the CPU time was simply being spent on manipulating the board state. So I ported my "Board" object to Rust, and it was a lot faster. Things like counting libertie…

What's the nuance? It's much faster?

Since neither of the others mentioned it. The nuance is the type system + multimethods. It's a gradually typed system that fully specializes code where it can (aided by the expressive power of multi-methods), and hence with some careful (or overkill) placement of types (and multimethods) it's easy to get large performance boosts with minor edits to one's code (rather than porting the whole thing to C which is the python strategy). But the first pass can still be like one is writing python code, with no typing at all (and sometimes you will get lucky, or be very smart, and that will be fast through specialization without extra work).

As a brief demonstration I can write:

foo(a, b, c) = (a + b) * c

And when I call it on integers, it emits only the necessary integer assembly, and when I call it on floats only the necessary float assembly, and when I broadcast it across vectors it emits SSE assembly. It's only when it can't prove the incoming types that it emits any sort of dynamic type code. It's also possible for the calling function to be ignorant of the types too, and so on, until a user decides to pass in an integer or a float, and all of the code is specialized to be as fast as possible.

Re: Statistics with Julia [pdf]

#83
post #77
post #61

Earlier quoted context omitted.

The classic example is getting the last element of an array. With 1-based indexing the length of the array is the index of the last element. It has a nice symmetry to it. Also I find it elegant that for 1-indexing that the start and end value for slices are both inclusive, instead of the first one being inclusive and the last being exclusive. Also, isn’t it just weird that the index of an element is one less than it’…

> The classic example is getting the last element of an array. Good point, in Python I don't notice that the last element is arr[len(arr)-1] because Python provides arr[-1]. I think in general your point is that it's natural for the nth element to be arr[n]. > The reason for zero indexing is historical, related to pointer offsets. There is that, but Dijkstra's paper makes the case from first-principles that the close…

Dijkstra's write-up is full of subjective aesthetic judgements that certain things are ugly. I personally don't find `1:0` for an empty sequence to be ugly, and I do find using `1:1` to refer to an empty sequence and `1:2` to refer to the sequence `{1}` to be ugly. I would encourage everyone to read over his reasoning and see if you agree with his aesthetic judgements.

Re: Statistics with Julia [pdf]

#84
post #80

I'd really recommend anyone doing mildly numerical / data-ey work in python to give Julia a patient and fair try. I think the language is really solidly designed, and gives you ridiculously more power AND productivity than python for a whole range of workloads. There are of course issues, but even in the short time I've been following & using the language these are being rapidly addressed. In particular: generally le…

Are all the plotting/visualization options still half baked?

I've found Plots.jl and PyPlots.jl to work well for most basic things, despite not always being entirely pleasant to use, for example the compilation time issue, but this should hopefully improve. The only real problem I had is that these are not quite sufficient for plots to be published in a paper, many visual tweaks you might want are broken or terribly documented, and I have to just use matplotlib or R. It is generally great for jupyter notebooks though. I see the current deficiencies as highlighting just how much work went into matplotlib and others to get where they are today (and even mpl is in some ways still lacking, for example 3D surfaces and meshes). It is unfortunate though, as plotting is a core functionality for their main target of computational science. But to answer your question, mostly yes. Everything seems to be slowly improving though.

Re: Statistics with Julia [pdf]

#85

I'd really recommend anyone doing mildly numerical / data-ey work in python to give Julia a patient and fair try. I think the language is really solidly designed, and gives you ridiculously more power AND productivity than python for a whole range of workloads. There are of course issues, but even in the short time I've been following & using the language these are being rapidly addressed. In particular: generally le…

>you can typically just write down the code you want to write, rather than being forced to find a library that wraps a C/C++ implementation like in python/r.

I don't think this is really a feature. It's nice that you can write more performant code in Julia directly and don't need to wrap lower level languages, without question, but the lack of libraries or library features is not a good thing. It's always better to use a general purpose library that's been battle tested than to write your own numerical mathematics code (because bugs in numerical code can take a long time to get noticed)

For specialized scientific computing applications, which would normally be written in C/C++, I would absolutely look into using Julia instead (though not sure what the openmp/mpi support is like). But I would also recommend against rolling your own numerical software unless you need to

Re: Statistics with Julia [pdf]

#86
post #81
post #80

Earlier quoted context omitted.

Are all the plotting/visualization options still half baked?

Yes, they are. Slow and hardly as expressive or rich as python/r counterparts.

One can use matplotlib in Julia by PyCall'ing it. So it is at least as good as anything else.

Re: Statistics with Julia [pdf]

#87
post #18

Julia looked interesting to me, so I tried 1.0 after it came out. I have a oldish laptop (fine for my needs), and every time I tried to do seemingly anything, it spent ~5 minutes recompiling libraries or something. So I've been waiting newer versions that hopefully stop doing that, or for me to buy a better computer.

Yes, this is ones of my problems with Julia. It seems to be optimized for long runs and REPL/notebook usage. Take, for example, a simple program that creates a line plot ( https://docs.juliaplots.org/latest/tutorial/ ): using Plots x = 1:10 y = rand(10) plot(x, y) After installing the package, the first run has to precompile(?), and subsequent runs use the package cache. But ~25 s to create a simple plot is incredibl…

The time to second plot will be a few milliseconds, in the same process - in the same Julia session. So, while the time to first plot is frustrating, it is ok if your interactive session times are longer.

Of course, we continue to work on improving compile times. About half of the time is spent in LLVM compilation, which has actually become slower over time.

Re: Statistics with Julia [pdf]

#88
post #9
post #8

Earlier quoted context omitted.

You can effectively bookmark submissions by using the "favorite" link or just upvoting. The submission will show up in your profile under "favorite submissions" or "upvoted submissions", respectively.

In addition, I hear that modern browsers support a ground-breaking functionality called "Bookmarks".

To be fair, many modern social media sites break bookmarks.

Re: Statistics with Julia [pdf]

#89

I'd really recommend anyone doing mildly numerical / data-ey work in python to give Julia a patient and fair try. I think the language is really solidly designed, and gives you ridiculously more power AND productivity than python for a whole range of workloads. There are of course issues, but even in the short time I've been following & using the language these are being rapidly addressed. In particular: generally le…

>you can typically just write down the code you want to write, rather than being forced to find a library that wraps a C/C++ implementation like in python/r. I don't think this is really a feature. It's nice that you can write more performant code in Julia directly and don't need to wrap lower level languages, without question, but the lack of libraries or library features is not a good thing. It's always better to u…

I don't just think it's a feature, I think it's a killer feature.

You are much less likely to reinvent the wheel if you can add your one critical niche feature / bugfix to an existing library. In python, learning C and C build systems and python's C API are gigantic barriers to doing that.

More importantly, if every fast data manipulation needs to be written in C, a few of them can be profitably shared, but you need more than a few of them. Pretty soon you wind up with a giant dumping ground of undiscoverable API bloat. See: pandas.

Re: Statistics with Julia [pdf]

#90
post #83
post #77

Earlier quoted context omitted.

> The classic example is getting the last element of an array. Good point, in Python I don't notice that the last element is arr[len(arr)-1] because Python provides arr[-1]. I think in general your point is that it's natural for the nth element to be arr[n]. > The reason for zero indexing is historical, related to pointer offsets. There is that, but Dijkstra's paper makes the case from first-principles that the close…

Dijkstra's write-up is full of subjective aesthetic judgements that certain things are ugly. I personally don't find `1:0` for an empty sequence to be ugly, and I do find using `1:1` to refer to an empty sequence and `1:2` to refer to the sequence `{1}` to be ugly. I would encourage everyone to read over his reasoning and see if you agree with his aesthetic judgements.

I’m constantly baffled by the way people hold that paper up as some sort of objective proof that 0 based indexing is superior.
Post reply on HN