Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Statistics with Julia [pdf]

people.smp.uq.edu.au

Post reply on HN
Print view View original

61–70 of 136 posts

Re: Statistics with Julia [pdf]

#61

Post by mruts »

kbd wrote: post #54
mruts wrote: post #14

Julia is everything python could have been, and much more. I'm stuck with python right now as a lot of people in the data science/ML community are, but it's becoming increasingly viable to use Julia for "real" work. The Python-Julia interop story is pretty strong as well, which allows you to (somewhat) easily convert pandas/pytorch/sklearn code into Julia using Python wrappers. Julia has some unconventional things in…

I can't believe I'm jumping into the inevitable 1-based indexing discussion, but I'm surprised to see you say that one-based indexing results in "less "+ 1" or "- 1" things in your code". Most arguments I've seen come out to "it's fine" (certainly) or "it's more comfortable for mathematicians" (which I can't speak to). Besides Dijkstra's classic paper[1] showing why 0-based indexing is superior, in practice I find my…

The classic example is getting the last element of an array. With 1-based indexing the length of the array is the index of the last element. It has a nice symmetry to it.

Also I find it elegant that for 1-indexing that the start and end value for slices are both inclusive, instead of the first one being inclusive and the last being exclusive.

Also, isn’t it just weird that the index of an element is one less than it’s “standard” index? Like if I take the first nth elements of a list, it would stand to reason that the nth element should be the last element, right?

The reason for zero indexing is historical, related to pointer offsets. I don’t think anyone chose them to be easier for people. They just made them that way because it maps closer to how contiguous values in arrays are accessed.

Also, with 1-indexing I can multiply numbers by arrays and get reasonable offsets. 3 x 1 is three, so I would get the third element of the list. But with 0-indexing, I have 0 x 3 which gives me the same element, clearly inconsistent.

There are some good reasons for 0-indexing and I have been using it in every language for my entire career. The amount of code I’ve written in Julia is marginal compared to my 0-indexing experience, so I might be missing something.

One nice this about 0-indexing is that I can slice a list in half with the same midpoint. For example a Python array with 10 elements:

fst, snd = arr[0:5], arr[5:10]

A little nicer than:

fst, snd = arr[1:5], arr[6:10]

Though you could have inclusive slices with 0-indexing, but it would be inconvenient and suffer from the same problem as 1-indexing.

Re: Statistics with Julia [pdf]

#62

Post by jointpdf »

psychometry wrote: post #50
jointpdf wrote: post #20

This looks like a good reference for the fundamentals of both statistics and Julia, as claimed. I have a small critique, since the authors asked for suggestions. The format for the code samples goes like (code chunk —> output/plots —> bullet points explaining the code line-by-line). This creates a bit of a readability issue. The reader will likely follow a pattern like: (Skim past the code chunk to the explanation —>…

Not using PDF would be a good start. Bookdown texts tend to be good for mixed code/prose sections.

Yeah but PDF itself isn’t really the problem. Bookdown is nice, but if you’re using Bookdown then you’re using RMarkdown, so you can easily output the same .Rmd file as HTML/PDF/Reveal.js/EPub/etc. I’m trying to find well-executed examples or templates for what I have in mind (two column layout of code/text, maybe in landscape orientation for better spacing), but I’m drawing up blanks so far. Specifically I’m looking for either LaTeX or Reveal.js packages/templates for this.

Re: Statistics with Julia [pdf]

#63

Post by eigenspace »

j88439h84 wrote: post #53
Buttons840 wrote: post #51

Earlier quoted context omitted.

It's a bit more nuanced than that. It's "as fast" without having to write any C. I tried to recreate something like AlphaGo in Python using Keras, I never got the learning to work (probably because I was impatient and training on a laptop CPU), but a lot of the CPU time was simply being spent on manipulating the board state. So I ported my "Board" object to Rust, and it was a lot faster. Things like counting libertie…

What's the nuance? It's much faster?

The nuance is that for someone who mainly just calls functions from packages, they probably won’t notice any real speed difference since performance sensitive packages in python and R are typically written in C or C++. Additionally, there are various tools like Numba for accelerating Python code that will make certain restricted subsets of Python just as fast (or sometimes faster) than Julia.

However, as soon as you try to do something a bit more complicated then you’ll notice the speed and flexibility differences.

Re: Statistics with Julia [pdf]

#64

Post by anthony_doan »

jointpdf wrote: post #31
bdod6 wrote: post #28

Can someone explain how this is more powerful than someone use an Python/R based workflow? E.g., I currently use a combination .ipynb, python scripts, and RStudio and this feels like it covers everything I need for any data science project.

I think Julia has a cleaner focus on scientific and mathematical computing than either R or Python (both for performance and understanding). i.e. the language is designed in such a way that corresponds more directly to mathematical notation and ways of thinking. If you’ve been in a graduate program that’s heavily mathematical, where you spend equal time doing pen and paper proofs and hacking together simulations and…

> R is too quirky to fulfill this niche

I'd like to offer a counter point or add on to this.

It's quirky enough to have many packages backed by some expert statistician.

I hope Julia get to be successful in this regard too.

Re: Statistics with Julia [pdf]

#65

Post by Someone »

cshenton wrote: post #57
j88439h84 wrote: post #53

Earlier quoted context omitted.

What's the nuance? It's much faster?

Julia code gets compiled native via LLVM so it is about as fast as other natively compiled languages.

Not making any claims about Julia, but “gets compiled native via LLVM” doesn’t imply ”is about as fast as other natively compiled languages”

For example, a straightforward Python-to-LLVM compiler would generate code with every variable being a PyObject (https://docs.python.org/3/c-api/structures.html) instance, and “switch(obj.ob_type)” equivalents that would require a “sufficiently advanced compiler” to get to equivalent speed as, say, C.

Re: Statistics with Julia [pdf]

#66

Post by chakerb »

I was going to ask is there any Kindle version of this, then I skimmed over the book, and I don't think it will be readable on a Kindle. And even if it does, the reading experience will definitely be inferior.

Re: Statistics with Julia [pdf]

#67

Post by eigenspace »

j88439h84 wrote: post #60
mruts wrote: post #42

Earlier quoted context omitted.

Julia has static typing. I wouldn’t go that far and say Python is suitable for large programs. It’s clearly not. Working on a large python code base is hell.

> Julia has static typing. Julia has its own type system, which doesn't conform to the traditional static/dynamic divide. But AFAIK it doesn't have a compile-time type checker like mypy to help me catch type errors early.

Yeah, Julia’s type system is really just not designed for the sort of error catching patterns people seem to want static type systems for. Instead, Julia’s type system is designed for enabling multiple dispatch which I’d argue is a much greater boon than the dubious claims of error catching due to static type systems.

However, you can simulate static typing with the @inferred macro from Test.jl. It will throw an error if all types are not infer able ahead of time which is essentially static typing.

Re: Statistics with Julia [pdf]

#68

Post by improbable22 »

superdimwit wrote: post #46
cwyers wrote: post #36

Earlier quoted context omitted.

In R, most of the high performance code isn't written in R, it's written in Fortran or C or C++ (R has really good C++ integration via Rcpp). Python has something similar. The value prop of Julia is supposed to be that you have a language flexible enough to do the high-level stuff you'd normally do in R/Python, plus the ability to write high-performance code without having to drop into another language. I remain skep…

I think if you're just plugging together reasonably "vanilla" components from python / R libraries, and only using vectorised operations, those languages are fine and you can get away with using vectorised libraries wrapping C++. The moment Julia shines is when your workloads can't be phrased by stringing together the limited set of vectorised verbs that python / r libraries give you: this is anything stateful and lo…

One more advantage to these libraries being written in Julia is that, if they are almost do what you need but not quite, it's often pretty easy to reach inside and patch the function which needs changing. You already speak the language and don't need to stop the world to do this. The barrier to doing this to (say) numpy is just much higher.

Re: Statistics with Julia [pdf]

#69

Post by Merrill »

In section "1.2 Setup and Interface" there is a very short description of the REPL and how it can be downloaded from julialang.org, as well as a much longer description of JuliaBox and how Jupyter notebooks can be run from juliabox.com for free.

Although JuliaBox has been provided for free by Julia Computing, there has been discussion that this may not be possible in the future. However, Julia Computing does provide a distribution of Julia, the Juno IDE, and supported packages known as JuliaPro for free.

For new users, would the free JuliaPro distribution be a good alternative to JuliaBox and/or downloading the REPL and kernal from julialang.org?

Re: Statistics with Julia [pdf]

#70

Post by ddragon »

aurelian15 wrote: post #16
SolarNet wrote: post #12

Earlier quoted context omitted.

This is a core part of the design. It's part of why Julia is so useful for scientific computing, where one often has a large job that will require a lot of processing time, such that it is worth it to do an intensive JIT cycle every-time. And part of that is the analysis to take python-esque code and turning it into C levels of performance.

I just looked into Julia (1.1) for scientific use (simulation of very simple dynamical systems) a few days ago. I have to admit that by the end of the day I was surprisingly frustrated. I felt that type annotations were insufficient (one of the reasons to move away from Python); in particular, I didn't find a way to specify statically sized array types as you can do with Eigen, a feature that I find incredibly useful…

>since I'm relying on fast editing/execution cycles

While you can't do it from the shell very well right now (rerunning the program at each step like you would with an interpreted language), that kind of fast cycle is something very common in Julia development but with a particular REPL based workflow [1] in which you use a tool like Revise.jl [2] to automatically update the definition whenever you save a file in your project (the only restriction is that it doesn't automatically updates new type definitions) and directly interacting with the program in the REPL. This way it will only recompile what you just altered, and it's very fast to actually run the code. Other interesting tools are Rebugger.jl (debugger for the REPL) [3] and OhMyREPL (coloring for the REPL) [4], which you can add to your startup.jl to always automatically load them.

[1] https://docs.julialang.org/en/v1/manual/workflow-tips/index....

[2] https://github.com/timholy/Revise.jl

[3] https://github.com/timholy/Rebugger.jl

[4] https://github.com/KristofferC/OhMyREPL.jl

Post reply on HN