Julia library for fast machine learning
1–10 of 40 posts
Re: Julia library for fast machine learning
#2Can someone informed give some suggestions as to compare/contrast to other tools at the intersection of probabilistic programming and deep learning? What are relative strengths and weaknesses vs edward or pyro?
Re: Julia library for fast machine learning
#3Can someone informed give some suggestions as to compare/contrast to other tools at the intersection of probabilistic programming and deep learning? What are relative strengths and weaknesses vs edward or pyro?
I've only used each of those a little.
Turing is a bit like Stan, JAGS, or BUGS, it's [closer to] a general probabilistic programming system with a Bayesian emphasis, although maybe less Bayesian in emphasis than those other DSLs. PyMC would be another comparison.
Edward is (or when I used it) coming more from a very general latent variable modeling framework, encompassing hidden variable models, and is less focused on Bayesian modeling per se, maybe more variational inferential approaches. It seems to be broadening in scope over time.
Pyro I think is further down the deep learning/NN path than Edward.
It's hard for me to ennumerate strengths/weaknesses, as they have different foci and are parts of different language ecosystems. It depends a bit on the use case. My own experience with each is such that I might use Turing, or move to something like TensorFlow; things like Edward or Pyro seem to occupy this intermediate ground that was difficult for me to utilize in the way I thought I might.
I've been excited by Turing, just to see a probabilistic programming framework like that in Julia. I think the expressiveness of Julia and it being native to that framework will be helpful.
Re: Julia library for fast machine learning
#4Can someone informed give some suggestions as to compare/contrast to other tools at the intersection of probabilistic programming and deep learning? What are relative strengths and weaknesses vs edward or pyro?
I've only used each of those a little. Turing is a bit like Stan, JAGS, or BUGS, it's [closer to] a general probabilistic programming system with a Bayesian emphasis, although maybe less Bayesian in emphasis than those other DSLs. PyMC would be another comparison. Edward is (or when I used it) coming more from a very general latent variable modeling framework, encompassing hidden variable models, and is less focused…
I've been excited by Turing, just to see a probabilistic programming framework like that in Julia. I think the expressiveness of Julia and it being native to that framework will be helpful.
I hope so too. But hasn't Julia's TF/Torch equivalent, Flux, had performance problems? That was the rumor I heard anyway, I haven't had the chance to use it myself.
Re: Julia library for fast machine learning
#5Earlier quoted context omitted.
I've only used each of those a little. Turing is a bit like Stan, JAGS, or BUGS, it's [closer to] a general probabilistic programming system with a Bayesian emphasis, although maybe less Bayesian in emphasis than those other DSLs. PyMC would be another comparison. Edward is (or when I used it) coming more from a very general latent variable modeling framework, encompassing hidden variable models, and is less focused…
I've been excited by Turing, just to see a probabilistic programming framework like that in Julia. I think the expressiveness of Julia and it being native to that framework will be helpful. I hope so too. But hasn't Julia's TF/Torch equivalent, Flux, had performance problems? That was the rumor I heard anyway, I haven't had the chance to use it myself.
It currently has problems with some classes of models on the GPU, but this is just due to memory management and is going to fixed soon. Julia is natively compiled, so doesn't use a tape, tracing, or for memory management simple ref counting (in this case not as good).
This is slated to be fixed in the short term with an abstract tracing framework which eliminates memory allocations. Given Julia's type information and ability to manipulate IR from third party programs, this is more general and powerful than pytorch's tracing system. Works on a larger set of code (the whole language), doesn't require actually running the code (abstract tracing), and allows for other program transforms/ analysis like compilation to XLA, shape inference, compile time errors, source to source prob programming : https://github.com/MikeInnes/Poirot.jl, and other things.
That's in the short term and should bring flux up to SOTA for speed (it already is on CPU). In the medium term, a general framework for optimizer passes will allow for more general compile time memory management.
Re: Julia library for fast machine learning
#6Earlier quoted context omitted.
I've been excited by Turing, just to see a probabilistic programming framework like that in Julia. I think the expressiveness of Julia and it being native to that framework will be helpful. I hope so too. But hasn't Julia's TF/Torch equivalent, Flux, had performance problems? That was the rumor I heard anyway, I haven't had the chance to use it myself.
It currently has problems with some classes of models on the GPU, but this is just due to memory management and is going to fixed soon. Julia is natively compiled, so doesn't use a tape, tracing, or for memory management simple ref counting (in this case not as good). This is slated to be fixed in the short term with an abstract tracing framework which eliminates memory allocations. Given Julia's type information and…
Note you can use this on Zygote to preallocate stuff: https://github.com/oxinabox/AutoPreallocation.jl . It doesn't support GPUs yet, mainly because the dev on it needs a GPU CI setup, but it should mostly just work (issue https://github.com/oxinabox/AutoPreallocation.jl/issues/10)
Re: Julia library for fast machine learning
#7The documentation / project page is very well-done, something unfortunately rare in the Julia ecosystem.
Re: Julia library for fast machine learning
#8Can someone informed give some suggestions as to compare/contrast to other tools at the intersection of probabilistic programming and deep learning? What are relative strengths and weaknesses vs edward or pyro?
Turing.jl is in an interesting spot because it is essentially a DSL-free probabilistic programming language. While it technically has a DSL of sorts given by the `@model` macro, anything that is AD-compatible can be used in this macro and since Julia's AD tools work on things written in the Julia language, this means that you can just throw code from other Julia packages into Turing and just expect AD-compatible things to work with Hamiltonian Monte Carlo and all of that. So things like DifferentialEquations.jl ODEs/SDEs/DAEs/DDEs/etc. work quite well with this, along with other "weird things for a probabilistic programming language to support" like nonlinear solving (via NLsolve.jl) or optimization (via Optim.jl, and I mean doing Bayesian inference where a value is defined as the result of an optimization). If you are using derivative-free inference methods, like particle sampling methods or variants of Metropolis-Hastings, then you can throw pretty much any existing Julia you had as a nonlinear function and do inference around it.
So while it's in some sense similar to PyMC3 or Stan, there's a huge difference in the effective functionality that you get by supporting a language-wide infrastructure vs the more traditional method of one-by-one adding features and documenting them. So while PyMC3 ran a Google Summer of Code to get some ODE support (https://docs.pymc.io/notebooks/ODE_API_introduction.html) and Stan has 2 built-in methods you're allowed to use (https://mc-stan.org/docs/2_19/stan-users-guide/ode-solver-ch...), with Julia you get all of DifferentialEquations.jl just because it exists (https://docs.sciml.ai/latest/). This means that Turing.jl doesn't document or doesn't have to document most of its features, but they exist due to composibility.
That's quite different from a "top down" approach to library support. This explains why Turing has been able to develop so fast as well, since it's developer community isn't just "the people who work on Turing", but it's pretty much the whole ecosystem of Julia. Its distributions are defined by Distributions.jl (https://github.com/JuliaStats/Distributions.jl), its parallelism is given by Julia's base parallelism work + everything around it like CuArrays.jl and KernelAbstractions.jl (https://github.com/JuliaGPU/KernelAbstractions.jl), derivatives come from 4 libraries, ODEs from etc. the list keeps going.
So bringing it back to deep learning, Turing currently has 4 modes for automatic differentiation (https://turing.ml/dev/docs/using-turing/autodiff), and thus supports any library that's compatible with those. It turns out that Flux.jl is compatible with them, so therefore Turing.jl can do Bayesian deep learning. In that sense it's like Edward or Pyro, but supporting "anything that AD's with Julia AD packages" (which soon will allow multi-AD overloads via ChainRules.jl) instead of "anything on TensorFlow graphs" or "anything compatible with PyTorch".
As for performance and robustness, I mentioned in a SciML ecosystem release today that our benchmarks pretty clearly show Turing.jl as being more robust than Stan while achieving about a 3x-5x speedup in ODE parameter estimation (https://sciml.ai/2020/05/09/ModelDiscovery.html). However, that's utilizing the fact that Turing.jl's composibility with packages gives it top notch support (I want to work with Stan developers so we can use our differential equation library with their samplers to better isolate differences and hopefully improve both PPLs, but for now we have what we have). If you isolate it down to just "Turing.jl itself", it has wins and losses against Stan (https://github.com/TuringLang/Turing.jl/wiki). That said, there's some benchmarks which indicate using the ReverseDiff AD backend will give about 2 orders of magnitude performance increases in many situations (https://github.com/TuringLang/Turing.jl/issues/1140, note that ThArrays is benchmarking PyTorch AD here) which would then probably tip the scales in Turing's favor. As for benchmarking against Pyro or Edward, it would probably just come down to benchmarking the AD implementations.
Re: Julia library for fast machine learning
#9The documentation / project page is very well-done, something unfortunately rare in the Julia ecosystem.
To add to that, even if Julia had excellent documentation literally everywhere and on every library, I wish there were better stack trace and meaningful error messages. Even if Julia performed 10x worse, this overlooked aspect of Julia would make up for it. It is rather unbelievable how much time I need to spend to figure out what's wrong with a particular piece of Julia code.
Founders of Julia - please focus on error messages. Some cool things from Rust: https://doc.rust-lang.org/edition-guide/rust-2018/the-compil...
Re: Julia library for fast machine learning
#10The documentation / project page is very well-done, something unfortunately rare in the Julia ecosystem.
To add to that, even if Julia had excellent documentation literally everywhere and on every library, I wish there were better stack trace and meaningful error messages. Even if Julia performed 10x worse, this overlooked aspect of Julia would make up for it. It is rather unbelievable how much time I need to spend to figure out what's wrong with a particular piece of Julia code. Founders of Julia - please focus on erro…
Lack of stack straces and poor error messages hasn't hindered the adoption of most compiled languages.
Naturally it is something to improve, but I bet that are still bigger fish to fry.