31–40 of 67 posts
What's the general type of use case where this default behavior is useless, and "non-discrete" (stochastic?) branching helps?
What do you mean by plain autodiff being mostly useless with normal/discrete branching? Wouldn't branches normally just be "ignored" by autodiff - each training sample being a different computational graph (but with parts in common) due to branching points, so the only effect of branching is which computational graph gets executed and backpropagated through? What's the general type of use case where this default beha…
The autodiff derivative of this is zero, wherever you evaluate it, so if you sample x and run your program on each x as in a classical ML setup, you'd be averaging over a series of zero-derivatives. This is of course not helpful to gradient descent. In more complex programs, it's less blatant, but the gist is that just averaging sampled gradients over programs (input-dependent!) branches yields biased or zero-valued derivatives. The traffic light optimization example shown on Github is a more complex example where averaged autodiff-gradients are always zero.
Does this do something similar or is it fancier?
This is the sort of thing I expected to see when Chris Lattner moved to Google and started working on the Swift for Tensorflow project. I am so grateful that someone is making it happen! I remember being taught how to write Prolog in University, and then being shown how close the relationship was between building something that parses a grammar and building something that generates valid examples of that grammar. Whe…
You can write down just about anything as a BUGS model for example, but “identifying the model” —- finding the uniquely best parameters, even though it’s a globally optimisation —- is often very difficult.
Gradient descent is significantly more limiting than that. Worth understanding MC. The old school is a high bar to jump.
Awesome! Several of my colleagues are working on differentiable physics simulations (mostly FEM type stuff for structural design optimization) so I’m excited to share this with them! They mostly work in Julia. My own experiments with auto-diff’d physics sims have been in Python (specifically, Taichi for the JIT/GPU acceleration or occasionally PyTorch/Jax). Can you talk a little bit about the challenges of bringing s…
We actually did some preliminary experiments with Taichi hoping to benefit from the GPU parallelization. I think generally, the world of autodiff tooling is in very good shape. For anything non-exotic, we just use JAX or Torch to get things done quickly and with good performance. Generally, integrating the ideas behind DiscoGrad into existing frameworks has been on our mind since day one, and the C++ implementation r…
I definitely hear you on this! As a grad student who is one of the only developers with actual professional dev xp in my lab, it can be brutal being tasked with turning academic spaghetti code into something semi-productionized/robust.
I used to replace strict Boolean conditionals with sigmoids in order to achieve continuous transfer for Bayesian change-point models. Does this do something similar or is it fancier?
On top of that, if the program branches on random numbers (which is common in simulations), that suffices for the maths to work out and you get an estimate of the asymptotic gradient (for samples -> infinity) of the original program, without any artificial smoothing.
So in short, I do think it is slightly fancier :)
Awesome! Several of my colleagues are working on differentiable physics simulations (mostly FEM type stuff for structural design optimization) so I’m excited to share this with them! They mostly work in Julia. My own experiments with auto-diff’d physics sims have been in Python (specifically, Taichi for the JIT/GPU acceleration or occasionally PyTorch/Jax). Can you talk a little bit about the challenges of bringing s…
Do you have any links to your experiments and/or those of your colleagues?
- Why do you think similar approaches never landed on jax? My guess is this is not that useful for the current optimizations in fashion (transformers)
- How would you convince jax to incorporate this?
This is the sort of thing I expected to see when Chris Lattner moved to Google and started working on the Swift for Tensorflow project. I am so grateful that someone is making it happen! I remember being taught how to write Prolog in University, and then being shown how close the relationship was between building something that parses a grammar and building something that generates valid examples of that grammar. Whe…
Somewhat related: is there autograd in python that uses AST analysis or something similar? The only method I’m familiar with uses tracer objects which have their gotchas. I’d like to compile a python function (at runtime) while writing its code as naturally as possible
https://docs.taichi-lang.org/docs/differentiable_programming