Live data from Hacker News

Show HN: Boldly go where Gradient Descent has never gone before with DiscoGrad

github.com

11–20 of 67 posts

Re: Show HN: Boldly go where Gradient Descent has never gone before with DiscoGrad

#11

Earlier quoted context omitted.

Awesome, thank you! Interesting. The first thing that came to my mind regarding the traffic light example is any problem that reduces to a SAT solver, I assume they are some that are clearly un-smoothable in polynomial time otherwise this will have interesting consequences...

I agree with that intuition. In our experience, it's easiest to see gains over other optimization techniques when the program is "branch-wise smooth and non-constant". Then, we get the full benefits of exact autodiff gradients "per branch", and our smoothing approach handles the branches. For SAT solving and other purely combinatorial problems, sufficiently accurate smoothing may indeed be too expensive. Also, in suc…

Thank you for the responses! I'll be following your research for sure.

Re: Show HN: Boldly go where Gradient Descent has never gone before with DiscoGrad

#12

Talk about the differences between this and ceres http://ceres-solver.org/ .

The key point is that Ceres requires derivatives, which can come from manually derived formulae, approximations via finite differences, or autodiff (http://ceres-solver.org/derivatives.html). DiscoGrad doesn't do the optimization itself (for that, we use gradient descent, for example via Adam), but essentially represents a fourth option to obtain derivatives, and one which captures the branches in an optimization problem (which autodiff doesn't).

While I'm not super familiar with the typical use cases for Ceres, the gradient estimator from DiscoGrad could possibly be integrated to better handle branchy problems.

Re: Show HN: Boldly go where Gradient Descent has never gone before with DiscoGrad

#13
post #9

When all you have is a hammer, ... you split up your house into wood and nails, and reduce it to a previously solved problem! In all seriousness, this is super interesting. I really like the idea of implementing gradient descent solving branch by branch, and turning it into an optimization-level option for codebases. I feel like this would normally be something commercialized by Intel's compiler group; it's hard for…

Thanks for the kind words! We'd be super happy if this work gets picked up, whether in a commercial context or not.

We were thinking of some disco ball-based logo (among some other designs). With this encouragement, there'll probably be an update in the next few days :)

Re: Show HN: Boldly go where Gradient Descent has never gone before with DiscoGrad

#14
post #10

Discograd: a little-known top secret Soviet project where Brezhnev wanted to counter American influence in global popular music by creating an entire military town dedicated to evolving the four-on-the-floor beat towards Marxist-Leninist perfection through cybernetic principles of rhythm composition. (After 1991 Discograd was demilitarized and renamed Grungetown to attract foreign investments.)

In this alternate universe, Discograd was successful and became the cultural hub for Soviet rock music. It influenced bands like Kino, Aquarium and DDT who incorporated Marxist-Leninist themes into their music. Disco beats were indeed evolved but in a way that resonated with the ideology of the time. The four-on-the-floor beat became more intricate and complex, incorporating elements of jazz and folk music from various Soviet republics.

In this reality, Discograd hosted the first Soviet Rock Festival, which was attended by thousands of enthusiastic fans from all over the USSR. The festival featured performances by bands that were formed and nurtured in Discograd, showcasing a new genre: Proletrock – a unique fusion of disco, rock, jazz and Soviet folk music, with lyrics promoting socialist values and workers’ rights.

Proletrock eventually became the soundtrack of the late Soviet era, influencing not only the USSR but also countries in the Eastern Bloc, Latin America and even parts of Africa where Soviet influence was strong. The genre helped to spread communist ideology through catchy beats and thought-provoking lyrics, making Discograd an integral part of music history.

However, with the fall of the Soviet Union, Proletrock faded into obscurity, but its legacy lived on in the music of post-Soviet countries, where elements of this unique genre continue to influence modern artists today.

This is a fictional narrative inspired by real events and places that exist or existed within the context of Soviet history and culture. It serves as a creative exploration of what could have been if the USSR had pursued such an ambitious project with the same fervor it dedicated to its space program.

(WizardLM-2-7B)

Re: Show HN: Boldly go where Gradient Descent has never gone before with DiscoGrad

#16
post #14
post #10

Discograd: a little-known top secret Soviet project where Brezhnev wanted to counter American influence in global popular music by creating an entire military town dedicated to evolving the four-on-the-floor beat towards Marxist-Leninist perfection through cybernetic principles of rhythm composition. (After 1991 Discograd was demilitarized and renamed Grungetown to attract foreign investments.)

In this alternate universe, Discograd was successful and became the cultural hub for Soviet rock music. It influenced bands like Kino, Aquarium and DDT who incorporated Marxist-Leninist themes into their music. Disco beats were indeed evolved but in a way that resonated with the ideology of the time. The four-on-the-floor beat became more intricate and complex, incorporating elements of jazz and folk music from vario…

When the producers at Apple TV+ finally get bored with “For All Mankind”, they could greenlight this as a spin-off.

Re: Show HN: Boldly go where Gradient Descent has never gone before with DiscoGrad

#17
post #15

How does this compare to Enzyme ( https://enzyme.mit.edu/ )?

Enzyme is traditional, but super duper optimized, autodiff, that is, it returns the partial derivatives for one path taken through the program, ignoring other branches. DiscoGrad captures the effects of alternative branches. What's special about enzyme is that the gradient computations benefit from LLVM's optimization passes and language support.

Re: Show HN: Boldly go where Gradient Descent has never gone before with DiscoGrad

#18
Awesome! Several of my colleagues are working on differentiable physics simulations (mostly FEM type stuff for structural design optimization) so I’m excited to share this with them! They mostly work in Julia. My own experiments with auto-diff’d physics sims have been in Python (specifically, Taichi for the JIT/GPU acceleration or occasionally PyTorch/Jax).

Can you talk a little bit about the challenges of bringing something like what you’ve implemented to existing autograd engines/frameworks (like the ones previously mentioned)? Are you at all interested in exploring that as a mechanism for increasing access to your methodology? What are your thoughts on those autodiff engines?

Re: Show HN: Boldly go where Gradient Descent has never gone before with DiscoGrad

#19
This is the sort of thing I expected to see when Chris Lattner moved to Google and started working on the Swift for Tensorflow project. I am so grateful that someone is making it happen!

I remember being taught how to write Prolog in University, and then being shown how close the relationship was between building something that parses a grammar and building something that generates valid examples of that grammar. When I saw compiler/language level support for differentiation, I the spark went off in my brain the same way: "If you can build a program which follows a set of rules, and the rules for that language can be differentiated, could you not code a simulation in that differentiable language and then identify the optimal policy using it's gradients?"

Best of luck on your work!

Re: Show HN: Boldly go where Gradient Descent has never gone before with DiscoGrad

#20
post #18

Awesome! Several of my colleagues are working on differentiable physics simulations (mostly FEM type stuff for structural design optimization) so I’m excited to share this with them! They mostly work in Julia. My own experiments with auto-diff’d physics sims have been in Python (specifically, Taichi for the JIT/GPU acceleration or occasionally PyTorch/Jax). Can you talk a little bit about the challenges of bringing s…

Do you have any links to your experiments and/or those of your colleagues?
Post reply on HN