Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Differentiable Programming Mega-Proposal

forums.swift.org

Post reply on HN
Print view View original

21–30 of 78 posts

Re: Differentiable Programming Mega-Proposal

#21

Post by taliesinb »

If you are interested in following along, the Swift for TensorFlow team has a design meeting every Friday. The meetings are live-streamed and anyone can join. I recommend them if you are curious and want to hear more about the challenges and opportunities! A recent talk had Jeremy Howard (of fasti.ai fame) present the full-featured deep learning API he has designed on top of Swift for TensorFlow.

You can find out more on the mailing list https://groups.google.com/a/tensorflow.org/forum/m/#!forum/s...

Re: Differentiable Programming Mega-Proposal

#22

Post by abeppu »

taliesinb wrote: post #18
wenc wrote: post #13

I'm curious (for folks in the know): how does differentiable programming handle non-differentiable points? Can it detect non-differentiable/non-smooth functions? Non-smooth functions like abs(), max(), min() have points where derivatives do not exist. ReLU functions are non-differentiable at their hinge points. Disjoint IF-THEN-ELSE conditions are discontinuities in the function space, and are traditionally handled i…

While the gradients don’t technically exist, you usually use the limiting value from one side (e.g for the commonly used ReLU activation, you can use either 0 or 1). This is justified by the observation that randomly initialized networks are extremely unlikely to encounter these particular values without some kinder of deeper conspiracy happening. You could put this on a mathematical footing by saying the set of non…

> randomly initialized networks

But "networks" here, you're thinking of ANNs, yes?

But in the context of proposing differential programming as an addition to a general purpose language (and where the proposal explicitly brings up a bunch of cases outside of deep learning), is it fair to justify behavior based on what makes sense in a popular but narrow application?

Re: Differentiable Programming Mega-Proposal

#23

Post by taliesinb »

hsaliak wrote: post #14

How does this compare to Jax? https://github.com/google/jax Why do this in the language instead of a library?

Probably the main justification is that the analysis and transformation steps needed to compute the vjp and jvp pullbacks of a function (which correspond to reverse- and forward-mode automatic differentiation) require enough of the other machinery of a compiler that they are best done WITHIN a compiler. Then other things become quite natural, too, like producing the tangent vector versions of data structures like tuples and maps!

Moreover, a statically typed language like Swift is a much better starting point for this kind of effort than Python. Array shapes and dimensions are already a type system - you might as well go the whole distance and get all the other safety, readability, and efficiency benefits!

PS shout out for named array axes as the future of array-based (and hence differentiable) programming... see http://nlp.seas.harvard.edu/NamedTensor for a good rationale

Re: Differentiable Programming Mega-Proposal

#24

Post by __erik »

adamnemecek wrote: post #3
__erik wrote: post #2

I'm really excited for this. I'm unaware of any mainstream language with first class support for differentiation, I think its going to be really interesting to see what people use it for out side of ML.

Julia is in the same space.

Julia is great but it doesnt play in the domain of apps and servers

Re: Differentiable Programming Mega-Proposal

#25

Post by mlevental »

taliesinb wrote: post #23
hsaliak wrote: post #14

How does this compare to Jax? https://github.com/google/jax Why do this in the language instead of a library?

Probably the main justification is that the analysis and transformation steps needed to compute the vjp and jvp pullbacks of a function (which correspond to reverse- and forward-mode automatic differentiation) require enough of the other machinery of a compiler that they are best done WITHIN a compiler. Then other things become quite natural, too, like producing the tangent vector versions of data structures like tup…

is there something in the proposal that addresses named tensors? ctrlf doesn't find anything.

Re: Differentiable Programming Mega-Proposal

#26

Post by taliesinb »

abeppu wrote: post #22
taliesinb wrote: post #18

Earlier quoted context omitted.

While the gradients don’t technically exist, you usually use the limiting value from one side (e.g for the commonly used ReLU activation, you can use either 0 or 1). This is justified by the observation that randomly initialized networks are extremely unlikely to encounter these particular values without some kinder of deeper conspiracy happening. You could put this on a mathematical footing by saying the set of non…

> randomly initialized networks But "networks" here, you're thinking of ANNs, yes? But in the context of proposing differential programming as an addition to a general purpose language (and where the proposal explicitly brings up a bunch of cases outside of deep learning), is it fair to justify behavior based on what makes sense in a popular but narrow application?

It’s a good question what the plans are for DP languages to handle situations where non-differentiability shouldn’t be ignored.

For sensitivity analysis it might be disastrous to conclude that an output is sensitive to an input when it is actually not, merely because an intermediary ReLU hit 0, for example.

A conservative approach could be to define versions of the relevant functions that threw exceptions at such points, or that also calculated the trusted margin of the resulting gradients; non-differentiability would then produce a zero trust margin.

Re: Differentiable Programming Mega-Proposal

#27

Post by mlevental »

abeppu wrote: post #20
thedataangel wrote: post #16

Earlier quoted context omitted.

My understanding of automatic differentiation (AD) is that it's only really possible at the compiler level, since you need the ability to interpret and manipulate function definitions themselves. Certainly, no library would be able to offer the same level of guarantees telling you if you've done it wrong, nor the same opportunities for optimisation.

It's certainly not the case that autodiff is only possible at the compiler level. I've implemented forward mode (via dual numbers) and reverse mode (via tapes / wengert nodes) autodiff in libraries before.

notice the qualifier "really". obviously you can implement autodiff kind of outside the complier since pytorch and tensor flow exist. but those implementations constrain you to a select few compositions (please no comments on Turing completeness with just loops and conditionals). so for example if statements in pytorch are not differentiable (they might have piece wise continuous derivates) because pytorch doesn't actually trace the ast. I'm not a languages expert but outside of implementing in the compiler I imagine you'd need a homoiconic language to implement as a library.

Re: Differentiable Programming Mega-Proposal

#28

Post by taliesinb »

adamnemecek wrote: post #3
__erik wrote: post #2

I'm really excited for this. I'm unaware of any mainstream language with first class support for differentiation, I think its going to be really interesting to see what people use it for out side of ML.

Julia is in the same space.

Yup, work is continuing apace with Julia’s next-gen Zygote project.

Also, from the GP’s thought about applications beyond DL, my favorite examples so far are for model-based RL [1] and Neural ODEs [2]

[1] https://fluxml.ai/2019/03/05/dp-vs-rl.html

[2] https://julialang.org/blog/2019/01/fluxdiffeq

Re: Differentiable Programming Mega-Proposal

#29

Post by mlevental »

abeppu wrote: post #22
taliesinb wrote: post #18

Earlier quoted context omitted.

While the gradients don’t technically exist, you usually use the limiting value from one side (e.g for the commonly used ReLU activation, you can use either 0 or 1). This is justified by the observation that randomly initialized networks are extremely unlikely to encounter these particular values without some kinder of deeper conspiracy happening. You could put this on a mathematical footing by saying the set of non…

> randomly initialized networks But "networks" here, you're thinking of ANNs, yes? But in the context of proposing differential programming as an addition to a general purpose language (and where the proposal explicitly brings up a bunch of cases outside of deep learning), is it fair to justify behavior based on what makes sense in a popular but narrow application?

he/she addressed that - the points at which the function isn't differentiable has measure zero. besides this isn't some kind of new hack - one sided limits (and therefore derivatives) were invented exactly for such cases (min, max, abs) and have been used by mathematicians probably since just about when calculus was invented.

Re: Differentiable Programming Mega-Proposal

#30

Post by mlevental »

krapht wrote: post #15
thedataangel wrote: post #5

This is actually huge. I saw a proof of concept of something like this in Haskell a few years back, but it's amazing it see it (probably) making it into the core of a mainstream language. This may let them capture a large chunk of the ML market from Python - and hopefully greatly improve ML apis while they're at it.

Huh? Nobody is writing numerically intensive libraries in Python. Clearly this language proposal is taking aim at C++ and Fortran. Even if this caused TensorFlow & others to rewrite everything in Swift, people would write Python bindings to it and keep using Python. I'll get excited if Apple actually merges this into Swift. It's a niche feature that their compiler team will need to maintain forever. I actually have b…

lol this is literally by the group that's rewriting tensorflow in Swift https://www.tensorflow.org/swift so you're off on the intention here in that it is exactly taking aim at python as the main data ecosystem language.
Post reply on HN