Alright, this looks neat, but I'm having a terrible time figuring out what's going on with the benchmark. Typically for AD, it's easiest to see four things: number of variables, time to compute function directly with no AD, time to compute function with AD enabled and calculate the gradient, ratio between these two numbers. Then, we run the same example with a varying number of variables to see how things scale. The advantage of reverse mode is that, theoretically, this ratio is fixed and bounded at 4 or 5 regardless of the number of variables. Realistically, this is much higher, but I think it's fine if it's somewhat between 10-40 times function evaluation as long as it scales. I can't figure out what their ratio is or whether or not it's appropriately scaling. Can anyone else?
Automatic Differentiation with Julia
41–50 of 83 posts
Re: Automatic Differentiation with Julia
#42For those curious about Julia, I just found this: https://www.infoworld.com/article/3284380/data-science/what-... Close to C speed in a dynamic language? Seems pretty great on paper. Is this generally the case?
surprisingly yes, it really is. Julia is smart about type inference, so it compiles a specialized version of a function as needed for whatever argument types actually get used.
the catch is, the first time you run a function there is often a noticeable compile time. but it’s cached after that.
other problems with Julia include a somewhat immature/unfinished set of libraries, in part because the language was constantly changing underneath people.
but now that 1.0 has been released, the language will be stable for a long time and you can expect that to improve quickly.
good language! it gets a lot of hype on HN but that’s because it is actually very nice.
Re: Automatic Differentiation with Julia
#43Julia is a great language, but I'm still waiting for someone to create a Julia fork that uses 0-based indexing.
Hear hear. Everyone that's making excuses for one based is wrong. You need a zero in your set of integers to form a Ring modulo N. I hereby declare everyone that wants 1 based indexing out of some naive sense of it being easier a scrub.
aesthetically i sort of agree, but i just want to type in equations from papers without having to do any error prone arithmetic — it is slightly less frustrating.
Re: Automatic Differentiation with Julia
#44For those unaware of what automatic differentiation is: It's a close-to-magical tool which turns code for evaluating a function f into code for evaluating its derivative f'. It uses a special sort of invented numbers which square to zero even though they are not themselves zero. Here is one of the many tutorials on automatic differentiation: https://pizzaseminar.speicherleck.de/automatic-differentiati...
To clarify, your statement that, "It uses a special sort of invented numbers which square to zero even though they are not themselves zero." is not entirely true. In case someone else is looking at this, what the OP is referring to is dual numbers. That is one way to implement things, but not the most common way for fast tools. Fundamentally, automatic differentiation is the methodical application of the chain rule.…
Thank you for the very insightful background.
Besides efficiency concerns (which I don't have a clue about), a disadvantage of the point of view using dual numbers is that, to my knowledge, it can only be used to derive the forward mode of automatic differentiation. Still I take pleasure in appreciating the slightly mystic aura of the dual numbers. :-)
Re: Automatic Differentiation with Julia
#45For those unaware of what automatic differentiation is: It's a close-to-magical tool which turns code for evaluating a function f into code for evaluating its derivative f'. It uses a special sort of invented numbers which square to zero even though they are not themselves zero. Here is one of the many tutorials on automatic differentiation: https://pizzaseminar.speicherleck.de/automatic-differentiati...
Thid is really neat. Is this really used in practice? It seems to me that most of the AD frameworks used for deep learning implement the backward function that returns the jacobian for every initial function, and then chain those backward functions
Yes, definitely, there are even battle-tested implementations for Fortran available.
Though I have never seen AD frameworks used in the production contexts for neural networks/backpropagation. As you say, the code for this seems to be mostly handrolled. Please take this negative statement with a grain of salt, I don't actually work in machine learning.
Re: Automatic Differentiation with Julia
#46Julia is a great language, but I'm still waiting for someone to create a Julia fork that uses 0-based indexing.
Hear hear. Everyone that's making excuses for one based is wrong. You need a zero in your set of integers to form a Ring modulo N. I hereby declare everyone that wants 1 based indexing out of some naive sense of it being easier a scrub.
That's so absurd I hope it's satire. Humans instinctively reason about iteration in terms of the natural numbers, not the semantic purity of the underlying abstract algebra.
Re: Automatic Differentiation with Julia
#47Earlier quoted context omitted.
Hear hear. Everyone that's making excuses for one based is wrong. You need a zero in your set of integers to form a Ring modulo N. I hereby declare everyone that wants 1 based indexing out of some naive sense of it being easier a scrub.
That's so absurd I hope it's satire. Humans instinctively reason about iteration in terms of the natural numbers, not the semantic purity of the underlying abstract algebra.
It's not absurd at all. In situations where you need to do modulo arithmetic on array indices 1-based indexing forces you to shift the offset by 1.
Re: Automatic Differentiation with Julia
#48For those unaware of what automatic differentiation is: It's a close-to-magical tool which turns code for evaluating a function f into code for evaluating its derivative f'. It uses a special sort of invented numbers which square to zero even though they are not themselves zero. Here is one of the many tutorials on automatic differentiation: https://pizzaseminar.speicherleck.de/automatic-differentiati...
How useful is it in practice?
Re: Automatic Differentiation with Julia
#49Earlier quoted context omitted.
That's so absurd I hope it's satire. Humans instinctively reason about iteration in terms of the natural numbers, not the semantic purity of the underlying abstract algebra.
It's not absurd at all. In situations where you need to do modulo arithmetic on array indices 1-based indexing forces you to shift the offset by 1.
Yes and that's something most programmers need to do almost never, so let's not make any important design decisions around that very specific case.
Re: Automatic Differentiation with Julia
#50For those curious about Julia, I just found this: https://www.infoworld.com/article/3284380/data-science/what-... Close to C speed in a dynamic language? Seems pretty great on paper. Is this generally the case?
As long as you write code that's type stable, yea. Type stable means that the compiler can deduce the types of the variables used in a function given the types of the arguments passed to the function. An example of code that's not type stable is code something like this: function test1(n) x = 0 for i = 1:n if i == 10 x = x + 0.1 else x = x + 1 end end return x end It isn't type stable because x starts out as an int b…
Julia is pretty good at dealing with this now though
julia> @btime test1(10^5)
147.427 μs (0 allocations: 0 bytes)
99999.1
julia> @btime test2(10^5)
88.472 μs (0 allocations: 0 bytes)
99999.1