51–60 of 83 posts
Julia is a great language, but I'm still waiting for someone to create a Julia fork that uses 0-based indexing.
They should compromise and use 0.5-based indexing.
For those unaware of what automatic differentiation is: It's a close-to-magical tool which turns code for evaluating a function f into code for evaluating its derivative f'. It uses a special sort of invented numbers which square to zero even though they are not themselves zero. Here is one of the many tutorials on automatic differentiation: https://pizzaseminar.speicherleck.de/automatic-differentiati...
How useful is it in practice?
Earlier quoted context omitted.
Julia was developed by scientists that used math software like MATLAB. Most mathematics software uses 1-based indexing. (Previous comment about this.[1]) >with 1-based indexing : T[(h-1) * W * C+(w-1) * C+c] Your example showing how cumbersome and ugly it is to subtract 1 -- isn't how the intended audience uses mathematics software. Instead, they would use idiomatic multidimensional access with commas or multi-bracke…
My argument was that in situations where the difference between 0 and 1 based indexing makes a difference it's ususally 0-based indexing that leads to simpler code.
This isn't common in 1-based indexing languages. Very common in 0-based indexing languages[3].
If the 0-based indexing was objectively superior, wouldn't there be fewer instances of such bugs?
[1] https://en.wikipedia.org/wiki/Off-by-one_error
[2] https://en.wikipedia.org/wiki/Zero-based_numbering
[3] https://softwareengineering.stackexchange.com/questions/4289...
Earlier quoted context omitted.
Thid is really neat. Is this really used in practice? It seems to me that most of the AD frameworks used for deep learning implement the backward function that returns the jacobian for every initial function, and then chain those backward functions
Yes, definitely, there are even battle-tested implementations for Fortran available. Though I have never seen AD frameworks used in the production contexts for neural networks/backpropagation. As you say, the code for this seems to be mostly handrolled. Please take this negative statement with a grain of salt, I don't actually work in machine learning.
For those unaware of what automatic differentiation is: It's a close-to-magical tool which turns code for evaluating a function f into code for evaluating its derivative f'. It uses a special sort of invented numbers which square to zero even though they are not themselves zero. Here is one of the many tutorials on automatic differentiation: https://pizzaseminar.speicherleck.de/automatic-differentiati...
As magical as the chain rule of differentiation.
Earlier quoted context omitted.
Hear hear. Everyone that's making excuses for one based is wrong. You need a zero in your set of integers to form a Ring modulo N. I hereby declare everyone that wants 1 based indexing out of some naive sense of it being easier a scrub.
That's so absurd I hope it's satire. Humans instinctively reason about iteration in terms of the natural numbers, not the semantic purity of the underlying abstract algebra.
For those unaware of what automatic differentiation is: It's a close-to-magical tool which turns code for evaluating a function f into code for evaluating its derivative f'. It uses a special sort of invented numbers which square to zero even though they are not themselves zero. Here is one of the many tutorials on automatic differentiation: https://pizzaseminar.speicherleck.de/automatic-differentiati...
Thid is really neat. Is this really used in practice? It seems to me that most of the AD frameworks used for deep learning implement the backward function that returns the jacobian for every initial function, and then chain those backward functions
From this it should be clear why machine learning uses reverse-mode.
On the other hand, forward-mode is better for e.g. calculating the tangent of a high-dimensional curve (i.e. R -> R^n).
For those unaware of what automatic differentiation is: It's a close-to-magical tool which turns code for evaluating a function f into code for evaluating its derivative f'. It uses a special sort of invented numbers which square to zero even though they are not themselves zero. Here is one of the many tutorials on automatic differentiation: https://pizzaseminar.speicherleck.de/automatic-differentiati...
>close-to-magical As magical as the chain rule of differentiation.
Earlier quoted context omitted.
To clarify, your statement that, "It uses a special sort of invented numbers which square to zero even though they are not themselves zero." is not entirely true. In case someone else is looking at this, what the OP is referring to is dual numbers. That is one way to implement things, but not the most common way for fast tools. Fundamentally, automatic differentiation is the methodical application of the chain rule.…
Thank you for the very insightful background. Besides efficiency concerns (which I don't have a clue about), a disadvantage of the point of view using dual numbers is that, to my knowledge, it can only be used to derive the forward mode of automatic differentiation. Still I take pleasure in appreciating the slightly mystic aura of the dual numbers. :-)
But yes, dual numbers only do forward mode. However, slightly related is tracker types, like ReverseDiff.jl or Flux.jl's Tracker, where you essentially push a type forward through the code similarly to a dual number, and use this to build the computational graph which you then run backwards with the chain rule.
For those curious about Julia, I just found this: https://www.infoworld.com/article/3284380/data-science/what-... Close to C speed in a dynamic language? Seems pretty great on paper. Is this generally the case?