Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Automatic Differentiation with Julia

blog.rogerluo.me

Post reply on HN
Print view View original

31–40 of 83 posts

Re: Automatic Differentiation with Julia

#31

Post by conjectures »

geezerjay wrote: post #18
lucozade wrote: post #15

Earlier quoted context omitted.

> This is too often the attitude of Julia people And it's a good attitude for the Julia people to have. Their target audience is not just (or even mainly) full time developers. They're targeting scientists, statisticians etc. As such, it makes sense for Julia to use conventions that are appropriate to their audience. This pervades the whole of Julia, not just indexing although the latter is exceptionally rich for thi…

> And it's a good attitude for the Julia people to have. Their target audience is not just (or even mainly) full time developers. They're targeting scientists, statisticians etc. I don't see how scientists or staticians would be unable to understand a basic knowledge like zero-based indexing. In fact, zero-based indexing already is widely used in basic areas such as series and sequences. Why are freshmen quite capabl…

The truth is that everyone who programs seriously at any level can deal with both 0- and 1- based indexing. It's just not that hard to get used to.

And science has lots of 1- based indexing: Matlab, Fortran, R, most math...

Re: Automatic Differentiation with Julia

#32

Post by shele »

geezerjay wrote: post #16
FridgeSeal wrote: post #7

Earlier quoted context omitted.

So, fun times: if the 1-based indexing throws you off that much, it is entirely straightforward to configure it to use 0 based indexing if you want (or any other kind of offset that you so desire) https://docs.julialang.org/en/latest/devdocs/offset-arrays/ Having said that, I encourage you to try out the 1-based indexing, as I think you might find a lot of things become surprisingly more intuitive.

> Having said that, I encourage you to try out the 1-based indexing, as I think you might find a lot of things become surprisingly more intuitive. Can you provide an example? It seems that the assertion only reflects a personal point of view, similar to the egg endianness discussion in Gulliver.

Counting is slightly easier: if you have one object, it has slot number 1, and if you have one more object (meaning you have two objects) that second object has slot number 2.

PS: I am joking here of course, but seriously, try counting things with your fingers starting with a fist, then one finger, then two.

Re: Automatic Differentiation with Julia

#33

Post by gct »

0-_-0 wrote: post #6

Julia is a great language, but I'm still waiting for someone to create a Julia fork that uses 0-based indexing.

Hear hear. Everyone that's making excuses for one based is wrong. You need a zero in your set of integers to form a Ring modulo N. I hereby declare everyone that wants 1 based indexing out of some naive sense of it being easier a scrub.

Re: Automatic Differentiation with Julia

#34

Post by mkborregaard »

0-_-0 wrote: post #29
newen wrote: post #27

Earlier quoted context omitted.

In Julia, you don't have to calculate linear indices yourself since you can create a reshaped view of a blob of memory in arbitrary number of dimensions. So this case doesn't come up. Also there are plenty of cases where 0-indexing is more awkward than 1-indexing. Actually reading your Dijkstra link will make that pretty clear. Reading it will also make it clear that the 0-indexing choice is pretty arbitrary and that…

Contrary to your summary, the quote makes a good case for 0-based indexing, both theoretical and practical. How can an argument based on reason become an argument from authority just because someone well known said it?

That is in fact how an argument becomes an argument from authority. Also, it has become an HN meme to post the Dijkstra comment every single time there's a post mentioning Julia. It's not like it's news to anyone.

Re: Automatic Differentiation with Julia

#35

Post by elcomet »

IngoBlechschmid wrote: post #28

For those unaware of what automatic differentiation is: It's a close-to-magical tool which turns code for evaluating a function f into code for evaluating its derivative f'. It uses a special sort of invented numbers which square to zero even though they are not themselves zero. Here is one of the many tutorials on automatic differentiation: https://pizzaseminar.speicherleck.de/automatic-differentiati...

Thid is really neat.

Is this really used in practice?

It seems to me that most of the AD frameworks used for deep learning implement the backward function that returns the jacobian for every initial function, and then chain those backward functions

Re: Automatic Differentiation with Julia

#36

Post by jasode »

0-_-0 wrote: post #17
red-tea wrote: post #11

Earlier quoted context omitted.

> or any other kind of offset that you so desire This is too often the attitude of Julia people. But zero-based indexing is not just some arbitrary offset. Programmers just like it. You say that many things are more intuitive with one-based indexing, but many things are less intuitive too. I don't think python ever would have become popular without zero-based indexing. The thing is, we really don't know why people pr…

Well for one, most situations I can think of when an index is calculated a 0-based index is more useful. Addressing an element T[h][w][c] in a linearized 3D tensor with 0-based indexing: T[h*W*C+w*C+c] with 1-based indexing: T[(h-1)*W*C+(w-1)*C+c] Or let's say you want to take a string "abc" and repeat it until the length is 10, getting "abcabcabca". With 0-based indexing: a,b = "abc", [" "]*10 for i in range(0,10):…

Julia was developed by scientists that used math software like MATLAB. Most mathematics software uses 1-based indexing. (Previous comment about this.[1])

>with 1-based indexing: T[(h-1) * W * C+(w-1) * C+c]

Your example showing how cumbersome and ugly it is to subtract 1 -- isn't how the intended audience uses mathematics software. Instead, they would use idiomatic multidimensional access with commas or multi-brackets syntax that understands 1-based indexes. They do not manually multiply and add the offsets into a raw 1-dimensional array. (E.g. idiomatic T[x,y] or T[x][y] instead of contrived T[(x-1) * W+(y-1)])

It's also the same concept for MS Excel. In VBA macro programming code, one doesn't need to litter the code with "minus 1" everywhere. The 1st row and 1st column (cell A1) is addressed as "1,1" in the Visual Basic function "Worksheets!Sheet1.Cells(1, 1)"

Mathematics software has a tradition of 1-based indexing probably because many (most?) equations in written mathematics are 1-based. (Examples [2] [3].) It doesn't seem so unreasonable that scientists prefer that their math programming code has 1-based indexes to match the 1-based indexes in traditional math notation.

[1] https://news.ycombinator.com/item?id=11072438

[2] average : summation symbol ∑ has "i=1..n" not "i=0..n-1": https://en.wikipedia.org/wiki/Average#Arithmetic_mean

[3] linear algebra matrices : i and j subscripts begin at 1,1 and not 0,0: https://en.wikipedia.org/wiki/Matrix_(mathematics)

Re: Automatic Differentiation with Julia

#37

Post by cultus »

elcomet wrote: post #35
IngoBlechschmid wrote: post #28

For those unaware of what automatic differentiation is: It's a close-to-magical tool which turns code for evaluating a function f into code for evaluating its derivative f'. It uses a special sort of invented numbers which square to zero even though they are not themselves zero. Here is one of the many tutorials on automatic differentiation: https://pizzaseminar.speicherleck.de/automatic-differentiati...

Thid is really neat. Is this really used in practice? It seems to me that most of the AD frameworks used for deep learning implement the backward function that returns the jacobian for every initial function, and then chain those backward functions

I used it for non ML tasks in geophysics, which made my life a lot easier. However, I think most scientists and engineers aren't aware of it. It has been described as "criminally underused."

Re: Automatic Differentiation with Julia

#38

Post by cultus »

mkborregaard wrote: post #24
0-_-0 wrote: post #17

Earlier quoted context omitted.

Well for one, most situations I can think of when an index is calculated a 0-based index is more useful. Addressing an element T[h][w][c] in a linearized 3D tensor with 0-based indexing: T[h*W*C+w*C+c] with 1-based indexing: T[(h-1)*W*C+(w-1)*C+c] Or let's say you want to take a string "abc" and repeat it until the length is 10, getting "abcabcabca". With 0-based indexing: a,b = "abc", [" "]*10 for i in range(0,10):…

Some algorithms work better with 0-based indexing. But maybe, just maybe, there are other aspects to look at when evaluating a programming language than "it doesn't by default follow my favourite indexing type". Especially given that 0-based indexing is very easy to use for any array you may wish to use it for.

Any mention of Julia immediately degenerates into bikeshedding about indexing.

Re: Automatic Differentiation with Julia

#39

Post by 0-_-0 »

jasode wrote: post #36
0-_-0 wrote: post #17

Earlier quoted context omitted.

Well for one, most situations I can think of when an index is calculated a 0-based index is more useful. Addressing an element T[h][w][c] in a linearized 3D tensor with 0-based indexing: T[h*W*C+w*C+c] with 1-based indexing: T[(h-1)*W*C+(w-1)*C+c] Or let's say you want to take a string "abc" and repeat it until the length is 10, getting "abcabcabca". With 0-based indexing: a,b = "abc", [" "]*10 for i in range(0,10):…

Julia was developed by scientists that used math software like MATLAB. Most mathematics software uses 1-based indexing. (Previous comment about this.[1]) >with 1-based indexing : T[(h-1) * W * C+(w-1) * C+c] Your example showing how cumbersome and ugly it is to subtract 1 -- isn't how the intended audience uses mathematics software. Instead, they would use idiomatic multidimensional access with commas or multi-bracke…

My argument was that in situations where the difference between 0 and 1 based indexing makes a difference it's ususally 0-based indexing that leads to simpler code.

Re: Automatic Differentiation with Julia

#40

Post by kxyvr »

IngoBlechschmid wrote: post #28

For those unaware of what automatic differentiation is: It's a close-to-magical tool which turns code for evaluating a function f into code for evaluating its derivative f'. It uses a special sort of invented numbers which square to zero even though they are not themselves zero. Here is one of the many tutorials on automatic differentiation: https://pizzaseminar.speicherleck.de/automatic-differentiati...

To clarify, your statement that, "It uses a special sort of invented numbers which square to zero even though they are not themselves zero." is not entirely true. In case someone else is looking at this, what the OP is referring to is dual numbers. That is one way to implement things, but not the most common way for fast tools.

Fundamentally, automatic differentiation is the methodical application of the chain rule. Forward mode results from the application of the rules for directional derivatives. Reverse mode results from the total derivative. The reason that we have a reverse pass in reverse mode can be seen from this perspective. The directional derivative is `(f o g)'(x)dx = f'(g(x)) g'(x)`. Note, we compute `g(x)` before `f(g(x))` and the derivative `g'(x)` before `f'(g(x))`. Therefore, we can compute the derivatives as we compute the answer. If we want the gradient, which results from the total derivative, we have `grad (f o g)(x)` = `g'(x)* grad f(g(x))`. Although we still compute `g(x)` before `f(g(x))` during our computation, the gradient requires the computation of `grad f(g(x))` before the application of the adjoint operator `g'(x)*`. We do the evaluations on the first pass, and cache extra values, and then compute the gradient on a reverse pass because we need the adjoint of the total derivatives in the reverse order.

Or, at least that's my bias in how to derive things.

Post reply on HN