You've just described the dual numbers, which provide a way of implementing forward-mode autodiff.
> If the derivative is abnormal in a way such that the normal
> truncated calculative formulas would conceal it, then AD
> would fail to give an accurate result as well.
What I'm seeing directly contradicts that. I've tested the example above (the one I called f(x,a) with a=1e-36, trying to find its derivative at x=0), and it gave the right answer with dual numbers, but not with finite differencing. So what you're saying isn't true. I'll post a minimal implementation here later.
[edit]
I've used Sympy out of laziness, which is inelegant. I don't know how clear this will be.
First of all, I have to change the number 1 in the definition of f(x,a) into a matrix. We need this because we'll be representing dual numbers as matrices, and you can't add the scalar 1 to a matrix:
one = eye(2)
def f(x, a=1e-36):
return one + a * sin(x / a)
Sympy doesn't have its own implementation of `sin`, so we'll need to provide one:
def sin(M):
return im(exp(I*M).n())
At the dual number ε, we have f(ε) = f(0) + ε f'(0). The dual number ε will be represented as the matrix
⎡0 1⎤
⎢ ⎥
⎣0 0⎦
As code:
epsilon = Matrix([[0,1],[0,0]])
Evaluating f on the above matrix gives:
In: f(epsilon)
Out:
⎡1 1.0⎤
⎢ ⎥
⎣0 1 ⎦
Which tells us that f(0) ≈ 1 and f'(0) ≈ 1. Both are correct.
If we use finite differencing instead, we get:
In: (f(h * eye(2)) - f(0 * eye(2)))/h
Out:
⎡0 0⎤
⎢ ⎥
⎣0 0⎦
So it's claiming incorrectly that the derivative at 0 is 0. No finite differencing scheme can fix this because floating point causes the function f to become constant.
Autodiff using the dual numbers has pulled off the seemingly impossible.
[edit: An early typo caused the code to produce an incorrect result. Now fixed, and my claim holds.]