The simple Essence of Automatic DifferentiationMarch 1, 2021 | Reading Time: 6 min
The simple Essence of Automatic Differentiation Disclaimer: I assume you know math up to derivatives and are somewhat comfortable with programming. Intuitively, Automatic differentiation (AD) is a black box that given a function gives you a derivative of it. We can use it to automagically perform things like backpropagation, given that the function you differentiate computes a feed-forward pass of the neural network. But what is a derivative? Linear Approximations The standard definition of a derivative is the following: