Posts

The simple Essence of Automatic Differentiation

March 1, 2021 | Reading Time: 6 min

The simple Essence of Automatic Differentiation Disclaimer: I assume you know math up to derivatives and are somewhat comfortable with programming. Intuitively, Automatic differentiation (AD) is a black box that given a function gives you a derivative of it. We can use it to automagically perform things like backpropagation, given that the function you differentiate computes a feed-forward pass of the neural network. But what is a derivative? Linear Approximations The standard definition of a derivative is the following: