Activation functions play an important role in neural nets. An activation function transforms the weighted sum of the input into an output from a node.
Simply put, an activation function defines the output of a neuron given a set of inputs. It is called "activation" to mimic the working of a brain neuron. Our neurons get activated due to some stimulus. Similarly the activation function will decide which neurons in our neural net get activated.
Each hidden layer of a neural net needs to be assigned an activation function. Even the output layer of a neural net would use an activation function.
The ReLU (Rectified Linear Unit) function is the most common function used for activation.
The ReLU function is a simple function: max(0.0, x). So essentially it takes the max of either 0 or x. Hence all negative values are ignored.
Other activation functions are Sigmoid and Tanh. The sigmoid activation function generates an output value between 0 and 1.
An excellent video explaining activation function is here - https://youtu.be/m0pIlLfpXWE
The activation functions that are typically used for the output layer are Linear, Sigmoid (Logistic) or Softmax. A good explanation of when to use what is available here - https://machinelearningmastery.com/choose-an-activation-function-for-deep-learning/
No comments:
Post a Comment