Nonlinear Activations for Neural Networks

Non-linear activations are important in deep neural networks. It is important in the sense that without non-linear activation functions, even if you have many linear layers, the end results is like you have only one linear layer, and the approximation ability of the network is very limited¹. Some of most commonly-used nonlinear activation functions are Sigmoid, ReLU and Tanh.