In one of my recent posts, I talked about pretext training, in which the Machine Learning model is trained to perform an upstream task prior to being trained on the actual, downstream task. This helps in reducing the amount of labeled data required to train a model and is used widely for Self Supervised Learning. …
Pretext Training Tasks for Self Supervised Learning in Computer Vision Read More »
The choice of activation function is very important and can greatly influence the accuracy and training time of a model. Swish is one of the new activation functions which was first proposed in 2017 by using a combination of exhaustive and reinforcement learning-based search. The authors of the research paper first proposing the Swish Activation …
Swish as an Activation Function in Neural Network Read More »
ELU, also know as Exponential Linear Unit is an activation function which is somewhat similar to the ReLU with some differences. Similar to other non-saturating activation functions, ELU does not suffer from the problem of vanishing gradients and exploding gradients. And similar to Leaky-ReLU and PReLU, and unlike ReLU, ELU does not suffer from the …
ELU as an Activation Function in Neural Networks Read More »
Rectified Linear Unit, also known as ReLU overcame some of the serious disadvantages of the earlier used activation functions such as Sigmoid and Hyperbolic Tangent activation functions. These include the exploding gradients and vanishing gradients problem. ReLU overcame these problems and it was fast and simple to calculate. However, despite all this it was far …
Leaky ReLU as an Activation Function in Neural Networks Read More »
Sigmoid function also known as logistic function is one of the activation functions used in the neural network. An activation function is the one which decides the output of the neuron in a neural network based on the input. The activation function is applied to the weighted sum of all the inputs and the bias …
Sigmoid as an Activation Function in Neural Networks Read More »