University of Michigan - Ann Arbor

Google

A Feed forward neural network model training occurs in six stages:

- Iterate until convergence
   - Initialization
   - Forward propagation
   - Error function (Objective function)
   - Backpropagation
   - Weight update

At the end of this process, the model is ready to make predictions for unknown input data. New data can be fed to the model, a forward pass is performed, and the model generates its prediction.

Stages of Feed Forward Neural Network Learning

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

When we use a feedforward neural network to accept an input $x$ from training set and produce an output $\hat{y}$, information ﬂows forward through the network. The input $x$ provides the initial information that then propagates up to the hidden units at each layer and ﬁnally produces $\hat{y}$. This is called forward propagation. 


Forward Propagation

Weights are changed to the optimal values according to the results of the backpropagation algorithm.

Since the weights are updated a small delta step at a time, several iterations are required in order for the network to learn. After each iteration, the gradient descent force updates the weights towards less and less global loss function. The amount of iterations needed to converge depends on the learning rate, the network meta-parameters, and the optimization method used.

Update Weight Iteratively Until Convergence

Initial weights are applied to all the neurons.
It is necessary to set initial weights for the first forward pass. Two basic options are to set weights to zero or to randomize them. However, this can result in a vanishing or exploding gradient, which will make it difficult to train the model. To mitigate this problem, you can use a heuristic (a formula tied to the number of neuron layers) to determine the weights. A common heuristic used for the Tanh activation is called Xavier initialization.

On top of that if the architecture being used has already been trained by someone else (such as ImageNet) the node values can be initialized to those values. This is known as transfer learning and it can be very useful to substantially increase training speed and model accuracy.

Deep Learning Weight Initialization

What is the "cache" used for in our implementation of forward propagation and backward propagation?

Which of the following statements are True? (Check all that apply).

Consider the following 1 hidden layer neural network:

Backpropagation is a systematic computational procedure for applying the chain rule to calculate gradients automatically. It operates by traversing a computational graph in a backwards direction—from the output loss back to the input parameters—multiplying matrices of partial derivatives at each step to determine how parameters affect the final output.

Backpropagation

A learned scoring rule can cause inference errors if it gives the correct answer a lower score than the model’s proposed answer. In that situation, the right fix is usually to improve the score-estimating model, not the search procedure.

Learn Before

Related