https://datascience.stackexchange.com/questions/24534/does-gradient-descent-always-converge-to-an-optimum

University of Michigan - Ann Arbor

Gradient Descent is an algorithm which is designed to find the optimal points, but these optimal points are not necessarily global. And yes if it happens that it diverges from a local location it may converge to another optimal point but its probability is not too much.

Consider the following "recliner chair" type of function(image below).

Obviously, this can be constructed so that there is a range in the middle where the gradient is the 0 vector, casuing the fail to find global optima. 


Why Gradient descent might fail?

If failure is defined as a model becoming unable to escape a non-global local minimum through training, it is almost guaranteed to occur with a non-trivial and finite training set. This can be attributed to inherent $$\textbf{model identifiability}$$ issues, which may allow uncountably infinite local minima to exist. In practice, however, experts believe that most relevant local minima are already close to the global minimum when given sufficiently large training sets, although this remains an active area of research.

Learn Before

Related