- In a nutshell: the deeper the network becomes, the harder the optimization problem becomes.
- The simplest neural network is the single-node perceptron , whose optimization problem is convex .
- To provably solve optimization problems for general neural networks with two or more layers, the algorithms that would be necessary hit some of the biggest open problems in computer science.
- There is a rich variety of optimization algorithms to handle convex optimization problems, and every few years a better polynomial-time algorithm for convex optimization is discovered.
- Judd also shows that the problem remains NP-hard even if it only requires a network to produce the correct output for just two-thirds of the training examples, which implies that even approximately training a neural network is intrinsically difficult in the worst case.

Deeper neural nets often yield harder optimization problems.

Continue reading “The hard thing about deep learning”