Problem No. 1 : Understanding what pretraining and training a supervised layer over it changes in the model

Our ability to characterize exactly what aspects of the pretrained parameters are retained during the supervised training stage is limited.

Problem No. 2 :

For example, if we train a generative model of images of cars and motorcycles, it will need to know about wheels, and about how many wheels should be in an image. If we are fortunate, the representation of the wheels will take on a form that is easy for the supervised learner to access. This is not yet understood at a mathematical, theoretical level, so it is not always possible to predict which tasks will benefit from unsupervised learning in this way.

Problem No. 3: Pretraining simultaneously vs Pretraining complete then supervised fine tuning Comparison

For example, if we wish to add a linear classifier on top of pretrained features, the features must make the underlying classes linearly separable. These properties often occur naturally but do not always do so. This is another reason that simultaneous supervised and unsupervised learning can be preferable—the constraints imposed by the output layer are naturally included from the start.

Problem No. 4: How to make output of neural networks deterministic? Neural network training is non-deterministic, and converges to a different function every time it is run.

Problem No 5: Adjusting amount of effect created by unsupervised pretraining

Unsupervised pretraining does not offer a clear way to adjust the the strength of the regularization arising from the unsupervised stage.

Problem No 6: