Whitening and second order optimization both destroy information about the dataset, and can make generalization impossible

Recommendable paper by UC Berkeley and Google! It raises e.g. some serious questions about the commonly and widely applied image pre-processing in computer vision. How much does this pre-processing affect generalization negatively?



"Models trained using whitened data, or with certain second order optimization schemes, have less access to this information; in the high dimensional regime they have no access at all, producing models that generalize poorly or not at all."



[2008.07545] Whitening and second order optimization both destroy information about the dataset, and can make generalization impossible