Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers

Very recommendable! These researchers made a tremendous effort in evaluating a large number of machine learning optimization methods.



"Choosing the optimizer is among the most crucial decisions of deep learning engineers, and it is not an easy one. The growing literature now lists literally hundreds of optimization methods. In the absence of clear theoretical guidance and conclusive empirical evidence, the decision is often done according to personal anecdotes. ... Evaluating 14 optimizers on eight deep learning problems using four different schedules, tuning over dozens of hyperparameter settings, to our knowledge, this is the most comprehensive empirical evaluation of deep learning optimizers up-to-date ..."



[2007.01547] Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers