µTransfer: A technique for hyperparameter tuning of enormous neural networks

Appears to be very recommendable (but I have not read the paper yet)! Perhaps a breakthrough in large neural network optimization!

This new approach could accelerate the training of larger and larger models! Perhaps, resources constrained researchers in business and academia may benefit from it as well. 

"... Before this work, the larger a model was, the less well-tuned we expected it to be due to the high cost of tuning. ..."

From the abstract:

"... We show that, in the recently discovered Maximal Update Parametrization (μP), many optimal HPs [Hyperparameters] remain stable even as model size changes. This leads to a new HP tuning paradigm we call *μTransfer*: parametrize the target model in μP, tune the HP indirectly on a smaller model, and *zero-shot transfer* them to the full-sized model, i.e., without directly tuning the latter at all. ..."

µTransfer: A technique for hyperparameter tuning of enormous neural networks - Microsoft Research

Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer

Credits: Last Week in AI #158