Efficient E-Matching for Super Optimizers
https://blog.vortan.dev/ematching/
#HackerNews #Efficient #E-Matching #Super #Optimizers #E-Matching #Technology #Optimization
blog.vortan.devSinon's Blog
Administered by:
#optimizers
0 posts0 participants0 posts today
This MicroAdam paper from #NeurIPS2024 is nicely written! The algorithm is walked through in plain language first, and all the equations and proofs placed in the appendix. Super understandable, kudos to the authors.
https://arxiv.org/abs/2405.15593
#AI #MachineLearning #LLMs #optimizers
arXiv.orgMicroAdam: Accurate Adaptive Optimization with Low Space Overhead and Provable ConvergenceWe propose a new variant of the Adam optimizer called MicroAdam that specifically minimizes memory overheads, while maintaining theoretical convergence guarantees. We achieve this by compressing the gradient information before it is fed into the optimizer state, thereby reducing its memory footprint significantly. We control the resulting compression error via a novel instance of the classical \emph{error feedback} mechanism from distributed optimization in which *the error correction information is itself compressed* to allow for practical memory gains. We prove that the resulting approach maintains theoretical convergence guarantees competitive to those of AMSGrad, while providing good practical performance. Specifically, we show that MicroAdam can be implemented efficiently on GPUs: on both million-scale (BERT) and billion-scale (LLaMA) models, MicroAdam provides practical convergence competitive to that of the uncompressed Adam baseline, with lower memory usage and similar running time. Our code is available at https://github.com/IST-DASLab/MicroAdam.
'PROMISE: Preconditioned Stochastic Optimization Methods by Incorporating Scalable Curvature Estimates', by Zachary Frangella, Pratik Rathore, Shipu Zhao, Madeleine Udell.
http://jmlr.org/papers/v25/23-1187.html
#optimizers #optimization #preconditioned
'PyPop7: A Pure-Python Library for Population-Based Black-Box Optimization', by Qiqi Duan et al.
http://jmlr.org/papers/v25/23-0386.html
#optimizers #optimization #pypop7
'Multi-Objective Neural Architecture Search by Learning Search Space Partitions', by Yiyang Zhao, Linnan Wang, Tian Guo.
http://jmlr.org/papers/v25/23-1013.html
#optimizers #optimizer #optimizations
'Robust Black-Box Optimization for Stochastic Search and Episodic Reinforcement Learning', by Maximilian Hüttenrauch, Gerhard Neumann.
http://jmlr.org/papers/v25/22-0564.html
#reinforcement #optimizers #optimizes
'Neural Feature Learning in Function Space', by Xiangxiang Xu, Lizhong Zheng.
http://jmlr.org/papers/v25/23-1202.html
#features #feature #optimizers
'Win: Weight-Decay-Integrated Nesterov Acceleration for Faster Network Training', by Pan Zhou, Xingyu Xie, Zhouchen Lin, Kim-Chuan Toh, Shuicheng Yan.
http://jmlr.org/papers/v25/23-1073.html
#accelerated #optimizers #adaptive
'Scaling the Convex Barrier with Sparse Dual Algorithms', by Alessandro De Palma, Harkirat Singh Behl, Rudy Bunel, Philip H.S. Torr, M. Pawan Kumar.
http://jmlr.org/papers/v25/21-0076.html
#optimizers #sparse #dual
'Polygonal Unadjusted Langevin Algorithms: Creating stable and efficient adaptive algorithms for neural networks', by Dong-Young Lim, Sotirios Sabanis.
http://jmlr.org/papers/v25/22-0796.html
#langevin #adaptive #optimizers
'Improving physics-informed neural networks with meta-learned optimization', by Alex Bihlo.
http://jmlr.org/papers/v25/23-0356.html
#optimizers #learnable #learned
A DNN Optimizer that Improves over AdaBelief by Suppression of the Adaptive Stepsize Range
Guoqiang Zhang, Kenta Niwa, W. Bastiaan Kleijn
Action editor: Rémi Flamary.
Auto-configuring Exploration-Exploitation Tradeoff in Population-Based Optimization: A Deep Reinforcement Learning Approach
Learning to Optimize Quasi-Newton Methods
The Slingshot Effect: A Late-Stage Optimization Anomaly in Adaptive Gradient Methods
Personalized Federated Learning: A Unified Framework and Universal Optimization Techniques
Filip Hanzely, Boxin Zhao, mladen kolar
Action editor: Naman Agarwal.
A DNN Optimizer that Improves over AdaBelief by Suppression of the Adaptive Stepsize Range
Constrained Parameter Inference as a Principle for Learning
Nasir Ahmad, Ellen Schrader, Marcel van Gerven
Continued thread
it is now evolving into a #meta #physics #engine #MetaPhysics #PhysicsEngine
a #protocol #dedicating #its #self to the #optimization of #optimizers