Adan. A faster optimizer for training AI models in PyTorch.

github.com/lucidrains/Adan-pytorch

Vaya's read on this project

Problem, audience, market, and the verdict — sign in to see it.

Updates

September 2022
  • link to official code, released last night
August 2022
  • Release —0.1.0
  • code was reviewed by author @XingyuXie https://github.com/lucidrains/…
  • Release —0.0.5
  • add bias correction terms, as authors mentioned they had it https://g…
  • link to research scientist twitter thread
  • Release —0.0.4
  • get the restart condition in there, even if it is not used
  • Release —0.0.3
  • use in-place update for saving previous grad
  • Release —0.0.2
  • change to non-txl betas hparams, suggested by @sdtblck and just zero …
  • Release —0.0.1
  • first commit
  • Initial commit