Gradient Accumulation. Train large AI models on a single computer by breaking the work into smaller chunks.

sebastianraschka.com/blog/2023/llm-grad-accumulation.html

Vaya's read on this project

Problem, audience, market, and the verdict — sign in to see it.

Updates

March 2023
  • add torch.compile
  • Update and rename 2_batchsize-16.py to 2_batchsize-8.py
  • Update 1_batchsize-1.py
  • add code
  • Initial commit