Rare find
Build NanoGPT. Learn how to build a working AI language model from scratch with video lessons and clean code steps.
github.com/karpathy/build-nanogpt
Vaya's read on this project
Problem, audience, market, and the verdict — sign in to see it.
Updates
June 2024
- fix guard of torch.cuda based on device type
- Merge pull request #28 from WilsonCWu/patch_ylim
- Merge pull request #30 from WilsonCWu/fix_model.require_backward_grad…
- fix model.require_backward_grad_sync
- fix ylim warning in play.ipynb
- point to litgpt and tinyllama which are like nanoGPT but a bit more m…
- fix device_type, which i understand to lead to errors in some version…
- convert uint16 to int32 before cast to long, otherwise pytorch doesn'…
- Delete unused register_bias once we switched to flash attention
- Update train_gpt2.py
- add the YouTube url now that we have it
- add visualization code for train,val,hella score
- add checkpointing
- readme tweaks more infos
- add example generations from 124M
- add discord and discussions
- add readme v1
- hellaswag adeval d to main train file, and logging
- move up the samepling code into the main loop, but disable it because…
- add validation split