Rare find

nGPT. A PyTorch implementation of Nvidia's normalized GPT that trains AI models using a different mathematical approach.

github.com/lucidrains/nGPT-pytorch

Problem

The full read on this project is a sign-in away — what it solves, who it serves, and where it sits in the market.

Audience

The full read on this project is a sign-in away — what it solves, who it serves, and where it sits in the market.

Market

The full read on this project is a sign-in away — what it solves, who it serves, and where it sits in the market.

Pricing

The full read on this project is a sign-in away — what it solves, who it serves, and where it sits in the market.

Vaya's read on this project

Problem, audience, market, and the verdict — sign in to see it.

Updates

June 2025

Release —0.2.16 Jun 3
0.2.16 Jun 3
address https://github.com/lucidrains/nGPT-pytorch/issues/12 Jun 3

March 2025

Release —0.2.15 Mar 19
complete ability to breed two normalized transformers, add another fu… Mar 19
Release —0.2.14 Mar 16
refactor evo functions to ready for full transformer Mar 16
Release —0.2.12 Mar 15
prepare to crossover entire transformer Mar 15
Release —0.2.11 Mar 14
evolution will be part of the final equation. Mar 14
update Mar 13
Release —0.2.10 Mar 13
cross over the feedforward scales as well Mar 13
Release —0.2.9 Mar 13
prepare for breeding the nGPT feedforwards Mar 13

November 2024

Release —0.2.7 Nov 7
add enable_cudnn Nov 7
Release —0.2.6 Nov 3
fix returning zeros when row is all masked out for non-causal, and re… Nov 3
Release —0.2.5 Nov 2
protect against row being all masked out in sdpa Nov 2
oops, got it backwards Nov 2
address https://github.com/lucidrains/nGPT-pytorch/issues/11 with a h… Nov 2
0.2.2 Nov 2
Revert "address https://github.com/lucidrains/nGPT-pytorch/issues/9" Nov 2
address https://github.com/lucidrains/nGPT-pytorch/issues/9 Nov 2

October 2024

line things up a bit more accurately for follow up research Oct 31
prep for kv cache Oct 31
fix value residual Oct 31