gMLP. A simpler AI model that does what Transformers do, without the complexity.

github.com/lucidrains/g-mlp-pytorch
by @lucidrains

Vaya's read on this project

Problem, audience, market, and the verdict — sign in to see it.

Updates

August 2021
  • Release —0.1.5
  • add token shifting
  • Release —0.1.4
  • correctly attribute
  • Release —0.1.2
  • deploy multiheadedness
  • Release —0.1.1
  • give absolute positional parameters to circulant matrix
  • Release —0.1.0
  • release ability to use circulant matrix
June 2021
  • Release —0.0.18
  • nonsquare patch sizes too
  • Release —0.0.17
  • allow for non-square images
May 2021
  • Release —0.0.16
  • correct tiny attention
  • Release —0.0.15
  • add ability to add nonlinearity to spatial gate
  • Release —0.0.14
  • small initial epsilon for SGU weights
  • make gMLPBlock and SGU accessible
  • abstract out gMLPBlock, to be used in DALLE-pytorch
  • 0.0.10
  • make sure autoregressive gMLP can accept variable sequence lengths
  • 0.0.9
  • Merge pull request #5 from lucidrains/gmlp-gpt
  • fix number of parameters not matching paper
  • 0.0.8
  • Merge pull request #3 from lucidrains/gmlp-gpt
  • Merge pull request #2 from lucidrains/gmlp-gpt