Linear Attention Transformer. A smarter AI model that handles really long text without slowing down.

github.com/lucidrains/linear-attention-transformer
by @lucidrains

Vaya's read on this project

Problem, audience, market, and the verdict — sign in to see it.

Updates

September 2021
  • Release —0.19.1
  • tweak eps for causal linear attn
August 2021
  • Release —0.18.2
  • add token shifting feature
April 2021
  • Release —0.18.1
  • 0.18.1
  • fix stability issues
  • Release —0.18.0
  • add rotary embeddings
March 2021
  • Release —0.17.2
  • fix bug with context and D_inv
  • Release —0.17.1
  • Release —0.17.0
  • remove one kv head, and make sure causal linear attention is normaliz…
  • Release —0.16.1
  • fix leakage bug
  • Release —0.16.0
  • make sure no blindspot is supported
  • Release —0.15.5
  • fix leakage when blindspot size is 1
  • scale after norm
January 2021
  • update gcd to be used from math package, for python 3.9 compatibility
December 2020
  • numerical stability for causal linear attn
  • fix axial positional embedding if max sequence length is smaller than…
October 2020
  • add a final norm after attention layers
  • fix autopadder for linear attention transformer (not LM only)
  • recent linear papers seem to suggest that softmax normalization for k…
  • allow one to turn off normalization of queries for image linear atten…
August 2020
  • do not assert that dimension is divisible by number of heads if dim_h…
July 2020
  • update readme