Rare find

FLASH. A faster transformer model that runs in linear time instead of the usual slower approach.

github.com/lucidrains/FLASH-pytorch
by @lucidrains

Vaya's read on this project

Problem, audience, market, and the verdict — sign in to see it.

Updates

September 2023
  • Release —0.1.9
  • address https://github.com/lucidrains/FLASH-pytorch/issues/12
June 2023
  • clearer einops, addressing https://github.com/lucidrains/FLASH-pytorc…
May 2023
  • Release —0.1.8
  • add ability to directly handle t5 relative positional bias within GAU
February 2023
  • Release —0.1.7
  • fix laplace activation function, thanks to @boweny-cerebras
September 2022
  • Release —0.1.6
  • add the laplace attention function, which the authors of Mega propose…
June 2022
  • Release —v0.1.5
  • make reducing group for context in noncausal linear attention a hyper…
  • Release —v0.1.4
  • fix an error thanks to @ShomyLiu at https://github.com/lucidrains/FLA…
April 2022
  • Release —0.1.2
  • fix mask for linear attention
March 2022
  • comment - turn on shift tokens in enwik8 demo
  • advertise token shift
  • Release —0.1.1
  • add token shift, for dramatic improvements, cite @BlinkDL
  • perfect
  • Release —0.0.1
  • verify enwik8 autoregressive works, also remove todo, as non-overlapp…
  • Release —0.0.15
  • mask bug
  • remove wip
  • fix
  • take care of masking
  • it appears they shared the same values for quadratic and linear atten…
  • FLASH module have one more dropout after final projection
  • allow norm in GAU to be customizable as well