Performer. A faster way to build AI models that understand long sequences of information.

github.com/lucidrains/performer-pytorch
by @lucidrains

Vaya's read on this project

Problem, audience, market, and the verdict — sign in to see it.

Updates

February 2022
  • Release —1.1.4
  • make sure amax for numerical stability is detached
November 2021
  • Release —1.1.3
  • 1.1.3
  • Merge pull request #78 from AlexandreDey/patch-1
  • Update autoregressive_wrapper.py
October 2021
  • Add repetition penalty for text generation
  • Release —1.1.2
  • 1.1.2
  • Merge pull request #75 from Erotemic/dev/fix_deprecation_warn
  • Release —1.1.1
  • fix axis for torch max used for numerical stability when deriving key…
  • Fix torch.qr deprecation warning
August 2021
  • Release —1.1.0
  • update example
  • add token shifting feature, for better convergence
April 2021
  • Release —1.0.11
  • remove pytorch-fast-transformers from dep
  • Release —1.0.10
  • add a handy method for ProjectionUpdater
  • Release —1.0.9
  • expose projection update logic, for easy use in alphafold2 project
  • cite rotary embeddings
  • expose cross attention, in ready to use Performer in alphafold2 repo
  • Release —1.0.8
  • Release —1.0.7
  • allow for users to turn off bias for attention out projections
  • eps for non-cuda version of causal linear attention
  • only apply rotary embeddings to global heads (local attention library…
  • revert change