Rare find

Linformer. A faster attention mechanism for AI models that need to process long sequences of text or data.

github.com/lucidrains/linformer
by @lucidrains

Vaya's read on this project

Problem, audience, market, and the verdict — sign in to see it.

Updates

January 2024
  • Release —0.2.3
  • address https://github.com/lucidrains/linformer/issues/8
  • Release —0.2.2
January 2021
  • Update README.md
December 2020
  • Release —0.2.1
  • bump for dropout
  • Merge pull request #2 from umbertov/master
  • add dropout kwarg to LinformerLM as well
  • add option to set dropout in Linformer constructor
July 2020
  • Release —0.2.0
  • add reversibility, from Reformer paper
  • remove inaccurate comment due to fixed bug
  • Release —0.1.0
  • make it so each head dimension can be specified, for potentially scal…
June 2020
  • typo
  • Release —0.0.5
  • add dropout, initialization, proper scaling
  • update readme
  • Release —0.0.4
  • bump
  • make forward function accept kwargs
  • update readme
  • rename setup.py url
  • Release —0.0.3
  • further fix paths
  • Release —0.0.2
  • fix linformer paths
  • update linformer package name
  • Release —0.0.1
  • Create python-publish.yml