Rare find

Memorizing Transformers. AI model that remembers and retrieves relevant context from past data.

github.com/lucidrains/memorizing-transformers-pytorch
by @lucidrains

Vaya's read on this project

Problem, audience, market, and the verdict — sign in to see it.

Updates

July 2023
  • Release —0.4.1
  • address https://github.com/lucidrains/memorizing-transformers-pytorch…
March 2023
  • Release —0.4.0
  • prepare to use knn attention in another repository, for the ultimate …
November 2022
  • Release —0.3.10
  • 0.3.10
  • Merge pull request #11 from LakshyAAAgrawal/patch-2
  • Update KNNMemory.add to correctly index and update KNN list, after ad…
  • use pack correctly
  • Release —0.3.9a
  • fix setup.py
  • Release —0.3.9
  • address https://github.com/lucidrains/memorizing-transformers-pytorch…
  • Release —0.3.8
  • use the new einops unpack! thank you @arogozhnikov 🙏
September 2022
  • Merge pull request #8 from LakshyAAAgrawal/patch-1
  • Update train.py to correct implementation of val loss calculation
April 2022
  • Release —0.3.7
  • just give knn attention its own relative positional bias
  • Release —0.3.6
  • give knn attention layer one more way to tune out local if need be
  • Release —0.3.5
  • allow the network to pay more attention to memory later into training…
  • readme
  • Release —0.3.4
  • turn KNN attention into full cosine sim attention (from the paper que…
  • mixing cosine sim attention (for distant) and non-cosine sim (for loc…
  • protect against knn memory directory being used from two places
  • fix for normalized queries for distant knn attention
  • do hybrid local + knn distant attention, thanks to @igor0 for the sug…