Rare find

Memory Transformer. AI that remembers across millions of words instead of forgetting after a few thousand.

github.com/lucidrains/recurrent-memory-transformer-pytorch
by @lucidrains

Vaya's read on this project

Problem, audience, market, and the verdict — sign in to see it.

Updates

January 2025
  • Release —0.7.0
  • take the free lunch
  • Release —0.6.0
  • replace resiDual with its successor, hyper-connections
November 2024
  • Merge pull request #25 from avanturist322/patch-1
  • Update README.md
February 2024
  • Release —0.5.6
  • address https://github.com/lucidrains/recurrent-memory-transformer-py…
August 2023
  • Release —0.5.5
  • address https://github.com/lucidrains/recurrent-memory-transformer-py…
  • Release —0.5.4
  • address https://github.com/lucidrains/recurrent-memory-transformer-py…
  • Release —0.5.3
  • allow for customizing whether read memory is stop gradded, the one ad…
  • Release —0.5.2
  • reinject the write memory positions
  • Release —0.5.1
  • address https://github.com/lucidrains/recurrent-memory-transformer-py…
  • Release —0.5.0
  • give a null key / value to protect against entirely masked out row, a…
  • Release —0.4.3
  • address https://github.com/lucidrains/recurrent-memory-transformer-py…
  • address https://github.com/lucidrains/recurrent-memory-transformer-py…
June 2023
  • update
May 2023
  • turns out resiDual is prone to overflowing in fp16. add the scaling s…
  • adopt the resiDual norm pattern, may help with vanishing gradients in…
  • incorporate a lesson from block recurrent transformer - reinject each…
  • default the xl memories to be on, suspect could make training more st…
  • make rotary embeddings work with xl memories, project manage
  • allow for norming the write memories being passed on to the next recu…