Rare find

MEGABYTE. An AI model that processes very long sequences of raw data by breaking them into chunks and handling them at different scales.

github.com/lucidrains/MEGABYTE-pytorch
by @lucidrains

Vaya's read on this project

Problem, audience, market, and the verdict — sign in to see it.

Updates

December 2024
  • Release —0.3.6
  • address https://github.com/lucidrains/MEGABYTE-pytorch/issues/21
September 2024
  • Release —0.3.5
  • address https://github.com/lucidrains/MEGABYTE-pytorch/issues/20
  • Release —0.3.4
  • fix regression and some dimension conditional
  • Release —0.3.3
  • update torch dep to 2.3
  • Merge pull request #18 from eegli/update-flash-attention
  • Release —0.3.2
  • rotary embedding done in full prec
  • Release —0.3.1
  • address issue with abs pos emb https://github.com/lucidrains/MEGABYTE…
  • update flash attention config
May 2024
  • Release —0.3.0
  • address https://github.com/lucidrains/MEGABYTE-pytorch/issues/13
June 2023
  • Release —0.2.1
  • make sure it supports greater than 2 hierarchies
  • Release —0.2.0
  • move closer to what the paper did, with local and global token embedd…
  • Release —0.1.7
  • switch to rotary embeddings, as they did in the paper
  • evidence is emerging that decoders generate implicit absolute and rel…
May 2023
  • address https://github.com/lucidrains/MEGABYTE-pytorch/issues/10
  • learn on the very first start token
  • Merge pull request #9 from lucidrains/paper-variation
  • cleanup
  • Merge pull request #8 from lucidrains/paper-variation
  • fix and release 0.1.0
  • the paper did multiple start tokens and also had last hierarchy proje…