Routing Transformer. An AI attention model that groups similar data together for faster processing.

github.com/lucidrains/routing-transformer
by @lucidrains

Vaya's read on this project

Problem, audience, market, and the verdict — sign in to see it.

Updates

November 2021
  • Release —1.6.1
  • make sure rotary positional embeddings are same type as queries
August 2021
  • readme
  • Release —1.6.0
  • add token shifting
May 2021
  • Release —1.5.2
  • apply rotary embeddings to values instead of using depthwise conv on …
April 2021
  • Release —1.5.1
  • fix rotary positional embedding
  • Release —1.5.0
  • go all the way with rotary embeddings
March 2021
  • readme
February 2021
  • Release —1.4.3
  • allow for fine control over cancelling and registering kmeans update …
December 2020
  • Release —1.4.2
  • expose KmeansAttention layer for personal use in other repositories
November 2020
  • Release —1.4.1
  • fix examples
  • Release —1.4.0
  • use embedding dimension, due to embedding factorization
  • cite
  • Release —1.3.0
  • allow one to look multiple blocks of window size into the past for lo…
October 2020
  • allow use of absolute positional embedding
  • offer option to multiply by token embedding weights for final logits,…
  • add a final norm post-attention layers
  • make sure one can update kmeans from AR wrapper as well
  • offer way to manually update kmeans
  • remove stray debug print
  • cite