Sinkhorn Transformer. A memory-efficient AI model that processes long sequences by smartly matching related pieces of information.

github.com/lucidrains/sinkhorn-transformer
by @lucidrains

Vaya's read on this project

Problem, audience, market, and the verdict — sign in to see it.

Updates

August 2021
  • Release —0.11.4
  • fix autopadder when mask is not given
March 2021
  • Release —0.11.3
  • add local attention as dep
January 2021
  • Release —0.11.2
  • update gcd to be from math package, for python 3.9 compat
November 2020
  • Release —0.11.1
  • fix bug for post attention layer norms when using emb factorization
October 2020
  • Release —0.11.0
  • normalize after attention layers
September 2020
  • Release —0.10.3
  • Update setup.py
  • Release —0.10.2
  • add embedding dropout
August 2020
  • Release —0.10.1
  • bump for release
  • Merge pull request #24 from tatp22/bug_fix
  • Fixed bug with dim_head and dim
July 2020
  • Release —0.10.0
  • bump package
  • allow for better scaling of heads, from findings of new paper, with a…
  • Release —0.9.1
  • move local attention to separate repository
June 2020
  • remove absolute positional embedding in favor of axial pos emb
  • update axial positional embedding
  • move axial positional embedding to separate package
  • add product key memory
  • fix for nucleus sampling
May 2020
  • make sure zero optimization is turned on in deepspeed config
  • update readme