Rare find

Memory Efficient Attention. A way to run AI attention math without eating up all your computer's memory.

github.com/lucidrains/memory-efficient-attention-pytorch

Problem

The full read on this project is a sign-in away — what it solves, who it serves, and where it sits in the market.

Audience

The full read on this project is a sign-in away — what it solves, who it serves, and where it sits in the market.

Market

The full read on this project is a sign-in away — what it solves, who it serves, and where it sits in the market.

Pricing

The full read on this project is a sign-in away — what it solves, who it serves, and where it sits in the market.

Vaya's read on this project

Problem, audience, market, and the verdict — sign in to see it.

Updates

July 2023

comments Jul 18
readme Jul 18
Release —0.1.6 Jul 18
further simplification Jul 18
Release —0.1.5 Jul 18
add new trick from flash attention 2 that saves on division Jul 18
remove 3.7 support Jul 16
Release —0.1.4 Jul 16
bump version Jul 16
allow for full attention mask in naive flash attention impl Jul 16

March 2023

update Mar 17
Release —0.1.2 Mar 5
0.1.2 Mar 5
Merge pull request #7 from marksgraham/main Mar 5
Import partial Mar 2

December 2022

Release —0.1.1 Dec 30
fix tests Dec 30
Release —0.1.0 Dec 30
0.1.0 Dec 30
Merge pull request #6 from usryokousha/main Dec 30
Added dropout support to memory efficient variant Dec 30

November 2022

fix tests Nov 23
torch 1.13 is broken for tests Nov 1
Release —0.0.27 Nov 1
bring in the further simplification to flash attention that @tridao d… Nov 1

July 2022

Release —0.0.26 Jul 23
fix cosine sim flash attention as well Jul 23
Release —0.0.25 Jul 23
test out flash attention in GPT Jul 23
Release —0.0.24 Jul 23