Rare find

Native Sparse Attention. A faster way to run AI attention calculations by skipping unnecessary parts.

github.com/lucidrains/native-sparse-attention-pytorch

Problem

The full read on this project is a sign-in away — what it solves, who it serves, and where it sits in the market.

Audience

The full read on this project is a sign-in away — what it solves, who it serves, and where it sits in the market.

Market

The full read on this project is a sign-in away — what it solves, who it serves, and where it sits in the market.

Pricing

The full read on this project is a sign-in away — what it solves, who it serves, and where it sits in the market.

Vaya's read on this project

Problem, audience, market, and the verdict — sign in to see it.

Updates

August 2025

Release —0.2.3 Aug 15
0.2.3 Aug 15
credit assignment Aug 15
Merge pull request #33 from StrongSpoon/mask_fix Aug 15

July 2025

[bugfix] add mask for offs_n Jul 14

June 2025

Release —0.2.2 Jun 11
address https://github.com/lucidrains/native-sparse-attention-pytorch… Jun 11

May 2025

Release —0.2.1 May 16
fix maximum tracking in triton May 16

March 2025

turns out gemma v3 decided on qk norm over softcapping Mar 26
start fashioning a version with attention softclamping a la gemma and… Mar 25
Release —0.2.0 Mar 25
release new compression block hparams Mar 25
Merge pull request #25 from Mr-Grin/main Mar 25
additional assertion to make sure compress_block_sliding_stride must … Mar 25
minor fix Mar 25
minor modifications to accomodate compress_block_sliding_stride Mar 25
compress_block_sliding_stride Mar 24
Release —0.1.27 Mar 24
0.1.27 Mar 24
Merge pull request #24 from Pasewark/token_inference_fix Mar 24
Small change so token embeddings aren't looked up for past tokens dur… Mar 24
Release —0.1.26 Mar 23
0.1.26 Mar 23
address https://github.com/lucidrains/native-sparse-attention-pytorch… Mar 23
not using attn biases Mar 22
Release —0.1.25 Mar 22
Release —0.1.24 Mar 20
Release —0.1.23 Mar 19
Release —01.22 Mar 19