Rare find

Rotary Embedding. A tool that helps AI models understand word order and position.

github.com/lucidrains/rotary-embedding-torch

Problem

The full read on this project is a sign-in away — what it solves, who it serves, and where it sits in the market.

Audience

The full read on this project is a sign-in away — what it solves, who it serves, and where it sits in the market.

Market

The full read on this project is a sign-in away — what it solves, who it serves, and where it sits in the market.

Pricing

The full read on this project is a sign-in away — what it solves, who it serves, and where it sits in the market.

Vaya's read on this project

Problem, audience, market, and the verdict — sign in to see it.

Updates

June 2026

tests and gqa Jun 20
cleanup Jun 20
add a fused flash attention with rotary where one can define which to… Jun 20

January 2026

link to pope Jan 30

July 2025

Release —0.8.9 Jul 27
patch Jul 27
Release —0.8.8 Jul 7
some defaults for .get_seq_pos device and dtype Jul 7
Release —0.8.7 Jul 1
add ability to offset the axial dimensions when fetching axial freque… Jul 1

November 2024

Release —0.8.6 Nov 27
0.8.6 Nov 27
Merge pull request #38 from f0k/int-seq-len Nov 27
Change cached seq_len from buffer to plain int Nov 12
Release —0.8.5 Nov 11
assume sequence dimension for freqs to be first if freqs only has two… Nov 11

September 2024

Release —0.8.4 Sep 29
can slice the rotary pos emb being passed into apply_rotary_emb at th… Sep 29
Release —0.8.3 Sep 7
only set cache if offset is 0 Sep 7
Release —0.8.2 Sep 7
address https://github.com/lucidrains/rotary-embedding-torch/issues/1… Sep 7
Release —0.8.1 Sep 6
Release —0.8.0 Sep 6
xpos fix Sep 6
Merge pull request #34 from TimFelixBeyer/patch-1 Sep 6
Fix chi scale multiplication Sep 6
get rid of a warning Sep 2

August 2024

avoid an inplace-op Aug 20
Merge pull request #31 from blasscoc/blasscoc-patch-inplace-error Aug 20