Rare find

ST-MoE. A way to build AI models that use a team of specialist networks instead of one giant one.

github.com/lucidrains/st-moe-pytorch

Problem

The full read on this project is a sign-in away — what it solves, who it serves, and where it sits in the market.

Audience

The full read on this project is a sign-in away — what it solves, who it serves, and where it sits in the market.

Market

The full read on this project is a sign-in away — what it solves, who it serves, and where it sits in the market.

Pricing

The full read on this project is a sign-in away — what it solves, who it serves, and where it sits in the market.

Vaya's read on this project

Problem, audience, market, and the verdict — sign in to see it.

Updates

June 2024

cleanup with einx Jun 17
Release —0.1.8 Jun 4
0.1.8 Jun 4
address https://github.com/lucidrains/st-moe-pytorch/issues/13 Jun 4

February 2024

Release —0.1.7 Feb 29
remove erroneous backwards for split_by_rank Feb 29

January 2024

cleanup Jan 24
Release —0.1.6 Jan 24
address https://github.com/lucidrains/st-moe-pytorch/issues/4 Jan 24

December 2023

Release —0.1.5 Dec 14
make sure contiguous Dec 14

September 2023

Release —0.1.4 Sep 21
router z loss should be calculated on the unnoised gating logits Sep 21
Release —0.1.2 Sep 21
allow for noising of gates Sep 21
readme Sep 11
Release —0.1.1 Sep 11
researcher will want to log the unweighted auxiliary losses Sep 11
Release —0.1.0 Sep 11
rename loss_coef to balance_loss_coef, sum the balance and router z-l… Sep 11
validate that it works on single machine with multiple GPUs Sep 11
Release —0.0.30 Sep 11
handle variable sequence lengths if `allow_var_seq_len = True` on `Ex… Sep 11
project manage Sep 11
any combination of number of experts and world size should not break Sep 10
Release —0.0.29 Sep 10
oops Sep 10
chip away at edge cases Sep 10
another micro optimization for communication Sep 10
in split by rank function, cache the sizes so on backwards there is n… Sep 10