Rare find
Soft MoE. A smarter way to route data through multiple AI models at once.
github.com/lucidrains/soft-moe-pytorch
Vaya's read on this project
Problem, audience, market, and the verdict — sign in to see it.
Updates
April 2025
- Release —0.1.9
- address https://github.com/lucidrains/soft-moe-pytorch/issues/10
April 2024
- Release —0.1.8
- address https://github.com/lucidrains/soft-moe-pytorch/issues/7
February 2024
- Release —0.1.7
- remove erroneous split by rank backwards
- Release —0.1.6
- prepare for use in soft-actor-critic value network, given new deepmin…
January 2024
- Release —0.1.5
- add option to use layernorm instead of rmsnorm
December 2023
- Release —0.1.4
- expose an extra flag
- Release —0.1.2
- another quick fix
- Release —0.1.1
- add a contiguous, thanks to @conceptofmind
September 2023
- Release —0.1.0
- add ability to noise gates and anneal noise
- Release —0.0.9
- just port over some logic from st moe
- quick fix
- project management
- remove wip, project management
- naive distributed code
August 2023
- thoughts
- for dynamic slots, need to mask out slots too, in the case of variabl…
- credit assignment
- improvise a variant that calculate dynamic number of slots
- account for key-padding mask
- 0.0.3