Rare find

MinBPE. Clean, understandable code for the text-splitting method that powers modern AI models.

github.com/karpathy/minbpe

Problem

The full read on this project is a sign-in away — what it solves, who it serves, and where it sits in the market.

Audience

The full read on this project is a sign-in away — what it solves, who it serves, and where it sits in the market.

Market

The full read on this project is a sign-in away — what it solves, who it serves, and where it sits in the market.

Pricing

The full read on this project is a sign-in away — what it solves, who it serves, and where it sits in the market.

Vaya's read on this project

Problem, audience, market, and the verdict — sign in to see it.

Updates

April 2024

Merge pull request #67 from shubham0204/master Apr 22
add gnp/minbpe-rs as community extensions in README.md Apr 22

February 2024

Merge pull request #43 from AneeshBose/aneesh/lecture-md Feb 26
fix token count Feb 23
ugh fix image link Feb 23
more work on lecture hehe. this is hard work! not sure an LLM can do … Feb 23
text version of the lecture Feb 22
Merge pull request #33 from richzw/master Feb 21
update readme.md with video Feb 21
Merge pull request #24 from nobleaustine/pytest_modify Feb 20
updated .gitignore Feb 20
create exercise to rewrite minbpe for learners Feb 19
add small amount of code to write out the gpt4 vocab in the same form… Feb 19
Merge pull request #18 from ahmedivy/master Feb 19
add link to another relevant issue here Feb 19
create requirements.txt Feb 19
fix linter error Feb 19
Merge pull request #17 from eltociear/patch-1 Feb 19
change the handling of special tokens inside encode of RegexTokenizer… Feb 19
refactor the readme file and adjust the API on how special tokens are… Feb 19