SuperTokenizer. A high-performance tokenizer built to rival GPT-4, trained on the C4 dataset.

github.com/wassemgtk/SuperTokenizer

Vaya's read on this project

Problem, audience, market, and the verdict — sign in to see it.

Updates

No recent activity.