Ctok. Break down Python code into its smallest pieces.

github.com/gvanrossum/ctok

Vaya's read on this project

Problem, audience, market, and the verdict — sign in to see it.

Updates

June 2019
  • Rearrange tp_* fields in the order of the actual struct
  • Make input a property, add encoding property
  • Add line number to errors; with tests
  • Add input() and get_raw() methods
  • Add demo.py
  • Test for multi-line string (exposes Python version differences)
  • Check that newline+indent inside () is ignored
  • Symbolize tests
  • Add pytest integration to setup.py
  • Support Python 3.5 and up
  • Return line/column offsets
  • Make it an iterator
  • Basic tokenizer functionality