Description
Tokenizes Python 3.12 and newer source code quickly while following the language specification. It helps linters, formatters, analyzers, and editor tools understand Python syntax at the token level.
Tokenization is analysis, not execution. Tools using it still need to handle invalid, partial, or untrusted source code gracefully.