Description
Text tokenization for machine-learning models can use fast Rust-backed algorithms from Python. Researchers and ML engineers can train, load, and run tokenizers for NLP pipelines and transformer models. It processes text that may contain sensitive user or dataset content.