Description
Unsupervised text tokenization can be added to neural text generation and language-processing workflows. It is developer support for model preprocessing and vocabulary training.
FICHA · MANJARO
sentencepiece
Unsupervised text tokenizer for Neural Network-based text generation
- cli-tool
- CLI
- AI
- LANGUAGE
- Launchable
- Runs in terminal
- Same app · 2 sources
official+codex · reviewed · Jun 4, 2026 description in
en How to run
spm_train
Commands: spm_train
Permissions
Permissions not analysed for this source yet.