(Q2438971)
Statements
Identifiers
Sitelinks
Wikipedia(4 entries)
- dewiki Tokenisierung
- enwiki Tokenization (lexical analysis)
- idwiki Tokenisasi
- ruwiki Токен (лексический анализ)
breaking a stream of text up into chunks for analysis or further processing