Tokenize

Pronunciation

• GA IPA: en, /ˈtoÊŠ.kÉ™n.aɪz/

Origin

en + -token + ize

Full definition of tokenize

Verb

1. (transitive, computing) To reduce to a set of tokens by lexical analysis.
2. (transitive, computing) To substitute sensitive data with meaningless placeholders.
3. (transitive) To treat as a token minority.

Derived terms

• tokenizable
• tokenizer

Related terms

• tokenization