(computing) A system that parses an input stream into its component tokens.

plural of tokenizer

simple past tense and past participle of token

(transitive, computing) To reduce to a set of tokens by lexical analysis. / (transitive, computing) To substitute sensitive data with meaningless placeholders.

present participle of token

The act or process of marking or designating. / A token, sign, or proof. / That which a thing betokens; meaning, interpretation.

present participle of tokenize

simple past tense and past participle of tokenize

Third-person singular simple present indicative form of tokenize

plural of tokenization