(transitive, computing) To reduce to a set of tokens by lexical analysis. / (transitive, computing) To substitute sensitive data with meaningless placeholders.

Third-person singular simple present indicative form of tokenize

simple past tense and past participle of tokenize

(computing) A system that parses an input stream into its component tokens.

plural of tokenizer

The act or process of tokenizing. / Something tokenized.

plural of tokenization

present participle of tokenize

simple past tense and past participle of token

present participle of token