Last Updated
：2025/11/23
tokenize
IPA(Pronunciation)
Verb
transitive
Japanese Meaning
字句解析を通じて、入力されたテキストなどをトークン（基本要素）の集合に分解する処理を行うこと。 / 機密情報などの敏感なデータを意味を持たないプレースホルダに置換する処理を行うこと。
（ present singular third-person ）
（ participle present ）
（ participle past ）
（ past ）
Quizzes for review
(transitive, computing) To reduce to a set of tokens by lexical analysis. / (transitive, computing) To substitute sensitive data with meaningless placeholders.
音声機能が動作しない場合はこちらをご確認ください
See correct answer
tokenize
Before parsing, the compiler will tokenize the source code into a stream of tokens for syntactic analysis.
See correct answer
Before parsing, the compiler will tokenize the source code into a stream of tokens for syntactic analysis.
音声機能が動作しない場合はこちらをご確認ください
English - English
Word Edit Setting
- Users who have edit permission for words - All Users
- Screen new word creation
- Screen word edits
- Screen word deletion
- Screen the creation of new headword that may be duplicates
- Screen changing entry name
- Users authorized to vote on judging - Editor
- Number of votes required for decision - 1
Sentence Edit Setting
- Users who have edit permission for sentences - All Users
- Screen sentence deletion
- Users authorized to vote on judging - Editor
- Number of votes required for decision - 1
Quiz Edit Setting
- Users who have edit permission for quizzes - All Users
- Users authorized to vote on judging - Editor
- Number of votes required for decision - 1
