Dave's Patent Content Factory
Applications
Terms
Paragraphs
Claims
Docket
Logout
About
Term #2,283
Edit Term
Tokenization
View
List Terms
Delete
Claim Term
Application
2380.2.01
US-20150012794-A1
US-20150205664-A1
US-20100023800-A1
US-8737141-A1
US-10157004-B2
US10007433A1
US-9159419-B2
US-10114589-A1
US-10134728-A1
US-20200065270-A1
US-10637533-B2
US-9927986-A1
US-8380915-A1
US-9159419-A1
US-9208071-A1
US-20200098728-A1
US-10643676-A1
US-10468073-B2
US-10283200-A1
US-10461965-B1
US-20130279232-A1
US-8892980-B2
US9632727A1
US10558561A1
US20100023800A1
US7230213A1
OPT-9
FLO-2
FLO-5PROV
ONSO3175(B) - Onsemi378
ONSO3305US - Onsemi346
GTS-3DES
FLO-4
US8762658B2
US8533406B2
US9632727B2
KMN-1PROV
PAT-2
PER-8 PROV
PER-9 PROV
INS-4PROV
HAR-1
CES-16
NXT-5PROV NXT-5, 6, 7, 8
IPP-0051-US14 cross roads
FLO-7PROV
IMI-5PROV
IPP-0050-US35 nextremity
VIL-12
OPT-13
TOY-1
US10998041B1
FSP1845
US6559866B2
Placeholder App
PER-10
KBR-1 1400.2.623
PER-13PROV
PAT-3
US20030023453
RMS-1DES
SMG-1DES
FLO-5
US10318495
US10133662B2
PER-11
US20140066758
VIL-17
PER-17
JBR-1
PER-12
US11056880
US11302645
US20210407565
US11081191
PON-1PROV, 2PROV, 3PROV
PER-33
RMT-1PROV
PER-32
PER-34
MCC-1
FLO-10
PER-14
PER-19
PER-22
PER-18
PER-24
TMC-PAT-1
DAR-2
PER-23
TMC-PAT-4
PER-16
PER-4 DIV1
PER-20
PER-21
BRT-PAT-1
TMC-PAT-5
TMC-PAT-6PROV
BRT-PAT-2-PROV
71212.157.USU1
FPR-PAT-1-PROV
71212.158.USP1
RMT-1
DAR-1PROV
DAR-2PROV
PON-1PROV
PON-2PROV
PON-3PROV
PER-18PROV
TMC-1PROV
TMC-2PROV
PER-13PCT
PER-13
PER-16PROV
PER-14PROV
PER-34PROV
TMC-4PROV
TMC-3
PAS-1PROV
VEH-1
PER-29DES
TEST.001
E2E-TEST.001
TEST-001
TEST-002
TEST-003
TEST-004
ZED006
FSP1011
GAV-PAT-1-PROV
App Docket
Created Date
Full Desc
"Tokenization" refers to a computational process that segments a stream of input data—such as natural language text—into smaller units called tokens, which may be words, sub words, characters, punctuation marks, semantic elements, or symbolic representations suitable for downstream processing. Synonyms for "tokenization" include lexical segmentation, text parsing, linguistic decomposition, sub word encoding, or the like. Tokenization may be performed using rule-based, statistical, or learned approaches, and may involve fixed or variable-length segmentation strategies. Tokenization may operate in conjunction with vocabulary constraints, language-specific heuristics, byte-level encoding techniques, or sub word encoding algorithms such as byte-pair encoding (BPE), WordPiece, or SentencePiece. The tokenization process may include or be preceded by normalization procedures such as case folding, punctuation removal, whitespace trimming, or Unicode canonicalization. Tokenization enables artificial intelligence systems—including large language models, transformer-based models, and other neural network architectures—to represent text in structured or numerical form, thereby facilitating further analysis, embedding, attention modeling, or generation tasks. (Defined in conjunction with ChatGPT 4o Version, July 15, 2025.)
Submit