Speed:
1x
Step 1 of 0
BPE Steps
Tokenization Visualization
Token Pair Frequencies
Pair Frequencies
What is this?
No token pairs available
The BPE algorithm selects the most frequent pair (highlighted in yellow) to merge in each step. This creates a new token that replaces all occurrences of that pair.
Vocabulary Statistics
Vocabulary Size
0
unique tokens
Compression Ratio
1.00
original length / current length