tokenicer
A (nicer) tokenizer you want to use for model `inference` and `training`: with all known peventable `gotchas` normalized or auto-fixed.
tokenicer has been downloaded 241,417 times in total on PyPI, including 19,328 in the last 30 days. The latest version is 0.0.5, released Sep 4, 2025.
Version0.0.5
Downloads
241.42k
License—
AuthorModelCloud
UpdatedSep 4, 2025
Downloads
Weekly, last 90d.
Includes CI traffic.
VersionsTotal0.*
Range
View
Granularity
Group by
CI traffic
Stack: OffCI: Included2 / 15 series
Selected total140.00k
241.4kAll-time
19.3kLast 30 days
756Last 24 h
<0.01/sPer second
Sponsored
Sponsorships keep pepy free to read
Version distribution
Share of downloads by released version. Computed over the last quarter.
- 0167.3%
0.0.13
47.1k downloadsDownloads47.1k67.3% - 0225.9%
0.0.4
18.1k downloadsDownloads18.1k25.9% - 032.3%
0.0.12
1.6k downloadsDownloads1.6k2.3% - 041.4%
0.0.8
953 downloadsDownloads9531.4% - 050.7%
0.0.5
469 downloadsDownloads4690.7% - 060.5%
0.0.6
315 downloadsDownloads3150.5% - 070.4%
0.0.11
314 downloadsDownloads3140.4% - 080.3%
0.0.9
213 downloadsDownloads2130.3% - 091.3%
Other
911 downloadsDownloads9111.3%
Guess the next day
Thirteen recent days of tokenicer downloads. Drag the green handle on the right to guess where day fourteen lands.