tokenicer
A (nicer) tokenizer you want to use for model `inference` and `training`: with all known peventable `gotchas` normalized or auto-fixed.
tokenicer has been downloaded 212,301 times in total on PyPI, including 25,917 in the last 30 days. The latest version is 0.0.5, released Sep 4, 2025.
Version0.0.5
Downloads
212.30k
License—
AuthorModelCloud
UpdatedSep 4, 2025
Downloads
Weekly, last 90d.
Includes CI traffic.
VersionsTotal0.*
Range
View
Granularity
Group by
CI traffic
Stack: OffCI: Included2 / 15 series
Selected total145.31k
212.3kAll-time
25.9kLast 30 days
800Last 24 h
<0.01/sPer second
Sponsored
Sponsorships keep pepy free to read
Version distribution
Share of downloads by released version. Computed over the last quarter.
- 0138.3%
0.0.13
27.5k downloadsDownloads27.5k38.3% - 0231.7%
0.0.4
22.7k downloadsDownloads22.7k31.7% - 0313.1%
0.0.12
9.4k downloadsDownloads9.4k13.1% - 046.2%
0.0.8
4.5k downloadsDownloads4.5k6.2% - 055.9%
0.0.11
4.2k downloadsDownloads4.2k5.9% - 061.2%
0.0.5
885 downloadsDownloads8851.2% - 070.8%
0.0.6
579 downloadsDownloads5790.8% - 080.8%
0.0.9
540 downloadsDownloads5400.8% - 092.0%
Other
1.4k downloadsDownloads1.4k2.0%
Guess the next day
Thirteen recent days of tokenicer downloads. Drag the green handle on the right to guess where day fourteen lands.