tokenicer
A (nicer) tokenizer you want to use for model `inference` and `training`: with all known peventable `gotchas` normalized or auto-fixed.
tokenicer has been downloaded 259,712 times in total on PyPI, including 23,578 in the last 30 days. The latest version is 0.0.5, released Sep 4, 2025.
Version0.0.5
Downloads
259.71k
License—
AuthorModelCloud
UpdatedSep 4, 2025
Downloads
Weekly, last 90d.
Includes CI traffic.
VersionsTotal0.*
Range
View
Granularity
Group by
CI traffic
Stack: OffCI: Included2 / 16 series
Selected total140.99k
259.7kAll-time
23.6kLast 30 days
891Last 24 h
<0.01/sPer second
Sponsored
Sponsorships keep pepy free to read
Version distribution
Share of downloads by released version. Computed over the last quarter.
- 0157.5%
0.0.13
40.1k downloadsDownloads40.1k57.5% - 0221.4%
0.0.4
14.9k downloadsDownloads14.9k21.4% - 0313.1%
0.0.14
9.1k downloadsDownloads9.1k13.1% - 043.0%
0.0.12
2.1k downloadsDownloads2.1k3.0% - 051.9%
0.0.11
1.3k downloadsDownloads1.3k1.9% - 060.8%
0.0.8
526 downloadsDownloads5260.8% - 070.6%
0.0.5
397 downloadsDownloads3970.6% - 080.4%
0.0.6
273 downloadsDownloads2730.4% - 091.4%
Other
973 downloadsDownloads9731.4%
Guess the next day
Thirteen recent days of tokenicer downloads. Drag the green handle on the right to guess where day fourteen lands.