tokenicer
A (nicer) tokenizer you want to use for model `inference` and `training`: with all known peventable `gotchas` normalized or auto-fixed.
tokenicer has been downloaded 318,605 times in total on PyPI, including 24,257 in the last 30 days. The latest version is 0.0.14, released Jul 22, 2026. It is distributed under the Apache-2.0 license.
Version0.0.14
Downloads
318.61k
LicenseApache-2.0
AuthorModelCloud
UpdatedJul 22, 2026
Downloads
Weekly, last 90d.
Includes CI traffic.
VersionsTotal0.*
Range
View
Granularity
Group by
CI traffic
Stack: OffCI: Included2 / 16 series
Selected total126.20k
318.6kAll-time
24.3kLast 30 days
458Last 24 h
<0.01/sPer second
Sponsored
Sponsorships keep pepy free to read
Version distribution
Share of downloads by released version. Computed over the last quarter.
- 0173.2%
0.0.14
46.2k downloadsDownloads46.2k73.2% - 0213.1%
0.0.4
8.3k downloadsDownloads8.3k13.1% - 036.1%
0.0.12
3.8k downloadsDownloads3.8k6.1% - 042.1%
0.0.13
1.3k downloadsDownloads1.3k2.1% - 052.0%
0.0.11
1.3k downloadsDownloads1.3k2.0% - 060.6%
0.0.5
400 downloadsDownloads4000.6% - 070.5%
0.0.6
330 downloadsDownloads3300.5% - 080.4%
0.0.8
233 downloadsDownloads2330.4% - 092.0%
Other
1.3k downloadsDownloads1.3k2.0%
Guess the next day
Thirteen recent days of tokenicer downloads. Drag the green handle on the right to guess where day fourteen lands.