tokenicer
A (nicer) tokenizer you want to use for model `inference` and `training`: with all known peventable `gotchas` normalized or auto-fixed.
tokenicer has been downloaded 271,779 times in total on PyPI, including 20,580 in the last 30 days. The latest version is 0.0.5, released Sep 4, 2025.
Version0.0.5
Downloads
271.78k
License—
AuthorModelCloud
UpdatedSep 4, 2025
Downloads
Weekly, last 90d.
Includes CI traffic.
VersionsTotal0.*
Range
View
Granularity
Group by
CI traffic
Stack: OffCI: Included2 / 16 series
Selected total130.24k
271.8kAll-time
20.6kLast 30 days
647Last 24 h
<0.01/sPer second
Sponsored
Sponsorships keep pepy free to read
Version distribution
Share of downloads by released version. Computed over the last quarter.
- 0145.0%
0.0.13
28.9k downloadsDownloads28.9k45.0% - 0227.8%
0.0.14
17.9k downloadsDownloads17.9k27.8% - 0317.6%
0.0.4
11.3k downloadsDownloads11.3k17.6% - 043.8%
0.0.12
2.5k downloadsDownloads2.5k3.8% - 053.0%
0.0.11
1.9k downloadsDownloads1.9k3.0% - 060.6%
0.0.5
393 downloadsDownloads3930.6% - 070.4%
0.0.6
234 downloadsDownloads2340.4% - 080.3%
0.0.8
210 downloadsDownloads2100.3% - 091.4%
Other
925 downloadsDownloads9251.4%
Guess the next day
Thirteen recent days of tokenicer downloads. Drag the green handle on the right to guess where day fourteen lands.