pepy.tech

tokenicer

A (nicer) tokenizer you want to use for model `inference` and `training`: with all known peventable `gotchas` normalized or auto-fixed.

tokenicer has been downloaded 228,347 times in total on PyPI, including 23,583 in the last 30 days. The latest version is 0.0.5, released Sep 4, 2025.

Version0.0.5
Downloads
228.35k
License—
AuthorModelCloud
UpdatedSep 4, 2025
PyPI →

Downloads

Weekly, last 90d.
Includes CI traffic.

VersionsTotal0.*
Range
View
Granularity
Group by
CI traffic
Stack: OffCI: Included2 / 15 series
Selected total148.19k
228.3kAll-time
23.6kLast 30 days
886Last 24 h
<0.01/sPer second

Version distribution

Share of downloads by released version. Computed over the last quarter.

  • 01

    0.0.13

    38.0k downloads
    51.8%
  • 02

    0.0.4

    21.6k downloads
    29.5%
  • 03

    0.0.12

    9.4k downloads
    12.9%
  • 04

    0.0.8

    1.3k downloads
    1.8%
  • 05

    0.0.5

    590 downloads
    0.8%
  • 06

    0.0.6

    521 downloads
    0.7%
  • 07

    0.0.11

    395 downloads
    0.5%
  • 08

    0.0.9

    255 downloads
    0.3%
  • 09

    Other

    1.1k downloads
    1.6%

Guess the next day

Thirteen recent days of tokenicer downloads. Drag the green handle on the right to guess where day fourteen lands.

TRUTH886629THUFRISATSUNMONTUEWEDTHUFRISATSUNMONTUE