pepy.tech

protego

0.4.0
50.05M

Pure-Python robots.txt parser with support for modern conventions