branches with status:
Name Status Last Modified Last Commit
lp:mechanize 1 Development 2021-12-19 16:32:01 UTC
1023. Handle mal-encoded robots.txt files m...

Author: Kovid Goyal
Revision Date: 2021-12-19 16:32:01 UTC

Handle mal-encoded robots.txt files more gracefully

Invalid bytes are now ignored. This matches the behavior of the Google
robots.txt parser in the absence of any actual standard. Fixes #68

11 of 1 result