robofinder

Lists every path a site ever put in robots.txt, from the Wayback Machine's historical snapshots

Community-packaged from github.com/Spix0r/robofinder: the registry's community pins the upstream release; the tool is its authors' work. Maintain that project? Claim this package.