# osintverse-community/robofinder

Lists every path a site ever put in robots.txt, from the Wayback Machine's historical snapshots

## Install and run

```sh
ot install osintverse-community/robofinder
ot run osintverse-community/robofinder url:https://example.com
```

The first `ot run` installs the tool after asking once. New to `ot`? See https://otinstaller.com/docs/getting-started.

From an AI agent, through ot's MCP server (`ot mcp`):

```
run_tool {"tool":"osintverse-community/robofinder","inputs":{"url":"https://example.com"}}
```

## Inputs

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `url` | url | yes |  | A site URL or bare domain |

## Outputs

Finds: url.

- `url`: Allow, Disallow and Sitemap paths from old robots.txt files

## What a run does

- Does not contact the target.
- Does not collect personal data.
- Does not need an account.
- Does not notify the target.
- Usually takes about 60 seconds.
- Publisher's note: Only Archive.org is queried, at 2 requests a second by default; the target site sees nothing.
- Connects to the internet.

An agent needs the person's consent before running this tool, because it goes online. See https://otinstaller.com/docs/agents#consent.

## Trust

- Latest version: 0.2.2
- Signed: no
- Verified publisher: yes
- Health: unknown
- Installs in the last 30 days: 0
- License: MIT
- Runtime: Python wheel (pipx), pinned by hash
- Repository: https://github.com/osintverse-community/robofinder
- Packaged by the community from a third-party tool

## Links

- Web page: https://otinstaller.com/osintverse-community/robofinder
- JSON: https://api.otinstaller.com/v1/tools/osintverse-community/robofinder
- README: https://github.com/osintverse-community/robofinder#readme
- Homepage: https://github.com/Spix0r/robofinder
- Security details: https://otinstaller.com/osintverse-community/robofinder/security