# osintverse-community/xcrawl3r

Crawls a website from a start URL, parsing links, scripts, sitemaps and robots.txt, and lists the URLs it finds

## Install and run

```sh
ot install osintverse-community/xcrawl3r
ot run osintverse-community/xcrawl3r url:https://example.com
ot run osintverse-community/xcrawl3r d:example.com --url https://example.com
```

The first `ot run` installs the tool after asking once. New to `ot`? See https://otinstaller.com/docs/getting-started.

From an AI agent, through ot's MCP server (`ot mcp`):

```
run_tool {"tool":"osintverse-community/xcrawl3r","inputs":{"url":"https://example.com"}}
```

## Inputs

| Name | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `url` | url | yes |  | The URL to start crawling from |
| `domain` | domain | no |  | Only keep URLs on this domain |
| `depth` | integer | no |  | Maximum crawl depth (default 1, 0 means unlimited) |

## Outputs

Finds: url.

## What a run does

- Contacts the target directly (sends requests to sites or services about it).
- Does not collect personal data.
- Does not need an account.
- Does not notify the target.
- Usually takes about 60 seconds.
- Publisher's note: It crawls the site itself, so the web server's logs see your IP address and the crawler's user agent. Keep the depth low on sites you don't own.
- Connects to the internet.

An agent needs the person's consent before running this tool, because it contacts the target. See https://otinstaller.com/docs/agents#consent.

## Trust

- Latest version: 1.2.0
- Signed: yes
- Verified publisher: yes
- Health: unknown
- Installs in the last 30 days: 0
- License: MIT
- Runtime: Docker image, pinned by digest
- Repository: https://github.com/osintverse-community/xcrawl3r
- Packaged by the community from a third-party tool

## Links

- Web page: https://otinstaller.com/osintverse-community/xcrawl3r
- JSON: https://api.otinstaller.com/v1/tools/osintverse-community/xcrawl3r
- README: https://github.com/osintverse-community/xcrawl3r#readme
- Homepage: https://github.com/hueristiq/xcrawl3r
- Security details: https://otinstaller.com/osintverse-community/xcrawl3r/security