osintverse-community/katana
Crawls a website and lists the URLs, endpoints and JavaScript files it finds
Takes url → finds url
No key · Docker
No key · Docker
Verified publisher
Crawl a website for links, scripts and endpoints
7 tools. 7 run with ot in a sandbox on Windows, macOS and Linux. Compare them side by side ↓
A site's own pages link to more than its menus show: forms, API endpoints, JavaScript bundles, staging hosts and third-party services. These crawlers walk a site from a start URL and list what they find, and the JavaScript tools read the scripts themselves for paths and endpoints the pages never link to.
With ot, in a sandbox
All URL tools →Windows, macOS and Linux, with Docker or Podman. Each tool runs in a read-only container that can reach only the sites it declares, and every result comes back in the same JSON shape.
osintverse-community/hakrawler
Crawls a web page and lists the links, scripts and forms it finds
Takes url → finds url
No key · Docker
No key · Docker
Verified publisher
osintverse-community/webpalm
Crawls a website to a set depth and prints the tree of linked pages with their status codes
Takes url → finds url
No key · Docker
No key · Docker
Verified publisher
osintverse-community/xcrawl3r
Crawls a website from a start URL, parsing links, scripts, sitemaps and robots.txt, and lists the URLs it finds
Takes domain, url → finds url
No key · Docker
No key · Docker
Verified publisher
osintverse-community/photon
Crawls a website and extracts its URLs, files, scripts, endpoints and intel such as emails and social accounts
Takes url → finds account, email, url
No key · Docker
No key · Docker
Verified publisher
osintverse-community/getjs
Lists the JavaScript files a web page loads, with relative paths turned into full URLs
Takes url → finds url
No key · Docker
No key · Docker
Verified publisher
osintverse-community/linx
Finds links, endpoints and API paths hidden inside a JavaScript file
Takes url → finds url
No key · Docker
No key · Docker
Verified publisher
ot install osintverse-community/katana osintverse-community/hakrawler osintverse-community/webpalm osintverse-community/xcrawl3r osintverse-community/photonInstalls the first 5 in one go; ot auto url:… then runs every installed tool that takes an URL and saves the results. No ot yet? Install it in one line.
The 7 registry tools side by side
| osintverse-community/ | url | None | Contacts the target | No | Docker | No reports yet | fewer than 10 |
| osintverse-community/ | url | None | Contacts the target | No | Docker | No reports yet | fewer than 10 |
| osintverse-community/ | url | None | Contacts the target | No | Docker | No reports yet | fewer than 10 |
| osintverse-community/ | url | None | Contacts the target | No | Docker | No reports yet | fewer than 10 |
| osintverse-community/ | account, email, url | None | Contacts the target | Yes | Docker | No reports yet | fewer than 10 |
| osintverse-community/ | url | None | Contacts the target | No | Docker | No reports yet | fewer than 10 |
| osintverse-community/ | url | None | Contacts the target | No | Docker | No reports yet | fewer than 10 |
Facts from each tool's manifest and what people report about whether it works. "Reaches" says whether a run stays offline, reads public sources, sends requests about the target, or may notify them.
How to crawl a website for links, scripts and endpoints
- Crawl from the home page to a shallow depth (katana, hakrawler or webpalm) and list the URLs and forms.
- List the JavaScript files the pages load (getjs) and pull the endpoints and paths out of them (linx).
- Compare with the archived URLs on the domain page: old endpoints that still answer are the interesting ones.
Every registry tool here runs on your machine in a sandbox, and ot auto url:… runs all the installed ones on one value.
Before you run one
Is crawling a website allowed?
Crawling public pages at a modest rate is ordinary web traffic, but it does contact the target, and some sites forbid it in their terms. Every row here says it contacts the target; check your scope before you run one.
Crawler or archive?
A crawler sees the site as it is now, including pages no archive captured; archives show what it used to be, without contacting it. Investigations usually want both.
Do these tools send my search anywhere?
No. ot runs each tool on your own machine in a read-only container that can only reach the sites the tool declares, and results stay in your results folder. The registry only counts installs.
Do I need an API key?
Most tools here work without one; the row says "Key required" when one is needed, and ot keys set stores it in your OS keychain so only that tool's container sees it.
More from an URL
Archived copiesEmails and contactsPhishing or malwareAll URL tools