osintverse-community/waymore
Collects a domain's known URLs from the Wayback Machine, Common Crawl, AlienVault OTX and other archives
Takes domain → finds url
No key · Python wheel
No key · Python wheel
Verified publisher
Find every known URL of a domain
9 tools. 8 run with ot in a sandbox on Windows, macOS and Linux; 1 with otlite on Linux and macOS, no Docker. Compare them side by side ↓
Web archives and crawl indexes remember pages a site has since removed: old login portals, exposed files, API endpoints and their parameters. The tools below ask the Wayback Machine, Common Crawl, AlienVault OTX, urlscan and similar sources for every URL they hold under a domain, so you get a site's history without crawling it yourself.
With ot, in a sandbox
All domain tools →Windows, macOS and Linux, with Docker or Podman. Each tool runs in a read-only container that can reach only the sites it declares, and every result comes back in the same JSON shape.
osintverse-community/gau
Gets every known URL of a domain from the Wayback Machine, Common Crawl, OTX and urlscan
Takes domain → finds url
No key · Docker
No key · Docker
Verified publisher
osintverse-community/urlfinder
Gathers a domain's known URLs passively from web archives and other public sources
Takes domain → finds url
No key · Docker
No key · Docker
Verified publisher
osintverse-community/xurlfind3r
Finds known URLs for a domain from passive sources such as the Wayback Machine, Common Crawl and OTX
Takes domain → finds url
No key · Docker
No key · Docker
Verified publisher
osintverse-community/urx
Collects a domain's known URLs from the Wayback Machine, Common Crawl and AlienVault OTX
Takes domain → finds url
No key · Docker
No key · Docker
Verified publisher
osintverse-community/gogetcrawl
Lists archived URLs for a domain and its subdomains from the Wayback Machine and Common Crawl
Takes domain → finds url
No key · Docker
No key · Docker
Verified publisher
osintverse-community/cdx-toolkit
Lists a domain's pages in Common Crawl's URL index, with each capture's time, status, type and language
Takes domain → finds url
No key · Python wheel
No key · Python wheel
Verified publisher
osintverse-community/xcrawl3r
Crawls a website from a start URL, parsing links, scripts, sitemaps and robots.txt, and lists the URLs it finds
Takes domain, url → finds url
No key · Docker
No key · Docker
Verified publisher
ot install osintverse-community/waymore osintverse-community/gau osintverse-community/urlfinder osintverse-community/xurlfind3r osintverse-community/urxInstalls the first 5 in one go; ot auto d:… then runs every installed tool that takes a domain and saves the results. No ot yet? Install it in one line.
Without Docker, with otlite
All otlite tools →Linux and macOS. otlite installs each one with pip or git into its own virtualenv, pinned to a version it has tested: otlite install paramspider.
The 8 registry tools side by side
| osintverse-community/ | url | None | Online, passive | No | Python | No reports yet | fewer than 10 |
| osintverse-community/ | url | None | Online, passive | No | Docker | No reports yet | fewer than 10 |
| osintverse-community/ | url | None | Online, passive | No | Docker | No reports yet | fewer than 10 |
| osintverse-community/ | url | None | Online, passive | No | Docker | No reports yet | fewer than 10 |
| osintverse-community/ | url | None | Online, passive | No | Docker | No reports yet | fewer than 10 |
| osintverse-community/ | url | None | Online, passive | No | Python | No reports yet | fewer than 10 |
| osintverse-community/ | url | None | Contacts the target | No | Docker | No reports yet | fewer than 10 |
| osintverse-community/ | url | None | Online, passive | No | Docker | No reports yet | fewer than 10 |
Facts from each tool's manifest and what people report about whether it works. "Reaches" says whether a run stays offline, reads public sources, sends requests about the target, or may notify them.
How to find every known URL of a domain
- Pull the known URLs from the archives (waymore or gau) and remove duplicates.
- Filter for what matters to you: file extensions, parameters, admin or API paths.
- Open the interesting ones in the Wayback Machine (waybackpack, waybackpy) to see what they showed and when.
Every registry tool here runs on your machine in a sandbox, and ot auto d:… runs all the installed ones on one value.
Before you run one
Do these tools visit the website?
The archive tools read only from archives and indexes, never from the site. xcrawl3r is the exception in this list: it crawls the live site from a start URL, and its row says it contacts the target.
Why do the tools return different counts?
Each asks a different mix of sources and removes duplicates differently: waymore asks the most archives, cdx-toolkit reads only Common Crawl's index. Running two and merging the lists is common.
Do these tools send my search anywhere?
No. ot runs each tool on your own machine in a read-only container that can only reach the sites the tool declares, and results stay in your results folder. The registry only counts installs.
Do I need an API key?
Most tools here work without one; the row says "Key required" when one is needed, and ot keys set stores it in your OS keychain so only that tool's container sees it.
More from a domain
SubdomainsLook-alike domainsWHOIS recordDNS recordsEmail addressesAll domain tools