cdx-toolkit

v0.9.39Verified publisher

Lists a domain's pages in Common Crawl's URL index, with each capture's time, status, type and language

Community-packaged from github.com/commoncrawl/cdx_toolkit: the registry's community pins the upstream release; the tool is its authors' work. Maintain that project? Claim this package.

Run

TerminalAgent

Through ot mcp, an agent calls run_tool with this; ot asks you before anything that goes online or touches a target. Set it up: ot mcp install claude (or cursor, vscode, codex, gemini).

{"tool":"osintverse-community/cdx-toolkit","target":"example.com","input_type":"domain"}

Or just say: "Run osintverse-community/cdx-toolkit on example.com with OTInstaller." The registry's read-only MCP server at api.otinstaller.com/mcp can find tools without installing anything.

ot run osintverse-community/cdx-toolkit d:example.com --format table

ot install osintverse-community/cdx-toolkit

Python wheel, run in a read-only container (Docker optional with --native) · network allowed · no key · up to 10 min. The first ot run installs the tool after asking once. The example value is the registry's own test input, so this command has run. Drop --format table for JSON.

Don't have ot? Install it in one line

macOS / LinuxWindows

curl -fsSL https://get.otinstaller.com/ot | sh

What this script does

Downloads the latest ot release for your machine from GitHub Releases.
Checks its SHA-256 against checksums.txt, and the Ed25519 signature on that file against the key built into the script.
Puts ot in /usr/local/bin or ~/.local/bin. Nothing else is written. Read the script.

irm https://get.otinstaller.com/ot.ps1 | iex

What this script does

Downloads the latest ot release for your machine from GitHub Releases.
Checks its SHA-256 against checksums.txt, and the Ed25519 signature on that file against the key built into the script.
Puts ot in %LOCALAPPDATA%\Programs\ot, and adds it to your PATH. Nothing else is written. Read the script.

Evidence

Security tab →

Pinnedwheel sha256:577ae9a9…04e431 from PyPI. ot re-hashes the file after download and refuses a mismatch.
Signedby pypi:GitHub:commoncrawl/cdx_toolkit/publish_pypi.yaml via pypi.org, verified by the registry on 24 Oct 2026. ot checks the same identity before every install and refuses another signer.
Scanned24 Oct 2026: 0 critical, 0 high, 5 medium, 1 low vulnerabilities; 0 embedded secrets, 0 malware heuristics; SBOM with 9 packages.
Workssmoke test passed on 24 Oct 2026 (exit 0 in 16.3 s, offline sandbox); the nightly health check has not run yet.
Backed upa copy of this wheel is kept by the registry. If PyPI removes it, ot downloads the copy and checks the same hash.
Reviewedosintverse-community is a verified publisher: the registry confirmed who runs it, and a person reviewed its first release against the acceptable-use rules.

From the community

Issues →

Not health-checked yet; 0.9.39 came out today; nobody has said yet whether it works.

0.9.39 is in its first week. Nobody has said whether it works yet.

Used 0.9.39?It's broken for me

What you get back

Result envelope →

{"languages": "eng", "length": "996", "mime": "text/html", "status": "200", "timestamp": "20261005112841", "url": "https://example.com/"}
{"languages": "eng", "length": "999", "mime": "text/html", "status": "200", "timestamp": "20261005151603", "url": "https://example.com/"}
{"languages": "eng", "length": "993", "mime": "text/html", "status": "200", "timestamp": "20261005154750", "url": "http://example.com/"}
{"languages": "eng", "length": "995", "mime": "text/html", "status": "200", "timestamp": "20261005185728", "url": "http://example.com/"}
{"languages": "eng", "length": "998", "mime": "text/html", "status": "200", "timestamp": "20261005220006", "url": "https://example.com/"}
{"languages": "eng", "length": "999", "mime": "text/html", "status": "200", "timestamp": "20261005231354", "url": "https://example.com/"}
{"languages": "eng", "length": "993", "mime": "text/html", "status": "200", "timestamp": "20261005235803", "url": "http://example.com/"}
{"languages": "eng", "length": "996", "mime": "text/html", "status": "200", "timestamp": "20261006030441", "url": "https://example.com/"}
{"languages": "eng", "length": "1004", "mime": "text/html", "status": "200", "timestamp": "20261006033845", "url": "https://www.example.com/"}
{"languages": "eng", "length": "998", "mime": "text/html", "status": "200", "timestamp": "20261006082447", "url": "https://example.com/"}
{"languages": "eng", "length": "1002", "mime": "text/html", "status": "200", "timestamp": "20261006084244", "url": "https://www.example.com/"}
{"languages": "eng", "length": "1000", "mime": "text/html", "status": "200", "timestamp": "20261006124306", "url": "https://example.com/"}
{"languages": "eng", "length": "993", "mime": "text/html", "status": "200", "timestamp": "20261006143312", "url": "http://example.com/"}
{"languages": "eng", "length": "993", "mime": "text/html", "status": "200", "timestamp": "20261006173838", "url": "http://www.example.com/"}
{"languages": "eng", "length": "995", "mime": "text/html",

The start of what version 0.9.39 printed on its test input (jsonl), recorded by the registry on 24 Oct 2026. ot run wraps it in the result envelope.

cdx-toolkit (community package)

This repository packages github.com/commoncrawl/cdx_toolkit for the OTInstaller Registry as osintverse-community/cdx-toolkit. The tool is the work of its authors, under the Apache-2.0 license; this repository only holds the manifest and release workflow.

Report problems with the tool upstream, and problems with the packaging here. If you maintain the upstream project and want to publish it yourself, open an issue or run ot claim osintverse-community/cdx-toolkit: the package is transferred to you.

It installs the upstream project's own release from PyPI, pinned by version and hash; nothing here is rebuilt.

Searches Common Crawl's URL index, which needs no key, for pages on the domain itself (not its subdomains) in the latest crawl, or the latest --crawls. The index is often overloaded: cdx_toolkit retries its 502 and 504 answers, and searching many crawls at once can take minutes.

ot run osintverse-community/cdx-toolkit --domain …

Packaged version: 0.9.39.

What next

cdx-toolkit finds url. Tools that take those:

url → 10 tools ot auto, with a saved result

Other domain tools

All domain tools →

Alternatives to cdx-toolkit that also take a domain; ot auto runs every installed one on the same value.