CLI wrapper

Python wrappers around the Screaming Frog CLI executable. Start crawls, export crawl data, and manage spider configuration from Python or automation scripts.

Overview

The CLI wrapper functions call the Screaming Frog CLI (ScreamingFrogSEOSpiderCli) as a subprocess. They resolve the CLI path automatically from standard install locations or from the SCREAMINGFROG_CLI environment variable.

from screamingfrog import start_crawl, export_crawl

# Start a new crawl
start_crawl(
    "https://example.com",
    "./out",
    save_crawl=True,
    export_tabs=["Internal:All", "Response Codes:All"],
)

# Export from an existing .seospider or .dbseospider file
export_crawl(
    "./crawl.seospider",
    "./exports",
    export_tabs=["Internal:All", "Page Titles:Missing"],
)

---

Environment variables

Set SCREAMINGFROG_CLI to the full path of the CLI executable when it is not in a standard install location.

export SCREAMINGFROG_CLI="/opt/screamingfrog/ScreamingFrogSEOSpiderCli"

Standard paths searched automatically:

• Windows: C:\Program Files (x86)\Screaming Frog SEO Spider\ScreamingFrogSEOSpiderCli.exe
• macOS: /Applications/Screaming Frog SEO Spider.app/Contents/MacOS/ScreamingFrogSEOSpider
• Linux: /usr/bin/screamingfrogseospider, /usr/local/bin/screamingfrogseospider

---

start_crawl()

Start a new crawl from a URL via the Screaming Frog CLI.

from screamingfrog import start_crawl

result = start_crawl(
    "https://example.com",
    "./out",
    config="./my.seospiderconfig",
    save_crawl=True,
    export_tabs=["Internal:All", "Response Codes:All"],
    bulk_exports=["All Inlinks"],
)
print(result.returncode)

start_url · str · required

The URL to crawl.

output_dir · str | Path · required

Directory where exports and crawl files are saved. Created if it does not exist.

cli_path · str | None · default: "None"

Path to the CLI executable. Falls back to SCREAMINGFROG_CLI and standard install locations.

config · str | Path | None · default: "None"

Path to a .seospiderconfig file to use for this crawl.

auth_config · str | Path | None · default: "None"

Path to an auth config file.

export_tabs · Sequence[str] | None · default: "None"

Tabs to export (e.g. ["Internal:All", "Page Titles:Missing"]). When None, no tabs are exported unless export_profile is set.

bulk_exports · Sequence[str] | None · default: "None"

Bulk exports to run (e.g. ["All Inlinks", "All Outlinks"]).

save_reports · Sequence[str] | None · default: "None"

Reports to save.

export_format · str · default: "csv"

Export file format passed to --export-format.

headless · bool · default: "True"

Run in headless mode (--headless).

overwrite · bool · default: "True"

Pass --overwrite to overwrite existing export files.

save_crawl · bool · default: "False"

Pass --save-crawl to save the crawl as a .seospider file.

timestamped_output · bool · default: "False"

Pass --timestamped-output to suffix output files with a timestamp.

task_name · str | None · default: "None"

Task name passed to --task-name.

project_name · str | None · default: "None"

Project name passed to --project-name.

extra_args · Sequence[str] | None · default: "None"

Additional raw CLI arguments appended to the command.

Returns subprocess.CompletedProcess[str]

Raises RuntimeError if the CLI exits with a non-zero return code.

---

export_crawl()

Export data from an existing crawl file (.seospider or .dbseospider) using --load-crawl.

from screamingfrog import export_crawl

export_dir = export_crawl(
    "./crawl.seospider",
    "./exports",
    export_tabs=["Internal:All", "Page Titles:Missing"],
    bulk_exports=["All Inlinks"],
)
print(export_dir)

load_target · str · required

Path to the crawl file to load (.seospider or .dbseospider).

export_dir · str | Path | None · default: "None"

Directory to write exports. A temporary directory is created when None.

cli_path · str | None · default: "None"

Path to the CLI executable.

export_tabs · Sequence[str] | None · default: "None"

Tabs to export. Defaults to ["Internal:All"].

bulk_exports · Sequence[str] | None · default: "None"

Bulk exports to run.

save_reports · Sequence[str] | None · default: "None"

Reports to save.

export_format · str · default: "csv"

Export file format.

headless · bool · default: "True"

Run in headless mode.

overwrite · bool · default: "True"

Overwrite existing export files.

force · bool · default: "False"

Force re-export even if exported files already exist in export_dir.

export_profile · str | None · default: "None"

Named export profile (e.g. "kitchen_sink"). When set, export_tabs and bulk_exports are taken from the profile unless explicitly overridden.

Returns Path — the export directory path.

---

run_cli()

Run the Screaming Frog CLI with arbitrary arguments.

from screamingfrog.cli import run_cli

result = run_cli(["--version"])
print(result.stdout)

args · Sequence[str] · required

CLI arguments. If the first element is not the CLI executable path, it is prepended automatically.

cli_path · str | None · default: "None"

Override the CLI executable path.

check · bool · default: "True"

Raise RuntimeError on non-zero exit code.

Returns subprocess.CompletedProcess[str]

---

resolve_cli_path()

Resolve the Screaming Frog CLI executable path.

Lookup order:

1. cli_path argument
2. SCREAMINGFROG_CLI environment variable
3. Standard install paths for the current platform
4. PATH (via shutil.which)

from screamingfrog.cli import resolve_cli_path

path = resolve_cli_path()
print(path)

cli_path · str | None · default: "None"

Explicit path to the CLI. When provided, this is tried first.

Returns Path

Raises RuntimeError if the CLI cannot be found. Provide cli_path or set SCREAMINGFROG_CLI.

---

resolve_spider_config()

Resolve the spider.config file path used by Screaming Frog.

Lookup order:

1. config_path argument
2. SCREAMINGFROG_SPIDER_CONFIG environment variable
3. ~/.ScreamingFrogSEOSpider/spider.config
4. %APPDATA%/ScreamingFrogSEOSpider/spider.config (Windows)

from screamingfrog.cli import resolve_spider_config

path = resolve_spider_config()
print(path)

config_path · str | Path | None · default: "None"

Explicit path to spider.config.

Returns Path

---

ensure_storage_mode()

Context manager that temporarily forces storage.mode to a given value in spider.config. Restores the original content on exit.

from screamingfrog.cli import ensure_storage_mode

with ensure_storage_mode("DB"):
    # spider.config has storage.mode=DB for this block
    export_crawl("./crawl.seospider", "./exports")

# spider.config is restored to its original state

mode · str · default: "DB"

Storage mode value to write (e.g. "DB", "STORE_ON_DISK").

config_path · str | Path | None · default: "None"

Explicit path to spider.config. Resolved via resolve_spider_config() when None.

Returns context manager — yields the resolved Path to spider.config.

Used internally by Crawl.from_seospider() when ensure_db_mode=True (the default). You normally do not need to call this directly unless you are managing the CLI yourself.