Scheduled Crawls.

The Scheduling feature

Once you understand the potential of the “Scheduling” feature, all that’s left is to configure it by following the steps below.

Schedule the crawl

You create a scheduled crawl from 1File›2Scheduling (shortcut ⌘T on Mac).

12

1FileThe menu with save, import, export and scheduling.2SchedulingOpens the scheduled crawls window.

Click Add to create a new task. In the General tab you can set some general aspects of the crawl:

• Task Name: a descriptive name for the crawl.
• Project Name: the project the crawl belongs to.
• Description: very useful to describe in words all the specifics and options of the crawl.
• Date/Time: the date and time of the crawl and how often it runs (once, daily, weekly or monthly).

12345

1GeneralThe first tab of the scheduled task.2Task NameThe task name, also used in the names of exported files.3Project NameThe project folder where crawls are saved.4Date/TimeDate and time of the first crawl.5FrequencyOnce, Daily, Weekly or Monthly.

SEO Spider configuration

In the second tab, Spider, you can choose between Spider and List crawl mode, enter the root domain or subdomain in the Crawl Seed field and choose which configuration to apply by loading a saved configuration file (if you have one).

1234

1SpiderThe tab with the crawl start options.2Crawler ModeSpider for a whole site, List for a list of URLs.3Crawl SeedThe URL the crawl starts from.4Crawl Config and Auth ConfigThe configuration and authentication files to apply.

The Spider tab also has the Headless Mode option: the crawl runs without the interface, to avoid unwanted user interaction, and it’s required if you want to export data or receive notifications.

12

1Headless ModeRequired for exports and notifications.2Crawl SeedRemember to enter the URL to crawl.

APIs

In the APIs tab you can enrich the crawl with data from Google Analytics 4, Search Console and PageSpeed Insights, link metrics from Majestic, Ahrefs and Moz, and the AI model APIs (OpenAI, Gemini, Ollama and Anthropic) for a more granular view of the project.

1234

1APIsThe task’s integrations tab.2GA4 and Search ConsoleTraffic and search data, also for Orphan Pages.3PageSpeed and link metricsPageSpeed Insights, Majestic, Ahrefs and Moz.4AIOpenAI, Gemini, Ollama and Anthropic.

Saving the data

In the Exports tab you can define the tabs, “Bulk Exports” and reports you want to export. The tab is split into the Output Settings, Spreadsheets, Local Files, Data Studio and Presets pages. Remember that exports only work with Headless Mode on.

123

1ExportsThe tab with all the export options.2Output SettingsLocal folder, Google Drive account and format.3Data StudioThe Crawl Overview export for the Data Studio report.

In Output Settings you choose the local folder, the Google Drive account and the file format. Data is delivered as “.csv” (or other formats such as xlsx) and saved locally, or as “gsheet”: with this second option you’ll find the exported documents in the “Screaming Frog SEO Spider” folder on Google Drive (you’ll need to have connected your Google account).

1234

1Output SettingsThe first page of the Exports tab.2Google Drive AccountThe account where the Google Sheet for Data Studio is saved.3Output ModeA dated folder for each crawl, or overwrite the files.4Spreadsheet FormatThe format of the exported files.

Another very handy “Scheduling” option for recurring crawls is Output Mode: you can choose whether, at the end of each scheduled crawl, to automatically overwrite the original files so only the latest crawl is kept, or to create a “timestamped” folder that adds the new documents and keeps a history of your crawls.

On the Spreadsheets page you choose which tabs, Bulk Exports and reports to export: you can download just one specific tab (e.g. “Internal”, “Meta Description” or “H1”) or combine several tabs and reports.

Note: if you use “Database Storage” mode, the scheduled crawl is always available to reopen later.

If several scheduled crawls overlap, Screaming Frog runs them at the same time. This needs more system resources, so crawls may take longer and your computer may slow down.

Video tutorial (in Italian)

Video tutorialScreaming Frog: how to run a scheduled SEO auditWatch the video3:15