Crawl Audit automated data Studio

Learn how to configure Screaming Frog SEO Spider to process automated audits for Google Data Studio.


Automate Audit Seo

This guide tells you how to configure Screaming Frog SEO Spider to create fully automated crawl reports for Google Data Studio and be able to monitor the health of your site, detect problems, and track performance. web.

By linking a scheduled scheduled crawl to a Google Drive account the Seo Spider can update the Overview report data after every single crawl allowing you to integrate the data and give a historicity to your analysis.

Scheduling Function

1. Schedule the Scan

At this first stage you can define some general aspects of the crawl:

  • Name: descriptive name of the scan.
  • Project Name: name of the project in which to place the scan.
  • Description: very useful for textually defining all the peculiarities and options that will characterize the crawl.
  • Date/Time: the date, time of the scan and its periodicity (one-time, daily, weekly or monthly).

In the second tab “Start Options” you can choose the crawl mode between “spider” and “list,” enter the “root domain” or subdomain, and define which configuration to apply to the crawl, choosing from the various “Crawl Profiles” (if any).

In this tab you can also integrate through API scanning with data from Google Analytics, Search Console for a more granular view of the project and to define possible Orphan Pages.

From the Export Tab you must select “Headless” to export the data for Data Studio.

Select your Google account from the drop-down menu or, if you have not yet used Google Drive storage, select “Manage” and “Add.” To complete this step you will need to authorize Screaming Frog to access your Google account.

Crawl Setup for Data Studio

  • 2. Set crawl by date Studio

For automated Data Studio reporting, check the “Custom Crawl overview” option and click the “Configure” button.

In this panel you can customize the “Overview” report information you wish to include in the Google Sheets export. By default, I recommend that you select all available metrics and add them to the selected box on the right side. This can be done instantly by clicking the double arrow on the right.

Once the Seo Spider setup is completed and the scheduled scan is performed you will have your document in a Google Spreedsheet folder.

Copy to Clipboard

Following several scans you will have a complete document like the following.

Data Studio Connection

  • 3. Connect to Data Studio

In Data Studio you can decide which metrics and dimensions to use for your report in a completely customized way, but if you prefer, you can rely on the template provided by Screaming Frog at first. Download:
Screaming Frog Data Studio Template

Let’s see together how to use the template.

First, you need to create a copy of the template

Once you have created your copy, Data Studio allows you to create your new “Data Source”

In case you decide to start from scratch and design your own template you can choose the new data source directly from the main console screen.

In both the former and the latter case once you click on “Data Source” the Google console will present you with a whole series of connectors. In your case you will need to choose “Google Sheets” and then your scheduled scan file.

The file has the following filename by default:

Copy to Clipboard

[task_name]: corresponds to the name assigned to the scheduled scan in Seo Spider

After uploading the file to Data Studio, the console will present you with all the fields you entered; I recommend renaming the connector (top left) to have a clear reference of the data source in future tasks as well.
Unfortunately, some times Data Studio will mark various fields as “dates” rather than “Numbers.” The advice is to reorder the fields by “Type” and set all fields as “Number” leaving out only the date-specific field.

All you have to do is click “Add to Report” in the upper right corner and “Copy Report” to permanently import the file into Google Data Studio.

The Template is very comprehensive and includes the following analysis sheets:

  • Summary: General summary of the health of the website.
  • Response Codes: displays counts of “Response Codes” and blocked URLs.
  • URL Types: identifies the number of HTML pages, images, Javascript files, etc.
  • Indexability: monitors indexability of the entire site, easily identifies trends or increases in non-indexable URLs.
  • Site Structure: identifies changes in the structure of the website.
  • On-Page: displays changes in Meta tags or headings.
  • Content Issues: identifies changes to page content, duplicate page counts, or spelling problems.
  • PageSpeed: tracks CWV performance and identifies opportunities to improve the page experience.
  • Structured Data: monitors validation issues and the use of structured data throughout the site.
  • Security: analyzes site-wide security problems and insecure use of HTTP.
  • Hreflang: identifies the hreflang attribute and related criticality.
  • Sitemaps: shows errors in Sitemap validation, any orphan pages, and URLs not present in the sitemap.
  • JavaScript: analyzes the use of JavaScript and its impact on metadata, content, and internal links.

Remind Email

  • 4. Set an Email Reminder

Data Studio allows you to schedule email alerts to stay up-to-date on the health of your website. To set up this feature you simply turn on “Schedule Email Delivery.”

Copy to Clipboard

“Scheduled Email Delivery” allows you to set the recipient(s), the date or periodicity of sending, and the time at which you want to receive the information.

Instructions for use:

  1. For periodic scans remember that the device must be online and not in “Standby” mode otherwise the scan will not be performed.
  2. The page dedicated to PSI requires changing the source domain, which at the time of the template copy is set to the Screaming Frog site. To do this you can click on the graphs and change the parameter.
  3. Some times custom template fields do not show up in the copy. For example, a custom field is “not 200 URLs.” in which case Google Data Studio might label the field “Record count.” In this scenario click on the affected graph and add custom fields with tailored formulas. In this specific case go to “add Metric > Create Field” and enter the following formula.

Automated Data Studio Crawl


Seo Spider Tab