First Steps Screaming Frog

Discover the structure of the seo spider and start navigating the opportunities of screaming frog

SEO SPIDER

Overview of the Seo spider console and main areas of interest during Seo Audits.

Main menu

It represents the heart of the console and allows you to manage the most relevant settings that will affect your Seo Audit tenico.

File

You can create a new project, open previous scans in “Database Storage”(Crawl) or “Memory Storage” (Open Recent) mode, manage crawl scheduling (Scheduling), and change default scan configurations by also creating multiple ready-to-use custom profiles.

Scan Mode

It allows you to choose which scanning mode to use.

  • Spider: default mode for a complete scan of a Web site.
  • List: allows you to do specific crawls from lists of URLs that you can manually upload or copy into Seo Spider.
  • Compare: represents the latest feature added by Screaming Frog developers to compare two previously performed scans and analyze the differences in terms of URLs and filters (useful for staging sites vs. live sites, desktop vs. mobile version etc.).
  • Serp: allows you to preview snippets by uploading a tabulated .csv or .xls file.

Configuration

This menu item is among the most important of the Seo Spider and corresponds to the console engine putting at your disposal a whole series of customizations to perform Advanced Seo Audits and links to external services such as Google Analytics, Search Console, Lightspeed, Moz, Ahrefs.

  • Spider: through this option you can manage the behavior of the Spider, control which resources it scans, limit its raids, define the parameters of some filters, and decide whether to scan static HTML or also include Javascript rendering.
  • Content: you can define the area in which the Seo Spider should examine the content, set tolerability indices to consider a piece of content as duplicate, and choose various options regarding the vocabulary to be used for grammar and typing check.
  • Robots.txt: this option allows you to custom define the Robots.txt file by editing it directly in the Seo Spider or what attitude the Spider should have towards this file. In summary, you can decide whether the Seo Spider should follow the guidelines or ignore them for crawling.
  • URL Rewriting: allows you to rewrite URLs before scanning. Very useful for avoiding parameterized URLs in crawl results and other advanced best practices using RegEX syntax.
  • Include/Exclude: These features allow you to include or exclude URLs from crawling by setting RegEX rules that you can customize based on the focus of your analysis.
  • Speed: allows you to define the scanning speed of the Spider Tool; very useful for not overloading the server of the website to be scanned.
  • User Agent: as you know Screaming Frog is a Spider emulator, with this option you can choose which User Agent to scan with choosing from GoogleBot, GoogleBot for smartphone, Baidu for Eastern market, Yandex for Russian market, Yahoo/Bing etc. based on your market and focus.
  • HTTP Header: allows you to configure some HTTP Header options in a customized way.
  • Custom: through this item you are able to scan particular HTML elements and extract them to your liking for very custom analysis. Also very useful for defining the position of Links within the Html document.
  • Api Access: this feature allows you to set up API connections with services such as Google Analytics, Search Console, Ahrefs, Majestic, PageSpeed Insight, and Moz. Useful integrations to obtain comprehensive Seo Audits with granular data.
  • Authentication: allows you to scan a website that requires authentication.
  • System: allows you to choose the mode of storage of scan data.

Bulk Export

(Mass Exports): the items included in this menu allow you to download almost all the most relevant data from your scans.
These documents are in “.csv”, “.xlsx” and “.gsheet” formats. Below are the metrics available to you:

  • Links;
  • Web;
  • Path Type;
  • Security;
  • Response Code;
  • Content;
  • Images;
  • Canonicals;
  • Directives;
  • AMP;
  • Structured Data;
  • Sitemaps;
  • Custom Search;
  • Custom Extraction;

Report

This menu item allows you to find a whole series of reports that focus on redirects, canonicals, website pageviews, any content that is not considered safe, orphan page detection, structured data, and a lot of other useful information to export and manage during Seo optimization.

Visualisation

This feature allows you to visualize data regarding URLs in a very intuitive and impactful way according to different graphical modes. Views are interactive and ideal for focusing on different nodes or folders in the site structure and providing more specific and timely control.

Crawl Analysis

Crawl analysis is a new feature introduced by Screaming Frog in the latest Seo Spider updates. Through crawl analysis you are able to analyze, at the end of the crawl, metrics that require further calculation on the populated data (e.g. “Link Metrics”, some aspects of the Sitemap etc.).

From the menu you can start Crawl Analysis and/or configure which items to apply it to and/or automate at the end of the crawl.

Licensing/Help

They correspond to the last menu items and are dedicated to entering the Screaming Frog license and links for requesting insights and support.

Superior Tab

This section includes a whole series of tabs that are populated according to different analysis intents. It includes, for example, collecting URLs based on Response Code, Meta Description and Title, Directives etc.

Each tab has been commented on in later sections of this guide.

Upper Window

It corresponds to the main screen of Seo Spider and is populated by the URLs discovered during crawling with all related details. Remember that each URL can be filtered through the “Data Filters” in the different upper tabs.

Lower Tab

This area represents a deepening to the results obtained in the upper window and is populated whenever you select one or more URLs. from it. In this section you will find details about Inlinks, Outlinks, details about images, and more. As in the previous case, each individual Tab has been documented in detail on the following pages.

Sidebar

This area of the Seo Spider allows you a very interesting preliminary and aggregated view of the scan performed. In the first tab you find an overview with absolute and relative data of the individual elements scanned, in the second you can evaluate the architecture of the site according to “Crawl Depth,” in the third the aggregate page load times by time intervals, in the fourth the scan progress with API to third-party services, and in the last a summary of errors found in the content.

Screaming Frog Overview

User Interface Seo Spider

Configurations and Profiles

Seo Spider Tab