LIST Scan Mode

How to use the 'List' scanning mode with Screaming Frog

List Mode

The Seo Spider provides two scanning modes: the standard one called ‘Spider’ which scans an entire website, and the ‘List’ mode which limits scanning to loading a specific list of URLs.

Let’s see together how to make the most of the second option.

First of all, you need to change the crawling type by clicking on “Mode > List” from the main menu.

At the configuration level, you have to take into account two major differences between the “List” and “Spider” modes:

At the Ux level, the Seo Spider interface changes and shows a load button instead of an address bar.

In the list mode the “Limit Crawl Depth” determines the crawl depth automatically to “0”.

This means that only uploaded URLs are scanned unlike the “Spider” mode which also considers references to images, external links etc.

The only exception is for “Javascript Rendered,” where the Seo Spider needs to scan the page resources to accurately render the page in its Chrome browser.

  • 1. Switching to List Mode.

Having chosen the list mode (Mode > List) from the main menu, you simply click on the ‘Upload’ button and choose to upload your list of URLs from a file, paste a list of URLs, or download an XML Sitemap.

The characteristics you need to consider with this mode are:

URLs entered must include the “http” or “https” protocol otherwise the Seo Spider will not find any URLs to scan.

Keep in mind that Screaming Frog normalizes URLs upon upload, and de-duplicates them during crawl.

For example: you want to upload the following 4 URLs:

  • www.screamingfrog.co.uk/
  • www.screamingfrog.co.uk/#fragment-this-wont-be-crawled
  • www.screamingfrog.co.uk/seo-spider/
  • www.screamingfrog.co.uk/seo-spider/

The Seo Spider automatically determines how many unique URLs to scan. In the example it will only go to consider 2 URLs as unique (e.g., “#” does not consider the “# -fragment” as a separate url and the “/seo-spider” page is duplicated), seems like a trivial example but with lists of thousands of URLs, it may be less obvious.

Once the list has been uploaded, the Seo Spider will report that it has found the 4 URLs and normalize them in the dialog box.

  • 2. Data export.

Having obtained your scan data you simply use “Export” from any tab to export your report.

As you will be able to see from the export, the original data will be kept as a reference (Original URL column), but at the same time the data normalized by the crawler will also be brought back.

Given the inherent characteristics on how the Seo Spider reasons with queta mode let’s see together how to exploit its potential with some advanced best practices that allow you to better focus your analysis and save time.

  • 1.Crawling a list of URLs and other related element

A first application may include targeted scanning of URLs uploaded together with 1 or more referential elements such as images, canonicals, hreflangs.

For this option, simply remove the Limit Crawl Depth, which is by default set to ‘0’.

Config > Spider > Limits

By removing the crawl limit, the Seo Spider scans your URL list along with all URLs on the same subdomain to which they link. But this behavior is not exactly what we expect.

The next step is to granularly manage the resources and references to be crawled by going to ‘Config > Spider > Crawl’ and disabling all unnecessary ‘Resource Links’ and ‘Page Links’ and leaving only those relevant to your specific analysis.

Through this step you will be able to focus exactly on the URLs to be analyzed and their most important elements for you without wasting time.

  • 2. Audit of Redirects with List Mode (List).

If you are monitoring redirects in a site migration, it can be especially useful to crawl their destination URLs and any redirect chains encountered.

To perform this Audit you must use the “Always follow redirects” configuration found in “Config > Spider > Advanced.”

Enabling this configuration means that the ‘crawl depth limit’ is ignored and redirects are followed until they reach a non-3XX response (or until your ‘Max Redirects To Follow’ limit is reached under ‘Config > Spider > Limits’). This saves you from having to load multiple lists of target URLs each time to reach the end of the chain.

At the end of the crawl you can use the “All Redirects” report, which will map your entire chain of redirects into a single report.

Seo Spider Tab