Function Compare<\/h2><\/span><\/div><\/div><\/div>This tutorial tells you how to compare two crawls using Screaming Frog SEO Spider and analyze how crawl data changed in tabs and filters over time.<\/p>\n
Through this activity it is possible to monitor the progress of any optimizations or the progress of critical Seo issues by having useful granular data on what has changed between crawls.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/compare-crawls-seo-spider.jpeg\")
<\/a><\/span><\/div>In summary, this analysis helps you interpret several scenarios:<\/p>\n
\n- Compare crawl data and detect new problems.<\/li>\n
- Compare site structure and changes in directories, pages, and crawl depth between crawls.<\/li>\n
- Detect changes in elements and metrics, such as page titles, meta descriptions, scan depth, word count, internal links, content, and more.<\/li>\n<\/ul>\n<\/div><\/div><\/div><\/span>
Database Storage<\/h3><\/span><\/div><\/div><\/div>\n- 1. Use of database storage mode.<\/li>\n<\/ul>\n
To compare the crawls, it is necessary to store the different scans by selecting the “Database Storage” option.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/database-storage-mode-seo-spider.jpeg\")
<\/a><\/span><\/div>Database storage allows you to automatically save scans, quickly open crawls, and always have multiple scans of a project available directly locally.<\/p>\n<\/div>
<\/div><\/div><\/span>Comparison Crawl<\/h3><\/span><\/div><\/div><\/div>\n- 2. Select the crawls to compare<\/li>\n<\/ul>\n
To begin the comparison, simply select the scans from “File > Crawls,” highlight the two crawls by holding down ‘control’ on Windows, or the ‘command’ key on macOS, and click on “Select to Compare,” to switch from “Spider or “List” mode to “Compare” mode<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/compare-crawls-select-crawls-seo-spider.jpeg\")
<\/a><\/span><\/div>Alternatively, you can choose the “Compare” mode directly from the main menu.<\/p>\n<\/div><\/div><\/div><\/div><\/div>
Mode > Compare<\/span><\/p>\n<\/div><\/div><\/div><\/div><\/div>![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/mode-compare-seo-spider.jpeg\" "\\"mode-compare-seo-spider\\"")
<\/span><\/div><\/div><\/div><\/span>Change Detection<\/h3><\/span><\/div><\/div><\/div>\n- 3. Using Change Detection functionality.<\/li>\n<\/ul>\n
The SEO Spider will compare all the crawl data in the tabs and filters in the “Overviews” tab on the right.<\/p>\n
However, you can choose to configure the tool to perform additional analysis on more specific changes to key elements and metrics-such as page title, meta description, h1, word count, crawl depth, internal links, structured data, content, and more.<\/p>\n
This is useful when an element has changed but has not activated a filter. For example, the title of a page might change completely, but this would not be detected if you did not activate a change filter such as ‘Missing’, ‘Over X Characters’, ‘Duplicates’ etc. Change detection with ‘Change Detection’ lets you know if there has been any change to an element or metric.<\/p>\n
In “Compare” mode, click on the comparison configuration via “Config > Compare” (or the “gear” icon at the top) and select the items and metrics where you want to identify changes.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/change-detection-config-crawl-comparison-spider.jpeg\")
<\/a><\/span><\/div>Enabling these options enables the crawl results to activate the new “Change Detection” tab in the main view and Overview tab, containing all the filters of the selected items and metrics with details of the changes detected.<\/p>\n
If you opt for this element control consider that the Seo Spider requires more resources to process, store page HTML, analyze relative similarities using minhash, and determine the % change between current and previous content.<\/p>\n
\n- 4. Start the comparison scan by clicking “Compare”<\/li>\n<\/ul>\n
Simply click on “Compare” to start scanning between the two crawls according to the “Change Detection” configurations set earlier.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/compare-crawls-compare-button-spider.jpeg\" "\\"compare-crawls-compare-button-spider\\"")
<\/span><\/div>Please note: Seo Spider uses the latest crawl in date order as ‘Current’ and the oldest crawl as ‘Previous’ but you can reverse the selection by clicking on the arrow next to the gear icon at the top of the interface.<\/p>\n<\/div>
<\/div><\/div><\/span>Overview tab<\/h3><\/span><\/div><\/div><\/div>\n- 5. View the “Overview” tab<\/li>\n<\/ul>\n
When the comparison has been completed, the overview tab on the right sidebar populates to show the data and changes noted.<\/p>\n
By clicking on the tabs you can update the graphs below, as well as click on the filters and numbers in the columns that update the data in the main window.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/compare-crawls-sidebar-overview.jpeg\")
<\/a><\/span><\/div>The sidebar has eight columns while the main window uses 4 filters that allow you to segment the URLs that have changed in the tabs and filters.<\/p>\n
\n- Added<\/strong>: identifies all URLs that are present in both scans but have modified elements that are included in some “Current” crawl filter.
\nFor example, a URL had a “meta_title” in the previous version while in the current version this element is missing. In this case this indication will be present in the “Missing” filter of the “Meta Title” element.<\/li>\n- New<\/strong>: includes all new URLs that were not present in the previous scan but are included in the current scan.<\/li>\n
- Removed<\/strong>: URLs present in both scans but with elements present in the filter of the previous scan, but not in the filter of the current scan.
\nTaking the “Meta Title” example above, this filter identifies all URLs that did not have the page title and were included in the “Missing” filter but in the current scan are no longer in the filter because they have been optimized.<\/li>\n- Missing<\/strong>: URLs present in previous crawl but not detected in “Current” crawl<\/li>\n<\/ul>\n
In summary, “added” and “removed” are URLs that exist in both the current and previous scans but differ in the filters. “New” and “missing” are URLs that exist only in one of the crawls. These filters help you determine whether new problems or fixes affect existing URLs or new URLs.<\/p>\n
By clicking on the numbers in the columns you are able to see which URLs have changed, and use the filter in the main window to alternate between current and previous scans, as well as added, new, removed, or missing.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/compare-crawls-page-titles-seo-spider.gif\")
<\/a><\/span><\/div>Data from the “Overview – Overview” sidebar can be exported with the “Crawl Overview” document.<\/p>\n
<\/div>\n<\/div>![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/compare-crawls-crawl-overview.jpeg\")
<\/a><\/span><\/div>\n- 6. Analyze the changes in the “Change Detection” tab.<\/li>\n<\/ul>\n<\/div>\n<\/div><\/span><\/div>
In the above example, there are a total of 2,236 existing URLs in both scans, 1 Url changed the H1, 209 URLs changed the page title, 6 changed the scan depth, etc.<\/p>\n
To interact with this data, simply click on an item and view the changes in the main window. For example, if you want to view changes in page titles, you can click on the element and compare them side by side in the main tab.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/compare-crawls-change-detection-page-title.jpeg\")
<\/a><\/span><\/div>Another example of application would be to assess whether the architecture of the site has changed relative to “Crawl Depth.”<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/compare-crawls-change-detection-crawl-depth.jpeg\")
<\/a><\/span><\/div><\/div><\/div><\/span>Site structure analysis<\/h3><\/span><\/div><\/div><\/div>\n- 7. View the “Site Structure” tab to examine the directories<\/li>\n<\/ul>\n
The “Site Structured” tab shows you a tree view of the website directories and can be useful for identifying how the site architecture has evolved between scans by identifying any directory changes.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/compare-crawls-site-structure-sito-web.jpeg\")
<\/a><\/span><\/div>You can also click and expand directories to see which specific URLs have changed, as in the example in the reviews directory. Green upward arrows and numbers indicate URLs that have been added, while red downward arrows and numbers indicate URLs that are now missing.
\nIn our example, the reviews directory presents 3 new URLs.<\/p>\n<\/div>
<\/div><\/div><\/span>Bottom Tab Analysis<\/h3><\/span><\/div><\/div><\/div>\n- 8. Compare the data in the lower window tabs<\/li>\n<\/ul>\n
Another opportunity for comparison is to consult the lower window of the Seo Spider where you can view scan data by comparing them side by side.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/compare-crawls-finestra-inferiore-url-details-tab.jpeg\" "\\"compare-crawls-finestra-inferiore-url-details-tab\\"")
<\/span><\/div>Very useful for example to compare a crawl of a site in JavaScript and its behavior on desktop and mobile to display mobile and desktop rendering side by side.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/compare-crawls-rendered-page-scheda.jpeg\" "\\"compare-crawls-rendered-page-scheda\\"")
<\/span><\/div>Also in the lower window you can benefit (Tab inlink and outlink) from the additional filter “Auto” which by default shows you link data based on the upper window view.<\/p>\n
<\/div>\n<\/div>![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/compare-crawls-finestra-infeiore-filters.jpeg\")
<\/a><\/span><\/div>In our example, the Seo Spider shows link data for the “current” crawl, as selected from the upper main window. However, the lower window can be set independently to show data from the “previous” scan, even if the upper window is selected for the current scan.<\/p>\n
You will also be able to use the “Added” and “Removed” filters and highlight the relevant URLs to infer where internal links have changed.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/compare-crawls-finestra-inferiore-inlink-changes-1.gif\" "\\"compare-crawls-finestra-inferiore-inlink-changes-1\\"")
<\/span><\/div><\/div><\/div><\/span>Url Mapping<\/h3><\/span><\/div><\/div><\/div>So far we have always compared two subdomains with the same URL, but using the “URL Mapping” function it is also possible to compare scans between different subdomains. A very common application may involve comparing a live site to a staging site, or a mobile site to its desktop equivalent. Through the latest updates to Screaming Frog, it is possible to compare completely different host names, directories, or more subtle changes to URLs.<\/p>\n
To compare two different URL structures, click on the comparison configuration (“Config > Compare”) and “URL Mapping” and use RegEx rules for matching.<\/p>\n<\/div><\/div><\/div><\/div><\/div>
EXAMPLE:
\n<\/span><\/p>\nWe see how Screaming Frog changed hosts to Kinsta and tested the staging site before migration with URL Mapping.<\/span><\/p>\n<\/div><\/div><\/div><\/div><\/div>![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/crawl-comparison-url-mapping-seo-spider.jpeg\")
<\/a><\/span><\/div>In this case, the URLs of the staging site on Kinsta have a different hostname and through mapping were matched to the existing live site to allow comparison of the data.<\/p>\n
Another scenario for URL mapping could be to compare mobile and desktop websites if responsive versioning is not provided.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/compare-crawls-mobile-sottocartella.jpeg\")
<\/a><\/span><\/div>In the example above, the Chain Reactions Cycles website has a \/mobile\/ <\/strong>subfolder dedicated to the smartphone version, so to compare the mobile website with the desktop website, the mobile URLs were mapped to the desktop equivalent with the following URL Mapping.<\/p>\nFinally, if the two versions to be compared differ only by the final slash in the subdomain you can use “\/$” in the URL mapping.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/compare-crawls-trailing-slash-to-no-trailing-slash-url.jpeg\")
<\/a><\/span><\/div>Before applying Url Mapping you can also test using the dedicated tab to avoid making mistakes and wasting time.<\/p>\n<\/div>
<\/div><\/div><\/span>Crawl Compare<\/h3><\/span><\/div><\/div><\/div>Once the comparison is complete the data will be saved automatically and you can use it again by going to “File > Crawls.”<\/p>\n
Scans are identified with the item “Compare.”<\/p>\n<\/div>
This tutorial tells you how to compare two crawls using Screaming Frog SEO Spider and analyze how crawl data changed in tabs and filters over time.<\/p>\n
Through this activity it is possible to monitor the progress of any optimizations or the progress of critical Seo issues by having useful granular data on what has changed between crawls.<\/p>\n<\/div>
<\/a><\/span><\/div>In summary, this analysis helps you interpret several scenarios:<\/p>\n
- \n
- Compare crawl data and detect new problems.<\/li>\n
- Compare site structure and changes in directories, pages, and crawl depth between crawls.<\/li>\n
- Detect changes in elements and metrics, such as page titles, meta descriptions, scan depth, word count, internal links, content, and more.<\/li>\n<\/ul>\n<\/div><\/div><\/div><\/span>
Database Storage<\/h3><\/span>
<\/div><\/div><\/div>- \n
- 1. Use of database storage mode.<\/li>\n<\/ul>\n
To compare the crawls, it is necessary to store the different scans by selecting the “Database Storage” option.<\/p>\n<\/div>
<\/a><\/span><\/div>Database storage allows you to automatically save scans, quickly open crawls, and always have multiple scans of a project available directly locally.<\/p>\n<\/div>
<\/div><\/div><\/span>Comparison Crawl<\/h3><\/span>
<\/div><\/div><\/div>- \n
- 2. Select the crawls to compare<\/li>\n<\/ul>\n
To begin the comparison, simply select the scans from “File > Crawls,” highlight the two crawls by holding down ‘control’ on Windows, or the ‘command’ key on macOS, and click on “Select to Compare,” to switch from “Spider or “List” mode to “Compare” mode<\/p>\n<\/div>
<\/a><\/span><\/div>Alternatively, you can choose the “Compare” mode directly from the main menu.<\/p>\n<\/div><\/div><\/div><\/div><\/div>
Mode > Compare<\/span><\/p>\n<\/div><\/div><\/div><\/div><\/div>
<\/span><\/div><\/div><\/div><\/span>Change Detection<\/h3><\/span>
<\/div><\/div><\/div>- \n
- 3. Using Change Detection functionality.<\/li>\n<\/ul>\n
The SEO Spider will compare all the crawl data in the tabs and filters in the “Overviews” tab on the right.<\/p>\n
However, you can choose to configure the tool to perform additional analysis on more specific changes to key elements and metrics-such as page title, meta description, h1, word count, crawl depth, internal links, structured data, content, and more.<\/p>\n
This is useful when an element has changed but has not activated a filter. For example, the title of a page might change completely, but this would not be detected if you did not activate a change filter such as ‘Missing’, ‘Over X Characters’, ‘Duplicates’ etc. Change detection with ‘Change Detection’ lets you know if there has been any change to an element or metric.<\/p>\n
In “Compare” mode, click on the comparison configuration via “Config > Compare” (or the “gear” icon at the top) and select the items and metrics where you want to identify changes.<\/p>\n<\/div>
<\/a><\/span><\/div>Enabling these options enables the crawl results to activate the new “Change Detection” tab in the main view and Overview tab, containing all the filters of the selected items and metrics with details of the changes detected.<\/p>\n
If you opt for this element control consider that the Seo Spider requires more resources to process, store page HTML, analyze relative similarities using minhash, and determine the % change between current and previous content.<\/p>\n
- \n
- 4. Start the comparison scan by clicking “Compare”<\/li>\n<\/ul>\n
Simply click on “Compare” to start scanning between the two crawls according to the “Change Detection” configurations set earlier.<\/p>\n<\/div>
<\/span><\/div>Please note: Seo Spider uses the latest crawl in date order as ‘Current’ and the oldest crawl as ‘Previous’ but you can reverse the selection by clicking on the arrow next to the gear icon at the top of the interface.<\/p>\n<\/div>
<\/div><\/div><\/span>Overview tab<\/h3><\/span>
<\/div><\/div><\/div>- \n
- 5. View the “Overview” tab<\/li>\n<\/ul>\n
When the comparison has been completed, the overview tab on the right sidebar populates to show the data and changes noted.<\/p>\n
By clicking on the tabs you can update the graphs below, as well as click on the filters and numbers in the columns that update the data in the main window.<\/p>\n<\/div>
<\/a><\/span><\/div>The sidebar has eight columns while the main window uses 4 filters that allow you to segment the URLs that have changed in the tabs and filters.<\/p>\n
- \n
- Added<\/strong>: identifies all URLs that are present in both scans but have modified elements that are included in some “Current” crawl filter.
\nFor example, a URL had a “meta_title” in the previous version while in the current version this element is missing. In this case this indication will be present in the “Missing” filter of the “Meta Title” element.<\/li>\n- New<\/strong>: includes all new URLs that were not present in the previous scan but are included in the current scan.<\/li>\n
- Removed<\/strong>: URLs present in both scans but with elements present in the filter of the previous scan, but not in the filter of the current scan.
\nTaking the “Meta Title” example above, this filter identifies all URLs that did not have the page title and were included in the “Missing” filter but in the current scan are no longer in the filter because they have been optimized.<\/li>\n- Missing<\/strong>: URLs present in previous crawl but not detected in “Current” crawl<\/li>\n<\/ul>\n
In summary, “added” and “removed” are URLs that exist in both the current and previous scans but differ in the filters. “New” and “missing” are URLs that exist only in one of the crawls. These filters help you determine whether new problems or fixes affect existing URLs or new URLs.<\/p>\n
By clicking on the numbers in the columns you are able to see which URLs have changed, and use the filter in the main window to alternate between current and previous scans, as well as added, new, removed, or missing.<\/p>\n<\/div>
<\/a><\/span><\/div>Data from the “Overview – Overview” sidebar can be exported with the “Crawl Overview” document.<\/p>\n
<\/div>\n<\/div><\/a><\/span><\/div>- \n
- 6. Analyze the changes in the “Change Detection” tab.<\/li>\n<\/ul>\n<\/div>\n<\/div><\/span><\/div>
In the above example, there are a total of 2,236 existing URLs in both scans, 1 Url changed the H1, 209 URLs changed the page title, 6 changed the scan depth, etc.<\/p>\n
To interact with this data, simply click on an item and view the changes in the main window. For example, if you want to view changes in page titles, you can click on the element and compare them side by side in the main tab.<\/p>\n<\/div>
<\/a><\/span><\/div>Another example of application would be to assess whether the architecture of the site has changed relative to “Crawl Depth.”<\/p>\n<\/div>
<\/a><\/span><\/div><\/div><\/div><\/span>Site structure analysis<\/h3><\/span>
<\/div><\/div><\/div>- \n
- 7. View the “Site Structure” tab to examine the directories<\/li>\n<\/ul>\n
The “Site Structured” tab shows you a tree view of the website directories and can be useful for identifying how the site architecture has evolved between scans by identifying any directory changes.<\/p>\n<\/div>
<\/a><\/span><\/div>You can also click and expand directories to see which specific URLs have changed, as in the example in the reviews directory. Green upward arrows and numbers indicate URLs that have been added, while red downward arrows and numbers indicate URLs that are now missing.
\nIn our example, the reviews directory presents 3 new URLs.<\/p>\n<\/div><\/div><\/div><\/span>Bottom Tab Analysis<\/h3><\/span>
<\/div><\/div><\/div>- \n
- 8. Compare the data in the lower window tabs<\/li>\n<\/ul>\n
Another opportunity for comparison is to consult the lower window of the Seo Spider where you can view scan data by comparing them side by side.<\/p>\n<\/div>
<\/span><\/div>Very useful for example to compare a crawl of a site in JavaScript and its behavior on desktop and mobile to display mobile and desktop rendering side by side.<\/p>\n<\/div>
<\/span><\/div>Also in the lower window you can benefit (Tab inlink and outlink) from the additional filter “Auto” which by default shows you link data based on the upper window view.<\/p>\n
<\/div>\n<\/div><\/a><\/span><\/div>In our example, the Seo Spider shows link data for the “current” crawl, as selected from the upper main window. However, the lower window can be set independently to show data from the “previous” scan, even if the upper window is selected for the current scan.<\/p>\n
You will also be able to use the “Added” and “Removed” filters and highlight the relevant URLs to infer where internal links have changed.<\/p>\n<\/div>
<\/span><\/div><\/div><\/div><\/span>Url Mapping<\/h3><\/span>
<\/div><\/div><\/div>So far we have always compared two subdomains with the same URL, but using the “URL Mapping” function it is also possible to compare scans between different subdomains. A very common application may involve comparing a live site to a staging site, or a mobile site to its desktop equivalent. Through the latest updates to Screaming Frog, it is possible to compare completely different host names, directories, or more subtle changes to URLs.<\/p>\n
To compare two different URL structures, click on the comparison configuration (“Config > Compare”) and “URL Mapping” and use RegEx rules for matching.<\/p>\n<\/div><\/div><\/div><\/div><\/div>
EXAMPLE:
\n<\/span><\/p>\nWe see how Screaming Frog changed hosts to Kinsta and tested the staging site before migration with URL Mapping.<\/span><\/p>\n<\/div><\/div><\/div><\/div><\/div>
<\/a><\/span><\/div>In this case, the URLs of the staging site on Kinsta have a different hostname and through mapping were matched to the existing live site to allow comparison of the data.<\/p>\n
Another scenario for URL mapping could be to compare mobile and desktop websites if responsive versioning is not provided.<\/p>\n<\/div>
<\/a><\/span><\/div>In the example above, the Chain Reactions Cycles website has a \/mobile\/ <\/strong>subfolder dedicated to the smartphone version, so to compare the mobile website with the desktop website, the mobile URLs were mapped to the desktop equivalent with the following URL Mapping.<\/p>\n
Finally, if the two versions to be compared differ only by the final slash in the subdomain you can use “\/$” in the URL mapping.<\/p>\n<\/div>
<\/a><\/span><\/div>Before applying Url Mapping you can also test using the dedicated tab to avoid making mistakes and wasting time.<\/p>\n<\/div>
<\/div><\/div><\/span>Crawl Compare<\/h3><\/span>
<\/div><\/div><\/div>Once the comparison is complete the data will be saved automatically and you can use it again by going to “File > Crawls.”<\/p>\n
Scans are identified with the item “Compare.”<\/p>\n<\/div>
- 8. Compare the data in the lower window tabs<\/li>\n<\/ul>\n
- 7. View the “Site Structure” tab to examine the directories<\/li>\n<\/ul>\n
- New<\/strong>: includes all new URLs that were not present in the previous scan but are included in the current scan.<\/li>\n
- Added<\/strong>: identifies all URLs that are present in both scans but have modified elements that are included in some “Current” crawl filter.
- 5. View the “Overview” tab<\/li>\n<\/ul>\n
- 4. Start the comparison scan by clicking “Compare”<\/li>\n<\/ul>\n
- 3. Using Change Detection functionality.<\/li>\n<\/ul>\n
- 2. Select the crawls to compare<\/li>\n<\/ul>\n
- 1. Use of database storage mode.<\/li>\n<\/ul>\n