How to Migrate Website<\/h2><\/span><\/div><\/div><\/div>This article tells you which is the best solution to discover and solve the most common critical issues during the delicate phase of migration.<\/p>\n
Any migration is always an epic undertaking in order not to negatively impact the organic ranking of the website that is being migrated.<\/p>\n<\/div>
<\/div><\/div><\/span>Pre-migration<\/h3><\/span><\/div><\/div><\/div>Before delving into this process there are essential steps to put in place in the pre-migration phase in which a crawler can help you and which should not be overlooked.<\/p>\n
\n- 1. Perform a Crawl of the current site<\/li>\n<\/ul>\n
First, a full website scan is recommended for:<\/p>\n
Being able to compare it with the staging site;<\/p>\n
Check whether the old URLs have been correctly directed to the new resources.<\/p>\n
Have a backup to start from in case something goes wrong during the migration.<\/p>\n
In summary before migration you need to prepare the ground for a full backup that includes all the most important elements of the current site.<\/p>\n
To take advantage of Screaming Frog at this stage you need to enable scanning for all resources you may need to refer to in the future such as AMP versions, Hreflangs, structured data etc.<\/p>\n
In this shopping list, don’t forget if there are any client-side JavaScript dependencies that require JavaScript rendering and include any other domain, subdomain, or mobile and desktop site variants.<\/p>\n
Always integrate XML Sitemap into the crawl and connect via API to Google Analytics and Search Console to unearth any orphan pages (enable “Crawl New URLs Discovered GA\/GSC).<\/p>\n
Also collect link data from Ahrefs, Majestic, Moz, and performance on query performance with GSC to keep an eye on important pages based on revenue, conversions, traffic, impressions, clicks that allow you not to dissipate the ranking you have gained.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/site-migrations-configurazione.jpeg\" "\\"site-migrations-configurazione\\"")
<\/span><\/div>\n- 2. Export the URLs to be redirected<\/li>\n<\/ul>\n
Very often during migrations the elements that change most frequently are slugs, so it is necessary to collect a complete list of URLs to permanently redirect old pages to the equivalent ones on the new website.<\/p>\n
This step can be relatively straightforward if it is a domain migration and the URL path remains the same, but it becomes complex and risky if the entire URL structure of the website is changing or simply some strategic areas are being rethought based on new Search Intent.<\/p>\n
To get a complete list of existing URLs, simply export them from the crawl run of the live site by clicking on “Export” under the “Internal” tab.<\/p>\n
This mapping activity is one of the crucial steps in your migration to redirect each element in the most appropriate way.<\/p>\n
If comprehensive mapping is not possible, it is a good idea to prioritize top-performing pages based on conversions, user and links collected in step 1.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/migrazione-sito-web-export-internal-urls.jpeg\" "\\"migrazione-sito-web-export-internal-urls\\"")
<\/span><\/div>\n- 3. Pre-migration testing phase<\/li>\n<\/ul>\n
When the new Web site is in “Staging” it is time to start the checks before it goes online for good, and here again the crawler comes to your aid:<\/p>\n
The first activity is to crawl the staging site to identify differences, problems, and opportunities.<\/p>\n
Normally, sites at this stage are inhibited from being scanned by the robots.txt file or password so as not to be considered by the Search Engine Spider. Thanks to Screaming Frog you can force authentication, ignore the robots.txt file or, by changing configurations, manage the scanning of resources left in “noindex” or “nofollow” because they are still under development (see dedicated section “How to: Scan a password-protected site”).<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/migrazione-sito-web-crawl-staging-website-robots-txt.jpeg\")
<\/a><\/span><\/div>Once you have completed the production site scan you can begin the analysis by initially using the “Overview” tab in the sidebar.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/migrazione-sito-web-right-hand-overview-tab.jpeg\" "\\"migrazione-sito-web-right-hand-overview-tab\\"")
<\/span><\/div><\/div><\/div><\/span>site staging vs. live<\/h3><\/span><\/div><\/div><\/div>\n- 4. Comparison of staging and Live site (URL MAPPING).<\/li>\n<\/ul>\n
Now you are in possession of all the data you need: on one side you have scanned the site currently online and on the other the site in staging, all you have to do is compare the two versions to discover any inconsistencies.<\/p>\n
You can do this activity simply with the Seo Spider by using “URL Mapping” of the “Compare” mode and defining from the top bar the two scan versions to compare.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/modalita-scansione-compare.jpeg\" "\\"modalita-scansione-compare\\"")
<\/span><\/div>This opportunity introduced with the latest Seo Spider updates, allows you to compare different URL structures, such as different host names or directories using RegEX rules.<\/p>\n
To use it, simply change the crawl mode to “Compare,” choose the two saved crawls, and click on the “gear” icon in the top bar.<\/p>\n
On the “Crawl Comparison Configuration” popup all you have to do is click on “URL Mapping” and define the old domain with the new one via RegEX rules.<\/p>\n
In the example below, the existing site was mapped with the “Staging” version only for the hostname considering they share the same URL paths.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/migrazione-sito-url-mapping.jpeg\" "\\"migrazione-sito-url-mapping\\"")
<\/span><\/div>Through mapping, equivalent URLs are compared with each other to obtain the data from the overview tab, problems and opportunities, site structure tab, and change detection.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/migrazione-sito-web-url-mapping-crawl-differences.gif\")
<\/a><\/span><\/div>When the comparison is finished, you can click on the columns to see which URLs have changed and use the filter on the main window to alternate between current and previous scans, as well as to check URLs that have been added, missing, or have new or removed items.<\/p>\n
The Seo Spider includes four columns that help segment edited URLs according to these filters:<\/p>\n
Added<\/strong>: Identifies URLs that are present in the existing site and contextually in the staging site but differ at the filter level.
\nFor example by considering the Live site as “Current” and the Staging site as “Previous” and putting the focus on the “h1” headings this filter shows you all the URLs that have only in the “Live” site an optimized heading.<\/p>\nNew<\/strong>: displays all URLs that are not in the existing website, but have been introduced in the new website.<\/p>\nRemoved<\/strong>: shows all URLs that are present in the current site scan filter but not in the filter in the new site being staged.<\/p>\nMissing: <\/strong>URLs that exist in both scans but differ at the filter level.
\nFor example, consider the live site as “Current” in which there is no “meta Title” of a specific URL while in the staging site (considered in the example as “Previous”) this element had been optimized.<\/p>\n\n- 5. Identify differences between sites through “Change Detection”<\/li>\n<\/ul>\n
When performing a migration, it is a good idea to minimize the number of changes that could impact a site’s visibility.<\/p>\n
To keep track of these changes you can use the “Change Detection” feature in “Compare” mode that allows you to view all of the elements that have been changed between the current and “staging” sites.<\/p>\n
To enable “Change Detection” click on the “Config > Compare” comparison configuration (or the “gear” icon at the top) and select the items and metrics in which you want to examine changes.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/change-detection-configurazione-crawl-comparison.jpeg\" "\\"change-detection-configurazione-crawl-comparison\\"")
<\/span><\/div>Next, simply click “Compare” to run the scan. The ‘Change Detection’ tab, not present by default, will appear as the last tab in the sidebar and main view of Seo Spider.<\/p>\n
<\/div>\n<\/div>![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/migrazione-sito-web-change-detection.jpeg\")
<\/a><\/span><\/div>In the example above, 2 URLs changed the title of the page, 250 changed in word count while the meta description and Inlinks remained unchanged etc.<\/p>\n
In this screen you are able to click on each item listed to isolate it and see its changes. In the example below, “Meta Title” was isolated to analyze the semantic validity of the meta title.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/migrazione-sito-web-change-detection-page-titles.jpeg\" "\\"migrazione-sito-web-change-detection-page-titles\\"")
<\/span><\/div>If the site structure and internal links are changing considerably in the migration of your site, then it is really important to understand which pages have lost (or gained links), and changed the depth of crawling.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/migrazione-sito-web-lost-internal-links.jpeg\" "\\"migrazione-sito-web-lost-internal-links\\"")
<\/span><\/div>\n- 6. Redirect Mapping (Bonus)<\/li>\n<\/ul>\n
Although not built for this purpose, the “near duplicates” feature you can use it to do redirect mapping from old URLs to new URLs if the content is similar between them.<\/p>\n
To benefit from this, simply switch to ‘List -list’ mode (‘Mode-Mode > List-List’), enable ‘Near Duplicates’ and disable ‘Only Check Indexable Pages For Duplicates’.<\/p>\n<\/div><\/div><\/div><\/div><\/div>
Config > Content > Duplicates<\/span><\/p>\n<\/div><\/div><\/div><\/div><\/div>![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/migrazione-sito-web-enable-near-dups.jpeg\" "\\"migrazione-sito-web-enable-near-dups\\"")
<\/span><\/div>Next, you can refine the scan by configuring the content area used for similarity analysis between old and new URLs.<\/p>\n<\/div><\/div><\/div><\/div><\/div>
Config > Content > Area<\/span><\/p>\n<\/div><\/div><\/div><\/div><\/div>For example, you can “include” category classes and product descriptions from the old and new sites if they have changed. If the patterns are the same, then it is unlikely that you will need to adjust the content area.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/migrazione-sito-web-redirect-mapping-content-area-1.jpeg\" "\\"migrazione-sito-web-redirect-mapping-content-area")
<\/span><\/div>To use List mode for mapping, you can upload both new and old URLs in List mode and scan them, remembering to remove the scan depth in ‘Config > Spider > Limits’ and enter both the existing site and staging site homepages.<\/p>\n
With the crawl complete, all you need to do is turn on “Crawl Analysis” and navigate to the “Content” tab to view the details of the near-duplicates and the similarity index.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/migrazione-redirect-mapping-near-duplicates.jpeg\")
<\/a><\/span><\/div>If the content has a perfect match, each URL will have 1 near duplicate, with close to 100% similarity.<\/p>\n
If the content has changed significantly, the similarity scores and the number of near-duplicates will be different. The match between URLs will be viewable in the lower window of the Seo Spider.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/migrazione-redirect-mapping-near-duplicates-lower.jpeg\" "\\"migrazione-redirect-mapping-near-duplicates-lower\\"")
<\/span><\/div>At this point, simply export the “Near Duplicates” from Bulk Export>Content>Near Duplicates to get a good basis for properly setting up redirects and saving the site from the potential organic damage of migration.<\/p>\n
<\/div>\n<\/div>![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/migrazione-redirect-mapping-near-content-export.jpeg\")
<\/a><\/span><\/div>\n- 7. Post Migration: scanning the new site<\/li>\n<\/ul>\n
Upon publication, all you have to do is crawl the website to check that all pages on the site are “crawlable” and indexable and that it does not have important resources blocked by robots.txt (Response Codes > Blocked by Robots.txt)<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/migrazione-response-codes-blocked-by-robots-txt.jpeg\")
<\/a><\/span><\/div>Make sure there are no unwanted noindex, nofollow or none directives in the meta tags or HTTP header (Directives > Noindex).<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/migrazione-directives-noindex.jpeg\" "\\"migrazione-directives-noindex\\"")
<\/span><\/div>Verify that each URL you want to place is indexable.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/migrazione-indexability.jpeg\")
<\/a><\/span><\/div>Once you have completed this checking phase, I recommend that you take one more step by comparing the old site and the “Live” site to see if everything you had optimized on the staging site has carried over to the current version.<\/p>\n
\n- 8. Verify redirects<\/li>\n<\/ul>\n
If URLs have changed, it is critical to check all 301 redirects to the new destinations to avoid errors, skips, loops, temporary redirects, or to pages that cannot be indexed.<\/p>\n
Switch to list mode (‘Mode > List’) and enable ‘Always Follow Redirects’ in ‘Config > Spider > Advanced’.<\/p>\n<\/div>
This article tells you which is the best solution to discover and solve the most common critical issues during the delicate phase of migration.<\/p>\n
Any migration is always an epic undertaking in order not to negatively impact the organic ranking of the website that is being migrated.<\/p>\n<\/div>
Pre-migration<\/h3><\/span><\/div><\/div><\/div>Before delving into this process there are essential steps to put in place in the pre-migration phase in which a crawler can help you and which should not be overlooked.<\/p>\n
\n- 1. Perform a Crawl of the current site<\/li>\n<\/ul>\n
First, a full website scan is recommended for:<\/p>\n
Being able to compare it with the staging site;<\/p>\n
Check whether the old URLs have been correctly directed to the new resources.<\/p>\n
Have a backup to start from in case something goes wrong during the migration.<\/p>\n
In summary before migration you need to prepare the ground for a full backup that includes all the most important elements of the current site.<\/p>\n
To take advantage of Screaming Frog at this stage you need to enable scanning for all resources you may need to refer to in the future such as AMP versions, Hreflangs, structured data etc.<\/p>\n
In this shopping list, don’t forget if there are any client-side JavaScript dependencies that require JavaScript rendering and include any other domain, subdomain, or mobile and desktop site variants.<\/p>\n
Always integrate XML Sitemap into the crawl and connect via API to Google Analytics and Search Console to unearth any orphan pages (enable “Crawl New URLs Discovered GA\/GSC).<\/p>\n
Also collect link data from Ahrefs, Majestic, Moz, and performance on query performance with GSC to keep an eye on important pages based on revenue, conversions, traffic, impressions, clicks that allow you not to dissipate the ranking you have gained.<\/p>\n<\/div>
<\/span><\/div>\n- 2. Export the URLs to be redirected<\/li>\n<\/ul>\n
Very often during migrations the elements that change most frequently are slugs, so it is necessary to collect a complete list of URLs to permanently redirect old pages to the equivalent ones on the new website.<\/p>\n
This step can be relatively straightforward if it is a domain migration and the URL path remains the same, but it becomes complex and risky if the entire URL structure of the website is changing or simply some strategic areas are being rethought based on new Search Intent.<\/p>\n
To get a complete list of existing URLs, simply export them from the crawl run of the live site by clicking on “Export” under the “Internal” tab.<\/p>\n
This mapping activity is one of the crucial steps in your migration to redirect each element in the most appropriate way.<\/p>\n
If comprehensive mapping is not possible, it is a good idea to prioritize top-performing pages based on conversions, user and links collected in step 1.<\/p>\n<\/div>
<\/span><\/div>\n- 3. Pre-migration testing phase<\/li>\n<\/ul>\n
When the new Web site is in “Staging” it is time to start the checks before it goes online for good, and here again the crawler comes to your aid:<\/p>\n
The first activity is to crawl the staging site to identify differences, problems, and opportunities.<\/p>\n
Normally, sites at this stage are inhibited from being scanned by the robots.txt file or password so as not to be considered by the Search Engine Spider. Thanks to Screaming Frog you can force authentication, ignore the robots.txt file or, by changing configurations, manage the scanning of resources left in “noindex” or “nofollow” because they are still under development (see dedicated section “How to: Scan a password-protected site”).<\/p>\n<\/div>
<\/a><\/span><\/div>Once you have completed the production site scan you can begin the analysis by initially using the “Overview” tab in the sidebar.<\/p>\n<\/div>
<\/span><\/div><\/div><\/div><\/span>site staging vs. live<\/h3><\/span><\/div><\/div><\/div>\n- 4. Comparison of staging and Live site (URL MAPPING).<\/li>\n<\/ul>\n
Now you are in possession of all the data you need: on one side you have scanned the site currently online and on the other the site in staging, all you have to do is compare the two versions to discover any inconsistencies.<\/p>\n
You can do this activity simply with the Seo Spider by using “URL Mapping” of the “Compare” mode and defining from the top bar the two scan versions to compare.<\/p>\n<\/div>
<\/span><\/div>This opportunity introduced with the latest Seo Spider updates, allows you to compare different URL structures, such as different host names or directories using RegEX rules.<\/p>\n
To use it, simply change the crawl mode to “Compare,” choose the two saved crawls, and click on the “gear” icon in the top bar.<\/p>\n
On the “Crawl Comparison Configuration” popup all you have to do is click on “URL Mapping” and define the old domain with the new one via RegEX rules.<\/p>\n
In the example below, the existing site was mapped with the “Staging” version only for the hostname considering they share the same URL paths.<\/p>\n<\/div>
<\/span><\/div>Through mapping, equivalent URLs are compared with each other to obtain the data from the overview tab, problems and opportunities, site structure tab, and change detection.<\/p>\n<\/div>
<\/a><\/span><\/div>When the comparison is finished, you can click on the columns to see which URLs have changed and use the filter on the main window to alternate between current and previous scans, as well as to check URLs that have been added, missing, or have new or removed items.<\/p>\n
The Seo Spider includes four columns that help segment edited URLs according to these filters:<\/p>\n
Added<\/strong>: Identifies URLs that are present in the existing site and contextually in the staging site but differ at the filter level.
\nFor example by considering the Live site as “Current” and the Staging site as “Previous” and putting the focus on the “h1” headings this filter shows you all the URLs that have only in the “Live” site an optimized heading.<\/p>\nNew<\/strong>: displays all URLs that are not in the existing website, but have been introduced in the new website.<\/p>\nRemoved<\/strong>: shows all URLs that are present in the current site scan filter but not in the filter in the new site being staged.<\/p>\nMissing: <\/strong>URLs that exist in both scans but differ at the filter level.
\nFor example, consider the live site as “Current” in which there is no “meta Title” of a specific URL while in the staging site (considered in the example as “Previous”) this element had been optimized.<\/p>\n\n- 5. Identify differences between sites through “Change Detection”<\/li>\n<\/ul>\n
When performing a migration, it is a good idea to minimize the number of changes that could impact a site’s visibility.<\/p>\n
To keep track of these changes you can use the “Change Detection” feature in “Compare” mode that allows you to view all of the elements that have been changed between the current and “staging” sites.<\/p>\n
To enable “Change Detection” click on the “Config > Compare” comparison configuration (or the “gear” icon at the top) and select the items and metrics in which you want to examine changes.<\/p>\n<\/div>
<\/span><\/div>Next, simply click “Compare” to run the scan. The ‘Change Detection’ tab, not present by default, will appear as the last tab in the sidebar and main view of Seo Spider.<\/p>\n
<\/div>\n<\/div><\/a><\/span><\/div>In the example above, 2 URLs changed the title of the page, 250 changed in word count while the meta description and Inlinks remained unchanged etc.<\/p>\n
In this screen you are able to click on each item listed to isolate it and see its changes. In the example below, “Meta Title” was isolated to analyze the semantic validity of the meta title.<\/p>\n<\/div>
<\/span><\/div>If the site structure and internal links are changing considerably in the migration of your site, then it is really important to understand which pages have lost (or gained links), and changed the depth of crawling.<\/p>\n<\/div>
<\/span><\/div>\n- 6. Redirect Mapping (Bonus)<\/li>\n<\/ul>\n
Although not built for this purpose, the “near duplicates” feature you can use it to do redirect mapping from old URLs to new URLs if the content is similar between them.<\/p>\n
To benefit from this, simply switch to ‘List -list’ mode (‘Mode-Mode > List-List’), enable ‘Near Duplicates’ and disable ‘Only Check Indexable Pages For Duplicates’.<\/p>\n<\/div><\/div><\/div><\/div><\/div>
Config > Content > Duplicates<\/span><\/p>\n<\/div><\/div><\/div><\/div><\/div><\/span><\/div>Next, you can refine the scan by configuring the content area used for similarity analysis between old and new URLs.<\/p>\n<\/div><\/div><\/div><\/div><\/div>
Config > Content > Area<\/span><\/p>\n<\/div><\/div><\/div><\/div><\/div>For example, you can “include” category classes and product descriptions from the old and new sites if they have changed. If the patterns are the same, then it is unlikely that you will need to adjust the content area.<\/p>\n<\/div>
<\/span><\/div>To use List mode for mapping, you can upload both new and old URLs in List mode and scan them, remembering to remove the scan depth in ‘Config > Spider > Limits’ and enter both the existing site and staging site homepages.<\/p>\n
With the crawl complete, all you need to do is turn on “Crawl Analysis” and navigate to the “Content” tab to view the details of the near-duplicates and the similarity index.<\/p>\n<\/div>
<\/a><\/span><\/div>If the content has a perfect match, each URL will have 1 near duplicate, with close to 100% similarity.<\/p>\n
If the content has changed significantly, the similarity scores and the number of near-duplicates will be different. The match between URLs will be viewable in the lower window of the Seo Spider.<\/p>\n<\/div>
<\/span><\/div>At this point, simply export the “Near Duplicates” from Bulk Export>Content>Near Duplicates to get a good basis for properly setting up redirects and saving the site from the potential organic damage of migration.<\/p>\n
<\/div>\n<\/div><\/a><\/span><\/div>\n- 7. Post Migration: scanning the new site<\/li>\n<\/ul>\n
Upon publication, all you have to do is crawl the website to check that all pages on the site are “crawlable” and indexable and that it does not have important resources blocked by robots.txt (Response Codes > Blocked by Robots.txt)<\/p>\n<\/div>
<\/a><\/span><\/div>Make sure there are no unwanted noindex, nofollow or none directives in the meta tags or HTTP header (Directives > Noindex).<\/p>\n<\/div>
<\/span><\/div>Verify that each URL you want to place is indexable.<\/p>\n<\/div>
<\/a><\/span><\/div>Once you have completed this checking phase, I recommend that you take one more step by comparing the old site and the “Live” site to see if everything you had optimized on the staging site has carried over to the current version.<\/p>\n
\n- 8. Verify redirects<\/li>\n<\/ul>\n
If URLs have changed, it is critical to check all 301 redirects to the new destinations to avoid errors, skips, loops, temporary redirects, or to pages that cannot be indexed.<\/p>\n
Switch to list mode (‘Mode > List’) and enable ‘Always Follow Redirects’ in ‘Config > Spider > Advanced’.<\/p>\n<\/div>
Before delving into this process there are essential steps to put in place in the pre-migration phase in which a crawler can help you and which should not be overlooked.<\/p>\n
- \n
- 1. Perform a Crawl of the current site<\/li>\n<\/ul>\n
First, a full website scan is recommended for:<\/p>\n
Being able to compare it with the staging site;<\/p>\n
Check whether the old URLs have been correctly directed to the new resources.<\/p>\n
Have a backup to start from in case something goes wrong during the migration.<\/p>\n
In summary before migration you need to prepare the ground for a full backup that includes all the most important elements of the current site.<\/p>\n
To take advantage of Screaming Frog at this stage you need to enable scanning for all resources you may need to refer to in the future such as AMP versions, Hreflangs, structured data etc.<\/p>\n
In this shopping list, don’t forget if there are any client-side JavaScript dependencies that require JavaScript rendering and include any other domain, subdomain, or mobile and desktop site variants.<\/p>\n
Always integrate XML Sitemap into the crawl and connect via API to Google Analytics and Search Console to unearth any orphan pages (enable “Crawl New URLs Discovered GA\/GSC).<\/p>\n
Also collect link data from Ahrefs, Majestic, Moz, and performance on query performance with GSC to keep an eye on important pages based on revenue, conversions, traffic, impressions, clicks that allow you not to dissipate the ranking you have gained.<\/p>\n<\/div>
<\/span><\/div>- \n
- 2. Export the URLs to be redirected<\/li>\n<\/ul>\n
Very often during migrations the elements that change most frequently are slugs, so it is necessary to collect a complete list of URLs to permanently redirect old pages to the equivalent ones on the new website.<\/p>\n
This step can be relatively straightforward if it is a domain migration and the URL path remains the same, but it becomes complex and risky if the entire URL structure of the website is changing or simply some strategic areas are being rethought based on new Search Intent.<\/p>\n
To get a complete list of existing URLs, simply export them from the crawl run of the live site by clicking on “Export” under the “Internal” tab.<\/p>\n
This mapping activity is one of the crucial steps in your migration to redirect each element in the most appropriate way.<\/p>\n
If comprehensive mapping is not possible, it is a good idea to prioritize top-performing pages based on conversions, user and links collected in step 1.<\/p>\n<\/div>
<\/span><\/div>- \n
- 3. Pre-migration testing phase<\/li>\n<\/ul>\n
When the new Web site is in “Staging” it is time to start the checks before it goes online for good, and here again the crawler comes to your aid:<\/p>\n
The first activity is to crawl the staging site to identify differences, problems, and opportunities.<\/p>\n
Normally, sites at this stage are inhibited from being scanned by the robots.txt file or password so as not to be considered by the Search Engine Spider. Thanks to Screaming Frog you can force authentication, ignore the robots.txt file or, by changing configurations, manage the scanning of resources left in “noindex” or “nofollow” because they are still under development (see dedicated section “How to: Scan a password-protected site”).<\/p>\n<\/div>
<\/a><\/span><\/div>Once you have completed the production site scan you can begin the analysis by initially using the “Overview” tab in the sidebar.<\/p>\n<\/div>
<\/span><\/div><\/div><\/div><\/span>site staging vs. live<\/h3><\/span>
<\/div><\/div><\/div>- \n
- 4. Comparison of staging and Live site (URL MAPPING).<\/li>\n<\/ul>\n
Now you are in possession of all the data you need: on one side you have scanned the site currently online and on the other the site in staging, all you have to do is compare the two versions to discover any inconsistencies.<\/p>\n
You can do this activity simply with the Seo Spider by using “URL Mapping” of the “Compare” mode and defining from the top bar the two scan versions to compare.<\/p>\n<\/div>
<\/span><\/div>This opportunity introduced with the latest Seo Spider updates, allows you to compare different URL structures, such as different host names or directories using RegEX rules.<\/p>\n
To use it, simply change the crawl mode to “Compare,” choose the two saved crawls, and click on the “gear” icon in the top bar.<\/p>\n
On the “Crawl Comparison Configuration” popup all you have to do is click on “URL Mapping” and define the old domain with the new one via RegEX rules.<\/p>\n
In the example below, the existing site was mapped with the “Staging” version only for the hostname considering they share the same URL paths.<\/p>\n<\/div>
<\/span><\/div>Through mapping, equivalent URLs are compared with each other to obtain the data from the overview tab, problems and opportunities, site structure tab, and change detection.<\/p>\n<\/div>
<\/a><\/span><\/div>When the comparison is finished, you can click on the columns to see which URLs have changed and use the filter on the main window to alternate between current and previous scans, as well as to check URLs that have been added, missing, or have new or removed items.<\/p>\n
The Seo Spider includes four columns that help segment edited URLs according to these filters:<\/p>\n
Added<\/strong>: Identifies URLs that are present in the existing site and contextually in the staging site but differ at the filter level.
\nFor example by considering the Live site as “Current” and the Staging site as “Previous” and putting the focus on the “h1” headings this filter shows you all the URLs that have only in the “Live” site an optimized heading.<\/p>\nNew<\/strong>: displays all URLs that are not in the existing website, but have been introduced in the new website.<\/p>\n
Removed<\/strong>: shows all URLs that are present in the current site scan filter but not in the filter in the new site being staged.<\/p>\n
Missing: <\/strong>URLs that exist in both scans but differ at the filter level.
\nFor example, consider the live site as “Current” in which there is no “meta Title” of a specific URL while in the staging site (considered in the example as “Previous”) this element had been optimized.<\/p>\n- \n
- 5. Identify differences between sites through “Change Detection”<\/li>\n<\/ul>\n
When performing a migration, it is a good idea to minimize the number of changes that could impact a site’s visibility.<\/p>\n
To keep track of these changes you can use the “Change Detection” feature in “Compare” mode that allows you to view all of the elements that have been changed between the current and “staging” sites.<\/p>\n
To enable “Change Detection” click on the “Config > Compare” comparison configuration (or the “gear” icon at the top) and select the items and metrics in which you want to examine changes.<\/p>\n<\/div>
<\/span><\/div>Next, simply click “Compare” to run the scan. The ‘Change Detection’ tab, not present by default, will appear as the last tab in the sidebar and main view of Seo Spider.<\/p>\n
<\/div>\n<\/div><\/a><\/span><\/div>In the example above, 2 URLs changed the title of the page, 250 changed in word count while the meta description and Inlinks remained unchanged etc.<\/p>\n
In this screen you are able to click on each item listed to isolate it and see its changes. In the example below, “Meta Title” was isolated to analyze the semantic validity of the meta title.<\/p>\n<\/div>
<\/span><\/div>If the site structure and internal links are changing considerably in the migration of your site, then it is really important to understand which pages have lost (or gained links), and changed the depth of crawling.<\/p>\n<\/div>
<\/span><\/div>- \n
- 6. Redirect Mapping (Bonus)<\/li>\n<\/ul>\n
Although not built for this purpose, the “near duplicates” feature you can use it to do redirect mapping from old URLs to new URLs if the content is similar between them.<\/p>\n
To benefit from this, simply switch to ‘List -list’ mode (‘Mode-Mode > List-List’), enable ‘Near Duplicates’ and disable ‘Only Check Indexable Pages For Duplicates’.<\/p>\n<\/div><\/div><\/div><\/div><\/div>
Config > Content > Duplicates<\/span><\/p>\n<\/div><\/div><\/div><\/div><\/div>
<\/span><\/div>Next, you can refine the scan by configuring the content area used for similarity analysis between old and new URLs.<\/p>\n<\/div><\/div><\/div><\/div><\/div>
Config > Content > Area<\/span><\/p>\n<\/div><\/div><\/div><\/div><\/div>
For example, you can “include” category classes and product descriptions from the old and new sites if they have changed. If the patterns are the same, then it is unlikely that you will need to adjust the content area.<\/p>\n<\/div>
<\/span><\/div>To use List mode for mapping, you can upload both new and old URLs in List mode and scan them, remembering to remove the scan depth in ‘Config > Spider > Limits’ and enter both the existing site and staging site homepages.<\/p>\n
With the crawl complete, all you need to do is turn on “Crawl Analysis” and navigate to the “Content” tab to view the details of the near-duplicates and the similarity index.<\/p>\n<\/div>
<\/a><\/span><\/div>If the content has a perfect match, each URL will have 1 near duplicate, with close to 100% similarity.<\/p>\n
If the content has changed significantly, the similarity scores and the number of near-duplicates will be different. The match between URLs will be viewable in the lower window of the Seo Spider.<\/p>\n<\/div>
<\/span><\/div>At this point, simply export the “Near Duplicates” from Bulk Export>Content>Near Duplicates to get a good basis for properly setting up redirects and saving the site from the potential organic damage of migration.<\/p>\n
<\/div>\n<\/div><\/a><\/span><\/div>- \n
- 7. Post Migration: scanning the new site<\/li>\n<\/ul>\n
Upon publication, all you have to do is crawl the website to check that all pages on the site are “crawlable” and indexable and that it does not have important resources blocked by robots.txt (Response Codes > Blocked by Robots.txt)<\/p>\n<\/div>
<\/a><\/span><\/div>Make sure there are no unwanted noindex, nofollow or none directives in the meta tags or HTTP header (Directives > Noindex).<\/p>\n<\/div>
<\/span><\/div>Verify that each URL you want to place is indexable.<\/p>\n<\/div>
<\/a><\/span><\/div>Once you have completed this checking phase, I recommend that you take one more step by comparing the old site and the “Live” site to see if everything you had optimized on the staging site has carried over to the current version.<\/p>\n
- \n
- 8. Verify redirects<\/li>\n<\/ul>\n
If URLs have changed, it is critical to check all 301 redirects to the new destinations to avoid errors, skips, loops, temporary redirects, or to pages that cannot be indexed.<\/p>\n
Switch to list mode (‘Mode > List’) and enable ‘Always Follow Redirects’ in ‘Config > Spider > Advanced’.<\/p>\n<\/div>
- 8. Verify redirects<\/li>\n<\/ul>\n
- 7. Post Migration: scanning the new site<\/li>\n<\/ul>\n
- 6. Redirect Mapping (Bonus)<\/li>\n<\/ul>\n
- 5. Identify differences between sites through “Change Detection”<\/li>\n<\/ul>\n
- 4. Comparison of staging and Live site (URL MAPPING).<\/li>\n<\/ul>\n
- 3. Pre-migration testing phase<\/li>\n<\/ul>\n
- 2. Export the URLs to be redirected<\/li>\n<\/ul>\n