Staging Sites Analysis<\/h2><\/span><\/div><\/div><\/div>Staging websites usually cannot be crawled by Search Engines and crawlers. Existing various methods to prevent crawling there are several approaches or configurations of Screaming Frog to implement to achieve crawling and circumvent these limitations.<\/p>\n<\/div>
<\/div><\/div><\/span>Robots.txt<\/h3><\/span><\/div><\/div><\/div>\n- 1. Robots.txt<\/li>\n<\/ul>\n
The most common case involves the robots.txt file blocking the scan. In this case the Seo Spider collects only one URL , the record shows the message “Blocked by robots.txt” and the status is “Indexable”. To break through this barrier, simply set “Ignore Robots.txt” from Config. > Robots.txt<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/how-to-crawl-staging-website-blocked-robots.jpeg\" "\\"how-to-crawl-staging-website-blocked-robots\\"")
<\/span><\/div>![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/sito-bloccato-da-robots.txt.png\" "\\"sito")
<\/span><\/div>An alternative is to configure a custom robots.txt via “Config > robots.txt > Custom” to remove any “disallow” directives and add if necessary other customizations for the Seo Spider to follow.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/custom-robots-txt-con-seo-spider.png\")
<\/a><\/span><\/div><\/div><\/div><\/span>Authentication<\/h3><\/span><\/div><\/div><\/div>\n- 2. Authentication<\/li>\n<\/ul>\n
In case the server requires authentication, it is possible to provide username and password to the Seo Spider to scan the site.<\/p>\n<\/div>
<\/div><\/div><\/span>Basic authentication<\/h4><\/span><\/div><\/div><\/div>Basic authentication and digest is automatically detected by Seo Spider when scanning the website.<\/p>\n
When scanning the site under development, an authentication pop-up window appears, just as it does in a Web browser, asking for a user name and password.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/autenticazione-di-base-seo-spider.png\")
<\/a><\/span><\/div>By entering the correct credentials, the Seo Spider begins normal crawling of the staging pages of the website.<\/p>\n<\/div>
<\/div><\/div><\/span>Web Form Authentication<\/h4><\/span><\/div><\/div><\/div>If there are other web forms or areas that require cookie login for authentication, the Seo Spider allows users to access these web forms within the tool’s built-in Chrome browser.<\/p>\n
To log in using web forms authentication, click on “Configuration > Authentication > Forms-based.” Then click on the ‘Add’ button, enter the URL of the site you wish to crawl and log in from your browser.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/Autenticazione-web-form.png\")
<\/a><\/span><\/div><\/div><\/div><\/span>IP Address<\/h3><\/span><\/div><\/div><\/div>Some staging platforms may restrict access based on IP address.<\/p>\n
Since the SEO Spider crawls locally from the machine from which it runs, you must provide this IP address to be included in the “whitelist.”<\/p>\n<\/div>
<\/div><\/div><\/span>Seo Spider Configuration<\/h3><\/span><\/div><\/div><\/div>Sites in development may respond to HTTP requests differently than those in a live environment, and may often have robot directives that require additional configuration in the SEO Spider.<\/p>\n<\/div>
<\/div><\/div><\/span>Scanning speed<\/h4><\/span><\/div><\/div><\/div>Staged Web sites are generally slower and cannot carry the same load as a production server. If you notice site instability and timeout errors or server errors you need to reduce the Spider scan speed.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/configurazione-velocita-scansione-seo-spider.png\")
<\/a><\/span><\/div>Or manage the timeout and AJAX timeout in the case of scanning with Javascript rendering.<\/p>\n<\/div>
<\/div><\/div><\/span>No Follow<\/h4><\/span><\/div><\/div><\/div>To protect against possible BOT intrusions many developing sites might present a “nofollow” meta robots tag throughout the site, or X-Robots-Tag in the HTTP header. A “nofollow” is a very different directive from a noindex, and it tells a crawler not to follow any outlinks from a page.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/how-to-crawl-staging-sito-web-nofollow.jpeg\")
<\/a><\/span><\/div>In case you present the condition in the example go to “Config > Spider” and enable “Follow Internal Nofollow” to scan the outlinks from these pages and get a full Audit.<\/p>\n
<\/div>\n<\/div>![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/Crawler-pagine-con-no-follow-link.png\")
<\/a><\/span><\/div><\/div><\/div><\/span>No Index Urls<\/h4><\/span><\/div><\/div><\/div>Another tag you may encounter in a staging site is noindex, which allows the crawler to crawl but not index it. The use of noindex can be seen under the “Directives” tab and the “noindex” filter. What I recommend is to check that this tag is removed immediately when the website goes online, too many times you see sites with the robots.txt file or noindex tag still active even though they have already been published online.<\/p>\n<\/div>
Staging websites usually cannot be crawled by Search Engines and crawlers. Existing various methods to prevent crawling there are several approaches or configurations of Screaming Frog to implement to achieve crawling and circumvent these limitations.<\/p>\n<\/div>
Robots.txt<\/h3><\/span><\/div><\/div><\/div>\n- 1. Robots.txt<\/li>\n<\/ul>\n
The most common case involves the robots.txt file blocking the scan. In this case the Seo Spider collects only one URL , the record shows the message “Blocked by robots.txt” and the status is “Indexable”. To break through this barrier, simply set “Ignore Robots.txt” from Config. > Robots.txt<\/p>\n<\/div>
<\/span><\/div><\/span><\/div>An alternative is to configure a custom robots.txt via “Config > robots.txt > Custom” to remove any “disallow” directives and add if necessary other customizations for the Seo Spider to follow.<\/p>\n<\/div>
<\/a><\/span><\/div><\/div><\/div><\/span>Authentication<\/h3><\/span><\/div><\/div><\/div>\n- 2. Authentication<\/li>\n<\/ul>\n
In case the server requires authentication, it is possible to provide username and password to the Seo Spider to scan the site.<\/p>\n<\/div>
<\/div><\/div><\/span>Basic authentication<\/h4><\/span><\/div><\/div><\/div>Basic authentication and digest is automatically detected by Seo Spider when scanning the website.<\/p>\n
When scanning the site under development, an authentication pop-up window appears, just as it does in a Web browser, asking for a user name and password.<\/p>\n<\/div>
<\/a><\/span><\/div>By entering the correct credentials, the Seo Spider begins normal crawling of the staging pages of the website.<\/p>\n<\/div>
<\/div><\/div><\/span>Web Form Authentication<\/h4><\/span><\/div><\/div><\/div>If there are other web forms or areas that require cookie login for authentication, the Seo Spider allows users to access these web forms within the tool’s built-in Chrome browser.<\/p>\n
To log in using web forms authentication, click on “Configuration > Authentication > Forms-based.” Then click on the ‘Add’ button, enter the URL of the site you wish to crawl and log in from your browser.<\/p>\n<\/div>
<\/a><\/span><\/div><\/div><\/div><\/span>IP Address<\/h3><\/span><\/div><\/div><\/div>Some staging platforms may restrict access based on IP address.<\/p>\n
Since the SEO Spider crawls locally from the machine from which it runs, you must provide this IP address to be included in the “whitelist.”<\/p>\n<\/div>
<\/div><\/div><\/span>Seo Spider Configuration<\/h3><\/span><\/div><\/div><\/div>Sites in development may respond to HTTP requests differently than those in a live environment, and may often have robot directives that require additional configuration in the SEO Spider.<\/p>\n<\/div>
<\/div><\/div><\/span>Scanning speed<\/h4><\/span><\/div><\/div><\/div>Staged Web sites are generally slower and cannot carry the same load as a production server. If you notice site instability and timeout errors or server errors you need to reduce the Spider scan speed.<\/p>\n<\/div>
<\/a><\/span><\/div>Or manage the timeout and AJAX timeout in the case of scanning with Javascript rendering.<\/p>\n<\/div>
<\/div><\/div><\/span>No Follow<\/h4><\/span><\/div><\/div><\/div>To protect against possible BOT intrusions many developing sites might present a “nofollow” meta robots tag throughout the site, or X-Robots-Tag in the HTTP header. A “nofollow” is a very different directive from a noindex, and it tells a crawler not to follow any outlinks from a page.<\/p>\n<\/div>
<\/a><\/span><\/div>In case you present the condition in the example go to “Config > Spider” and enable “Follow Internal Nofollow” to scan the outlinks from these pages and get a full Audit.<\/p>\n
<\/div>\n<\/div><\/a><\/span><\/div><\/div><\/div><\/span>No Index Urls<\/h4><\/span><\/div><\/div><\/div>Another tag you may encounter in a staging site is noindex, which allows the crawler to crawl but not index it. The use of noindex can be seen under the “Directives” tab and the “noindex” filter. What I recommend is to check that this tag is removed immediately when the website goes online, too many times you see sites with the robots.txt file or noindex tag still active even though they have already been published online.<\/p>\n<\/div>
- \n
- 1. Robots.txt<\/li>\n<\/ul>\n
The most common case involves the robots.txt file blocking the scan. In this case the Seo Spider collects only one URL , the record shows the message “Blocked by robots.txt” and the status is “Indexable”. To break through this barrier, simply set “Ignore Robots.txt” from Config. > Robots.txt<\/p>\n<\/div>
<\/span><\/div><\/span><\/div>An alternative is to configure a custom robots.txt via “Config > robots.txt > Custom” to remove any “disallow” directives and add if necessary other customizations for the Seo Spider to follow.<\/p>\n<\/div>
<\/a><\/span><\/div><\/div><\/div><\/span>Authentication<\/h3><\/span>
<\/div><\/div><\/div>- \n
- 2. Authentication<\/li>\n<\/ul>\n
In case the server requires authentication, it is possible to provide username and password to the Seo Spider to scan the site.<\/p>\n<\/div>
<\/div><\/div><\/span>Basic authentication<\/h4><\/span>
<\/div><\/div><\/div>Basic authentication and digest is automatically detected by Seo Spider when scanning the website.<\/p>\n
When scanning the site under development, an authentication pop-up window appears, just as it does in a Web browser, asking for a user name and password.<\/p>\n<\/div>
<\/a><\/span><\/div>By entering the correct credentials, the Seo Spider begins normal crawling of the staging pages of the website.<\/p>\n<\/div>
<\/div><\/div><\/span>Web Form Authentication<\/h4><\/span>
<\/div><\/div><\/div>If there are other web forms or areas that require cookie login for authentication, the Seo Spider allows users to access these web forms within the tool’s built-in Chrome browser.<\/p>\n
To log in using web forms authentication, click on “Configuration > Authentication > Forms-based.” Then click on the ‘Add’ button, enter the URL of the site you wish to crawl and log in from your browser.<\/p>\n<\/div>
<\/a><\/span><\/div><\/div><\/div><\/span>IP Address<\/h3><\/span>
<\/div><\/div><\/div>Some staging platforms may restrict access based on IP address.<\/p>\n
Since the SEO Spider crawls locally from the machine from which it runs, you must provide this IP address to be included in the “whitelist.”<\/p>\n<\/div>
<\/div><\/div><\/span>Seo Spider Configuration<\/h3><\/span>
<\/div><\/div><\/div>Sites in development may respond to HTTP requests differently than those in a live environment, and may often have robot directives that require additional configuration in the SEO Spider.<\/p>\n<\/div>
<\/div><\/div><\/span>Scanning speed<\/h4><\/span>
<\/div><\/div><\/div>Staged Web sites are generally slower and cannot carry the same load as a production server. If you notice site instability and timeout errors or server errors you need to reduce the Spider scan speed.<\/p>\n<\/div>
<\/a><\/span><\/div>Or manage the timeout and AJAX timeout in the case of scanning with Javascript rendering.<\/p>\n<\/div>
<\/div><\/div><\/span>No Follow<\/h4><\/span>
<\/div><\/div><\/div>To protect against possible BOT intrusions many developing sites might present a “nofollow” meta robots tag throughout the site, or X-Robots-Tag in the HTTP header. A “nofollow” is a very different directive from a noindex, and it tells a crawler not to follow any outlinks from a page.<\/p>\n<\/div>
<\/a><\/span><\/div>In case you present the condition in the example go to “Config > Spider” and enable “Follow Internal Nofollow” to scan the outlinks from these pages and get a full Audit.<\/p>\n
<\/div>\n<\/div><\/a><\/span><\/div><\/div><\/div><\/span>No Index Urls<\/h4><\/span>
<\/div><\/div><\/div>Another tag you may encounter in a staging site is noindex, which allows the crawler to crawl but not index it. The use of noindex can be seen under the “Directives” tab and the “noindex” filter. What I recommend is to check that this tag is removed immediately when the website goes online, too many times you see sites with the robots.txt file or noindex tag still active even though they have already been published online.<\/p>\n<\/div>
- 2. Authentication<\/li>\n<\/ul>\n