Site Analysis with Password

Find out how to analyse staging sites or sites with passwords for viewing and scanning.


Protected sites

In the life of a Seo it often happens that you have to analyze staging websites or have to scan private sites that need passwords for viewing and scanning.

To walk you through this type of diagnosis comes to your aid Screaming Frog, which, since version 7.0 of Seo Spider, has introduced the “web form authentication” feature.

Before proceeding with the crawl, however, let us see what the potential drawbacks of this feature are and how to solve them for foolproof crawling.

  • 1. Managing credentials for the Seo Spider.

This new option is very powerful but must be handled with absolute responsibility since, if the Spider is authenticated as admin of the website, it will click on every link on the pages and being logged in the clicks could also affect logout, article creation, plugin installation or in the most extreme case data deletion.

For this reason, it is essential to set up authentication with limitations in credentials, leaving read-only permissions and no “write-to-site” permissions.

If you analyze common CMSs such as WordPress, Drupal or Joomla the best idea is to create a dedicated user with “subscriber” permissions.

  • 2. Use the “Exclude” function.

It is also a good idea to use the “Exclude” function on folders that might change the site in production or private; in the case of wordpress you could exclude from crawling any item in “/wp-admin/”.

For example, to exclude the admin dedicated folder and maybe logout you can set up the following RegEX expressions for the Seo Spider.

  • 3. Configure Authentications

Once you have secured the site to be analyzed and configured the exclusions, you can access the website by selecting “Authentication” from the main menu (Configuration), choosing the “Form Based” option and clicking on the ‘Add’ button.

Going forward with the procedure you will be prompted to enter the URL to be scanned and a browser will automatically open allowing you to enter your credentials.

Through this option you will be able, for example, to correct before going online all the critical issues found but, as mentioned at the beginning of the article, you must pay the utmost attention to the security of the website and avoid uncontrolled crawl raids!!!


Seo Spider Tab