EXTERNAL AND SECURITY TAB

Screaming Frog's External and Security tabs for analysing external subdomain elements.

Overview Tab
External

The “External” tab includes all data related to external URLs. URLs of subdomains other than that of the initial page of the scan are classified as “external”.

The Tab includes the following values divided by columns:

  • Address: the address of the external URL.
  • Content Type: the type of content discovered in the URL.
  • Status Code: returns the HTTP response code (200, 3xx, 4xx, 5xx).
  • Status: represents the HTTP header response (“Ok” for server 200 responses, “Internal Server Error” in case of server problems, “Forbitten” in case of 403 status, etc.).
  • Crawl Depth: numbers of clicks required to reach a URL from the homepage or initial crawl page.
  • Inlinks: number of links found during crawl that point to an external subdomain.
    If you select a URL from the upper window you can see the details (Ex. the link source in the InLinks >From) links tab in the bottom window of the Seo Spider.

The filters available to you are the same as in the Internal Tab.

Tab External of the Seo Spider Screaming Frog

Security Tab

The “Security” tab allows you to look up the security data for internal URLs in a crawl and includes the following data:

  • Address: the URL being scanned.
  • Content: represents the content type of the URL.
  • Status Code: the HTTP response code.
  • Status: the HTTP header response.
  • Indexability: identifies whether the URL is indexable or non-indexable.
  • Indexability Status: presents why a URL is not indexable, e.g., it might be “canonicalized” to another URL or have the “no index” tag.
  • Canonical Link Element 1/2 etc: identifies all instances found by the Spider related to canonical URL data.
  • Meta Robots 1/2 etc: presents all instances of the meta robots found in the URL.
  • X-Robots-Tag 1/2 etc – X-Robots-tag data.

For the tab dedicated to safety, several filters are available to you that give you a very detailed view of any critical issues:

  • HTTP URLs: this filter allows you to isolate all URLs that present as protocol (HTTP) or contain mixed content in which you load insecure resources. The “Https” protocol is both fundamental to security and vital to ensuring safe browsing for users.Chrome and other browsers use messages such as “Unsafe Content” next to the subdomain to discourage browsing to pages with this protocol.
    If you need to check which URLs have “mixed content” just consult the ‘inlinks’ tab of the lower window of the Seo Spider. You are also able to export these results with the “HTTP URLs Inlinks” report.

Bulk Export > Security > HTTP URLs Inlinks’.

  • HTTPS URL: includes all URLs that feature a trusted protocol such as HTTPS.
  • Mixed Content: through the filter you will be able to browse all HTML pages with HTTPS protocol but that have resources such as images, JavaScript or CSS with HTTP protocol that are not considered secure. Mixed content weakens HTTPS by making pages more vulnerable and compromisable.
    You can view the HTTP resources for each URL in the ‘outlinks’ tab and export them along with the source pages with the “Mixed Content” report.

Bulk Export > Security > Mixed Content.

  • Form URL Insecure: the filter displays HTML pages with a Contact Form that has an insecure (HTTP) link as an action attribute. All URLs contained in forms on a Web site should be encrypted and therefore must be HTTPS. The URL of the HTTP form can be viewed by clicking on the source URL in the upper window and then on the ‘URL Details’ tab of the lower screen. These can be exported along with the pages they are on with the report “Form URL Insecure.

Bulk Export > Security > Mixed Content

Bulk Export > Security > Form URL Insecure

  • Form on HTTP URL: identifies HTTP protocol pages that have a form and may be blocked by the browser. All data entered in the form, including usernames and passwords could be intercepted and create harm to the surfer. The form can be viewed by clicking on the URL in the upper window and then on the ‘URL Details’ tab of the lower window, which shows you the details of the form on the HTTP URL.
  • Unsafe Cross-Origin Links: all pages that link to external sites using the target=”_blank” attribute (to open a page in a new tab) are displayed, without using rel=”noopener” (or rel=”noreferrer”) at the same time. Just using target=”_blank” exposes pages to security and performance issues. Ideally rel=”noopener” should be used on all links that contain the target=”_blank” attribute to avoid “Reverse Tabnabbing.”

Bulk Export > Security > Mixed Content

IN-DEPTH STUDY:

Reverse tabnabbing: a situation in which the page opened using the “target=”_blank” attribute can gain control of the source page by using “window.opener.location.assign()” to replace the tab in the background with malicious documents. Through the rel=”noopener” the malicious user will not be able to access the window object via window.opener.

Using the “noopener” attribute does not affect Seo level but preserves security and performance by preventing processes on the target page from running in the same thread as the source page. Some doubts remain about the use of the “noreferrer” attribute, which removes the click reference data at the browser level and, does not allow the owner of the site linked to and opened via the click, to know its origin. In the case of Google Analytics the attribute “noreferrer, changes the “traffic source” from “referral” to “direct” altering the traffic statistics. Assuming that the inclusion of an external link is intended, we do not consider the attribute inescapable.

External links that contain the target=”_blank” attribute can be displayed in the ‘outlinks’ tab and the ‘target’ column. They can be exported along with source pages from the “Unsafe Cross-Origin Links” report

Bulk Export > Security > Mixed Content

Bulk Export > Unsafe> Cross Links

  • Protocol-Relative Resource Links: this filter shows you all pages that load resources such as images, JavaScript and CSS using relative links without specifying the protocol (e.g. “//screamingfrog.co.uk”).

This situation is very common and adopted by developers to save time by letting the browser specify the protocol. However, this habit may expose some sites to being compromised and could outline performance problems.

Resource links that are protocol related can be viewed for each URL by clicking on the ‘outlinks’ tab and viewing the ‘Path Type’ column for ‘Protocol Relative’. They can be exported together with source pages from the Protocol-Relative Resource Links report.

Bulk Export > Security > Mixed Content

Bulk Export > Security > Protocol-Relative Resource Links

  • Missing HSTS Header: identifies all URLs that are missing the HSTS response header. The HTTP Strict-Transport-Security (HSTS) response tells browsers that they should only be accessed via HTTPS, rather than HTTP. If a Web site accepts a connection to HTTP, before being redirected to HTTPS, visitors will still initially communicate over HTTP. The HSTS header tells the browser to never load over HTTP and to automatically convert all requests to HTTPS. The Seo Spider itself will follow the instructions of the HSTS header, but will report any link encountered on HTTP URLs with a 307 status code and ‘HSTS Policy’ status.
  • Missing Content-Security-Policy-Header: identifies any URL that is missing the Content-Security-Policy response header. This header allows a Web site to control what resources are loaded for a page. This policy can help protect against cross-site scripting (XSS) attacks that exploit the browser’s trust in the content received from the server. The Seo Spider only checks the existence of the header, and does not query the directives found within to determine if they are well set up for the website. This should be done manually.
  • Bad Content Type: displays any URL where the actual content type does not match the content type set in the Header header. It also identifies any used but invalid MIME type.

Bulk Export > Security > Mixed Content

Security Tab Video

Seo Spider Tab