Status Code Analysis: 4xx, 3xx and Server Errors

Tab Overview
Status Code

The dedicated “Response Tab” shows the HTTP status and Status Codes of the internal and external URLs discovered during the scan. Here are in detail all the columns available to you:

  • Address: The URL being scanned.
  • Content: identifies the type of content in the URL (text/html, image/png, image/jpg, etc).
  • Status Code: displays the HTTP response code (200, 3xx, 4xx, 5xx, no response).
  • Status: returns the HTTP header response (ok if it responds 200, Not Found, Moved Permanently, etc.).
  • Indexability: defines whether the URL is indexable or non-indexable.
  • Indexability Status: the reason why a URL is not indexable. For example, if it is canonicalized to another URL or “noindex” or “Redirected.”
  • Inlinks: corresponds to the number of internal inlinks on the page. ‘Internal inlinks’ are links that point to a given URL from resources in the same scanned subdomain.
  • Response Time: identifies the time taken in seconds to download the URL.
  • Redirect URL: in case the URL address makes a redirect, this column includes the destination of the URL.
  • Redirect Type: the column can contain
    • HTTP Redirect: redirection triggered by an HTTP header.
    • HSTS Policy: redirection caused by a previous HSTS header.
    • JavaScript Redirect: triggered by the execution of JavaScript. You can only get it if you have chosen the JavaScript rendering scan mode.
    • Meta Refresh Redirect: triggered by a meta refresh tag in the HTML of the page.

Status Code Filters

Under the “Response Code” tab you have the following filters available:

Blocked by Robots.txt: the filter defines all URLs blocked by the site’s robots.txt that do not allow the Spider to scan those resources and, consequently not being able to index them on Search Engines.

Each “0” response in the Status Code column identifies the lack of an HTTP response from the server; the Status column identifies the reason. In the example above the Spider user-agent was blocked by the robots.txt through the “disallow” directive and being blocked it cannot evaluate the HTTP response.

  • Blocked Resource: all resources that are blocked by a website’s robots.txt such as images, JavaScript, and CSS. This filter only populates if you have JavaScript rendering enabled. Remember that blocked resources appear under ‘Blocked by Robots.txt’ if Javascript mode has not been enabled from configurations.
  • No Response: displays data about URLs that do not send a response to the Seo Spider’s HTTP request. In the most common case it corresponds to an incorrectly typed URL, connection timeout, connection refused or connection error. With the Status “No Response” the SEO Spider has problems making connections or receiving responses. The problem may be the proxy.

The Status “DNS lookup failed” identifies that the site was not found at all due to a domain entry error or a lack of network connection.

The “Connection Timeout” alert identifies that Screaming Frog requested the HTTP response from the server but did not receive a response within 20 seconds. To remedy the problem, you can increase the “Response Timeout” from the configurations or decrease the speed of the spider (Lowering Speed) to not load the Server and give it more time for response.

A ‘Connection Refused’ is returned when the Seo Spider’s connection attempt was rejected somewhere between the local machine and the website. The most common problems are related to the presence of firewalls and from trying to block an unknown user agent. To solve this condition I suggest you change the “user-agent” from “Screaming Frog Seo Spider” to “Chrome”.

  • Success (2XX): the requested URL was successfully received, understood, accepted and processed. Ideally all URLs encountered in a crawl should return a ‘200’ status code with an ‘OK’ status, which is perfect for crawling and indexing content.
  • Redirection (3XX): a redirection such as 301 or 302 was encountered. Ideally each internal link should link to a response with status code 200 and avoid links that cause the Spider to take extra steps in an effort to safeguard the Crawling Budget.
  • Redirection (JavaScript): a JavaScript-side redirection was encountered.
  • Redirect (Meta Refresh): redirection by “Meta Refresh” was encountered.
  • Client Error (4xx): the filter displays the resources for which there was a problem with the request. These can include responses such as 400 Bad Request, 403 Forbidden, 404 Page Not Found, 410 Removed, 429 Too Many Requests, and more. Client Errors such as 404s should be optimized so as not to waste Crawling Budget. To do this you can change the link source, remove these links for example with the l htaccess file by changing the status code to 410 or redirect them where appropriate.
  • Server error (5XX): the server failed to fulfill an apparently valid request. This can include common responses such as “500 Internal Server Errors” and 503 “Server Unavailable.” A high percentage of this error could indicate a stressed server or a misconfiguration that requires further investigation.
Seo Spider Tab