{"id":3318,"date":"2024-03-21T13:49:43","date_gmt":"2024-03-21T13:49:43","guid":{"rendered":"http:\/\/screamingfrog.club\/seo-spider-crawling-problems\/"},"modified":"2024-07-03T07:11:48","modified_gmt":"2024-07-03T07:11:48","slug":"seo-spider-crawling-issues","status":"publish","type":"post","link":"https:\/\/screamingfrog.club\/en\/seo-spider-crawling-issues\/","title":{"rendered":"Seo Spider crawling issues"},"content":{"rendered":"
<\/div><\/div><\/span>

Intro<\/p><\/span>

<\/div><\/div><\/div>

Sometimes it may happen that Screaming Frog performs a one-page crawl or does not scan as expected.<\/p>\n

In that case, the first things to check are the ‘Status’ and ‘Status Code’ of the returned resources to identify the problem and fix it.<\/p>\n<\/div>

<\/div><\/div><\/span>

Blocked by Robots.txt<\/p><\/span>

<\/div><\/div><\/div>

The filter defines all URLs blocked by the site’s robots.txt that does not allow the Spider to scan those resources and consequently unable to index on Search Engines.<\/p>\n<\/div>

<\/span><\/div>

Each “0” response in the Status Code column identifies the lack of an HTTP response from the server; the Status column identifies the reason. In the example above the Spider’s user-agent was blocked by the robots.txt through the “disallow” directive and being blocked it cannot see the HTTP response.<\/p>\n

What to do<\/strong><\/p>\n

In the case of resources blocked by robots you can set the Seo Spider to ignore it via Configuration > Robots.txt > Settings > Ignore Robots.txt or by customizing it. By default Screaming Frog follows the “disallow” directions of the robots.txt file.<\/p>\n<\/div>

<\/div><\/div><\/span>

DNS Lookup Failed<\/p><\/span>

<\/div><\/div><\/div>

Identifies that the site was not found at all because of a domain entry error or a lack of network connection.<\/p>\n

What to do<\/strong><\/p>\n

Check that the site has been written correctly and is visible in your browser. In case you cannot view it via browser there might be problems with connectivity; on the other hand, in case the site is correctly rendered on the browser you should check if there are any Antivirus or Firewalls blocking the Seo Spider.<\/p>\n<\/div>

<\/span><\/div>
<\/div><\/div><\/span>

Connection Timeout<\/p><\/span>

<\/div><\/div><\/div>

Identifies that Screaming Frog requested the HTTP response from the server but did not receive a response within 20 seconds. To overcome the problem, you can increase the “Response Timeout” from the configurations or decrease the speed of the spider (lowering Speed) to not load the Server and give it more time for response.<\/p>\n<\/div>

<\/span><\/div>

What to Do<\/strong><\/p>\n

Check the site load times on your browser; if they are very long you can change the “Timeout” time on the Seo Spider and decrease the crawler speed to lighten the server.<\/p>\n

Check to see if you can scan other sites with Screaming Frog. If the problem is widespread there may be a problem with Antivirus and Firewall. Try in this case to create an exception to the Screaming Frog scan. If this solution does not solve the problem either, there is probably a network or local problem present. For example, check if the proxy is enabled (Configuration > System > Proxy and if it is, try disabling it and restart crawling.<\/p>\n<\/div>

<\/div><\/div><\/span>

Connection Refused<\/p><\/span>

<\/div><\/div><\/div>

It is returned when the Seo Spider’s connection attempt was rejected somewhere between the local machine and the website.<\/p>\n<\/div>

<\/span><\/div>

What to do<\/strong><\/p>\n

Can you scan other websites? If the answer is yes check the Antivirus and Firewall and create an exception for the Spider, if the problem is common to other sites check for a network or local problem to your pc.<\/p>\n

Can you view the page in the browser or does it return a similar error? If the page can be seen change User-agent and use “Chrome” (Configuration > User-Agent).<\/p>\n<\/div>

<\/div><\/div><\/span>

No Response<\/p><\/span>

<\/div><\/div><\/div>

The Seo Spider has trouble making connections or receiving responses. A common problem may be the proxy.<\/p>\n<\/div>

<\/span><\/div>

What to do<\/strong><\/p>\n

The first check concerns the proxy (Configuration > System > Proxy) where you can try to disable it. If it is not set correctly then this could mean that the Seo Spider is not sending\/receiving requests correctly.<\/p>\n<\/div>

<\/div><\/div><\/span>

Success (2xx)<\/p><\/span>

<\/div><\/div><\/div>

the requested URL was successfully received, understood, accepted, and processed. The check I suggest you do is about the presence of the “noFollow” attribute<\/p>\n

What to do<\/strong><\/p>\n

    \n
  • Does the requested page have the meta robots ‘nofollow’ directive on the page\/header HTTP or do all links on the page have rel=’nofollow’ attributes? In this case, simply set the Seo Spider to follow Internal\/External Nofollow (Configuration > Spider). This critical issue is due to the fact that Screaming Frog follows “nofollow” directions by default.<\/li>\n
  • Does the website have JavaScript links? Try disabling Javascript and display the pages in the browser. If you find yourself in this case you just need to enable JavaScript rendering (Configuration > Spider >Rendering > JavaScript). By default, the Seo Spider only scans for links <a href=””>, <img src=””> e <link rel=”canonical”> in the HTML source code and does not read the DOM.<\/li>\n
  • Check the “Limits” tab of “Configuration > Spider” specifically the “Limit Search Depth” and “Limit Search Total” options. In case they are set to 0 or 1 respectively, then the Seo Spider is instructed to scan only a single URL.<\/li>\n
  • Does the site require cookies? Try viewing your page with your browser and cookies disabled. This condition occurs because the Seo Spider is served a different page without hyperlinks in the case of disabled cookies. To resolve this critical scanning issue go to Configuration > Spider > Advanced Tab > Allow Cookies<\/li>\n
  • What is specified in the ‘Content’ column? If there is no content, enable JavaScript rendering (Configuration > Spider >Rendering > JavaScript) and try starting crawl again. This critical issue occurs if no content type is specified in the HTTP header and the Seo Spider does not know if the URL is an image, PDF, HTML pages, etc. so it cannot scan to determine if there are other links. This can be circumvented with the rendering mode as the tool checks if a <meta http-equiv> is specified in the <head> of the document when it is enabled.<\/li>\n
  • Are there any restrictions due to age? In this case try changing the user-agent to Googlebot (Configuration > User-Agent). The site\/server could be set up to serve HTML to search bots without requiring age input.<\/li>\n<\/ul>\n<\/div>
    <\/div><\/div><\/span>

    Redirection (3xx)<\/p><\/span>

    <\/div><\/div><\/div>

    redirection such as 301 or 302 was encountered. Ideally each internal link should link to a response with status code 200 and avoid links that cause the Spider to take extra steps in an effort to safeguard the Crawling Budget.<\/p>\n<\/div>

    <\/span><\/div>

    What to Do<\/strong><\/p>\n

      \n
    • Check what the destination of the redirection is (Check the outlinks of the returned URL). If you find a loop condition the Seo Spider fails to do a full scan.<\/li>\n
    • External tab: the seo spider treats several subdomains as external and does not scan them by default. To resolve this condition enable the option scan all subdomains. Configuration > Spider > Scan all subdomains.<\/li>\n
    • Does the site require cookies? Try disabling cookies from your browser, if you encounter the problem enable the “Allow Cookies” function in Seo Spider. Configuration > Spider > Advanced tab > Allow cookies. This condition occurs because the SEO Spider is redirected to a URL where a cookie is left, but does not accept cookies.<\/li>\n<\/ul>\n<\/div>
      <\/div><\/div><\/span>

      Bad request | 403 Forbidden<\/p><\/span>

      <\/div><\/div><\/div>

      The server fails or does not want to process the request and is denying the SEO Spider’s request to display the requested URL.<\/p>\n<\/div>

      <\/span><\/div>

      What to do<\/strong><\/p>\n

        \n
      • If the page can be seen through your browser, try setting Chrome as the user agent (Configuration > User-Agent). The site probably denies the page to our Seo Spider for security issues.<\/li>\n<\/ul>\n<\/div>
        <\/span><\/div>
          \n
        • If the page responds with error 404 the server is indicating that it no longer exists. If even by changing the user-agent in Chrome you find yourself in the same condition presumably it is a website problem, otherwise if with Chrome the page is returned ( or you can view it with the browser ) and you can start crawling probably the server was blocking our user-agent for a security issue.<\/li>\n
        • If the page has error 429, it means that too many requests have been made to the server in a given period of time.<\/li>\n<\/ul>\n

          What to do<\/strong><\/p>\n

          Can you view your site in the browser or does this show a similar error message? In this case you can lower the crawl speed and\/or test a different user agent such as Chrome or Googlebot.<\/p>\n<\/div>

          <\/div><\/div><\/span>

          Server Error (5xx)<\/p><\/span>

          <\/div><\/div><\/div>

          The server failed to fulfill an apparently valid request. This can include common responses such as 500 Internal Server Errors, 502 and 503 Server Unavailable.<\/p>\n<\/div>

          <\/span><\/div>

          What to do<\/strong><\/p>\n

            \n
          • Can you see your site in the browser or is it down? If the page can be seen, change user-agent and use Chrome as the user agent (Configuration > User-Agent).<\/li>\n<\/ul>\n<\/div><\/div><\/div><\/div><\/div>\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[38],"tags":[52,60],"class_list":["post-3318","post","type-post","status-publish","format-standard","hentry","category-guide-screaming-frog","tag-beginner-en","tag-guide"],"acf":[],"yoast_head":"\nHow to handle Seo Spider crawling issues<\/title>\n<meta name=\"description\" content=\"Learn how to solve any Seo Spider crawling problems with practical cases and best practices. No one will stop your advanced Seo Audits.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/screamingfrog.club\/en\/seo-spider-crawling-issues\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How to handle Seo Spider crawling issues\" \/>\n<meta property=\"og:description\" content=\"Learn how to solve any Seo Spider crawling problems with practical cases and best practices. No one will stop your advanced Seo Audits.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/screamingfrog.club\/en\/seo-spider-crawling-issues\/\" \/>\n<meta property=\"og:site_name\" content=\"Screaming Frog Club\" \/>\n<meta property=\"article:published_time\" content=\"2024-03-21T13:49:43+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-07-03T07:11:48+00:00\" \/>\n<meta property=\"og:image\" content=\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/SF_Back_dark.png\" \/>\n<meta name=\"author\" content=\"raffaele.visintin@gmail.com\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"raffaele.visintin@gmail.com\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"8 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/screamingfrog.club\/en\/seo-spider-crawling-issues\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/screamingfrog.club\/en\/seo-spider-crawling-issues\/\"},\"author\":{\"name\":\"raffaele.visintin@gmail.com\",\"@id\":\"https:\/\/screamingfrog.club\/en\/#\/schema\/person\/cd9ee509ae86128e5e339f9e3de1bc73\"},\"headline\":\"Seo Spider crawling issues\",\"datePublished\":\"2024-03-21T13:49:43+00:00\",\"dateModified\":\"2024-07-03T07:11:48+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/screamingfrog.club\/en\/seo-spider-crawling-issues\/\"},\"wordCount\":4997,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/screamingfrog.club\/en\/#organization\"},\"keywords\":[\"Beginner\",\"Guide\"],\"articleSection\":[\"Guide Screaming Frog\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/screamingfrog.club\/en\/seo-spider-crawling-issues\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/screamingfrog.club\/en\/seo-spider-crawling-issues\/\",\"url\":\"https:\/\/screamingfrog.club\/en\/seo-spider-crawling-issues\/\",\"name\":\"How to handle Seo Spider crawling issues\",\"isPartOf\":{\"@id\":\"https:\/\/screamingfrog.club\/en\/#website\"},\"datePublished\":\"2024-03-21T13:49:43+00:00\",\"dateModified\":\"2024-07-03T07:11:48+00:00\",\"description\":\"Learn how to solve any Seo Spider crawling problems with practical cases and best practices. No one will stop your advanced Seo Audits.\",\"breadcrumb\":{\"@id\":\"https:\/\/screamingfrog.club\/en\/seo-spider-crawling-issues\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/screamingfrog.club\/en\/seo-spider-crawling-issues\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/screamingfrog.club\/en\/seo-spider-crawling-issues\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/screamingfrog.club\/en\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Seo Spider crawling issues\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/screamingfrog.club\/en\/#website\",\"url\":\"https:\/\/screamingfrog.club\/en\/\",\"name\":\"Screaming Frog Club\",\"description\":\"Guide e Tutorial sul Seo SPider\",\"publisher\":{\"@id\":\"https:\/\/screamingfrog.club\/en\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/screamingfrog.club\/en\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/screamingfrog.club\/en\/#organization\",\"name\":\"Screaming Frog Club\",\"url\":\"https:\/\/screamingfrog.club\/en\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/screamingfrog.club\/en\/#\/schema\/logo\/image\/\",\"url\":\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/02\/SF_Club_Logo-2.png\",\"contentUrl\":\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/02\/SF_Club_Logo-2.png\",\"width\":117,\"height\":67,\"caption\":\"Screaming Frog Club\"},\"image\":{\"@id\":\"https:\/\/screamingfrog.club\/en\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/screamingfrog.club\/en\/#\/schema\/person\/cd9ee509ae86128e5e339f9e3de1bc73\",\"name\":\"raffaele.visintin@gmail.com\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/screamingfrog.club\/wp-content\/litespeed\/avatar\/5b2dd661ff57d360794386547fd7a2dd.jpg?ver=1775343410\",\"url\":\"https:\/\/screamingfrog.club\/wp-content\/litespeed\/avatar\/5b2dd661ff57d360794386547fd7a2dd.jpg?ver=1775343410\",\"contentUrl\":\"https:\/\/screamingfrog.club\/wp-content\/litespeed\/avatar\/5b2dd661ff57d360794386547fd7a2dd.jpg?ver=1775343410\",\"caption\":\"raffaele.visintin@gmail.com\"},\"sameAs\":[\"http:\/\/screamingfrog.club\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"How to handle Seo Spider crawling issues","description":"Learn how to solve any Seo Spider crawling problems with practical cases and best practices. No one will stop your advanced Seo Audits.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/screamingfrog.club\/en\/seo-spider-crawling-issues\/","og_locale":"en_US","og_type":"article","og_title":"How to handle Seo Spider crawling issues","og_description":"Learn how to solve any Seo Spider crawling problems with practical cases and best practices. No one will stop your advanced Seo Audits.","og_url":"https:\/\/screamingfrog.club\/en\/seo-spider-crawling-issues\/","og_site_name":"Screaming Frog Club","article_published_time":"2024-03-21T13:49:43+00:00","article_modified_time":"2024-07-03T07:11:48+00:00","og_image":[{"url":"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/SF_Back_dark.png","type":"","width":"","height":""}],"author":"raffaele.visintin@gmail.com","twitter_card":"summary_large_image","twitter_misc":{"Written by":"raffaele.visintin@gmail.com","Est. reading time":"8 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/screamingfrog.club\/en\/seo-spider-crawling-issues\/#article","isPartOf":{"@id":"https:\/\/screamingfrog.club\/en\/seo-spider-crawling-issues\/"},"author":{"name":"raffaele.visintin@gmail.com","@id":"https:\/\/screamingfrog.club\/en\/#\/schema\/person\/cd9ee509ae86128e5e339f9e3de1bc73"},"headline":"Seo Spider crawling issues","datePublished":"2024-03-21T13:49:43+00:00","dateModified":"2024-07-03T07:11:48+00:00","mainEntityOfPage":{"@id":"https:\/\/screamingfrog.club\/en\/seo-spider-crawling-issues\/"},"wordCount":4997,"commentCount":0,"publisher":{"@id":"https:\/\/screamingfrog.club\/en\/#organization"},"keywords":["Beginner","Guide"],"articleSection":["Guide Screaming Frog"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/screamingfrog.club\/en\/seo-spider-crawling-issues\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/screamingfrog.club\/en\/seo-spider-crawling-issues\/","url":"https:\/\/screamingfrog.club\/en\/seo-spider-crawling-issues\/","name":"How to handle Seo Spider crawling issues","isPartOf":{"@id":"https:\/\/screamingfrog.club\/en\/#website"},"datePublished":"2024-03-21T13:49:43+00:00","dateModified":"2024-07-03T07:11:48+00:00","description":"Learn how to solve any Seo Spider crawling problems with practical cases and best practices. No one will stop your advanced Seo Audits.","breadcrumb":{"@id":"https:\/\/screamingfrog.club\/en\/seo-spider-crawling-issues\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/screamingfrog.club\/en\/seo-spider-crawling-issues\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/screamingfrog.club\/en\/seo-spider-crawling-issues\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/screamingfrog.club\/en\/"},{"@type":"ListItem","position":2,"name":"Seo Spider crawling issues"}]},{"@type":"WebSite","@id":"https:\/\/screamingfrog.club\/en\/#website","url":"https:\/\/screamingfrog.club\/en\/","name":"Screaming Frog Club","description":"Guide e Tutorial sul Seo SPider","publisher":{"@id":"https:\/\/screamingfrog.club\/en\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/screamingfrog.club\/en\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/screamingfrog.club\/en\/#organization","name":"Screaming Frog Club","url":"https:\/\/screamingfrog.club\/en\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/screamingfrog.club\/en\/#\/schema\/logo\/image\/","url":"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/02\/SF_Club_Logo-2.png","contentUrl":"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/02\/SF_Club_Logo-2.png","width":117,"height":67,"caption":"Screaming Frog Club"},"image":{"@id":"https:\/\/screamingfrog.club\/en\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/screamingfrog.club\/en\/#\/schema\/person\/cd9ee509ae86128e5e339f9e3de1bc73","name":"raffaele.visintin@gmail.com","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/screamingfrog.club\/wp-content\/litespeed\/avatar\/5b2dd661ff57d360794386547fd7a2dd.jpg?ver=1775343410","url":"https:\/\/screamingfrog.club\/wp-content\/litespeed\/avatar\/5b2dd661ff57d360794386547fd7a2dd.jpg?ver=1775343410","contentUrl":"https:\/\/screamingfrog.club\/wp-content\/litespeed\/avatar\/5b2dd661ff57d360794386547fd7a2dd.jpg?ver=1775343410","caption":"raffaele.visintin@gmail.com"},"sameAs":["http:\/\/screamingfrog.club"]}]}},"_links":{"self":[{"href":"https:\/\/screamingfrog.club\/en\/wp-json\/wp\/v2\/posts\/3318","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/screamingfrog.club\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/screamingfrog.club\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/screamingfrog.club\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/screamingfrog.club\/en\/wp-json\/wp\/v2\/comments?post=3318"}],"version-history":[{"count":0,"href":"https:\/\/screamingfrog.club\/en\/wp-json\/wp\/v2\/posts\/3318\/revisions"}],"wp:attachment":[{"href":"https:\/\/screamingfrog.club\/en\/wp-json\/wp\/v2\/media?parent=3318"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/screamingfrog.club\/en\/wp-json\/wp\/v2\/categories?post=3318"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/screamingfrog.club\/en\/wp-json\/wp\/v2\/tags?post=3318"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}