{"id":3499,"date":"2024-03-25T10:12:58","date_gmt":"2024-03-25T10:12:58","guid":{"rendered":"http:\/\/screamingfrog.club\/crawl-of-subfolders\/"},"modified":"2024-07-03T07:07:19","modified_gmt":"2024-07-03T07:07:19","slug":"crawl-of-subfolders","status":"publish","type":"post","link":"https:\/\/screamingfrog.club\/en\/crawl-of-subfolders\/","title":{"rendered":"Crawl of Subfolders"},"content":{"rendered":"

<\/div><\/div><\/span>

Crawl Subfolder<\/h2><\/span>
<\/div><\/div><\/div>

By default the Seo Spider performs a full scan of the “root domain” or subdomain but it may happen due to time requirements or to meet a specific focus that you need to do only a partial scan of a website.
\nLet’s see together how to get a crawl of a specific subfolder.<\/p>\n<\/div><\/div><\/div><\/div><\/div>

Example:
\n<\/span>You want to analyze the errors and critical issues present only in the blog area of the Screaming Frog website:
\n<\/span>https:\/\/www.screamingfrog.co.uk\/blog\/ <\/span><\/p>\n<\/div><\/div><\/div><\/div><\/div>

For this partial crawl, simply enter the URL of the subfolder, start the Seo Spider, and let the crawl conclude. In this way you will find the main window populated with subfolder URLs, respective filters and metrics.<\/p>\n

<\/div>\n<\/div>
<\/a><\/span><\/div>

As you will see during your tests, in addition to the URLs in the “subfolder,” there will also be additional URLs that do not belong to the area you had restricted.<\/p>\n

However, this anomalous behavior is not a bug in the system but represents an inherent feature of Seo Spider.<\/p>\n

In fact, the crawler considers the subdirectory in the first instance but also contextually follows all the links present by proposing the data of the linked URLs.<\/p>\n

This expedient is vital for more accurate analysis and helps you discover any broken links starting from the subdirectory to external URLs.<\/p>\n<\/div><\/div><\/div><\/div><\/div>

Configuration > Spider > Crawl Behaviour > Check Links Outside of Start Folder<\/span><\/p>\n<\/div><\/div><\/div><\/div><\/div>

Please note: Seo Spider is only able to scan a subfolder with this syntax “..\/foldername\/<\/strong>“; if you forget the final “\/” Screaming Frog will not recognize it. The same situation occurs when the version with a final “\/” has a redirect to the version without.<\/p>\n

Another alternative for scanning a subfolder is to use the “include” function by taking advantage of the RegEX rules.<\/p>\n<\/div><\/div><\/div><\/div><\/div>

Configuration > Includes >.*\/blog.*<\/span><\/p>\n<\/div><\/div><\/div><\/div><\/div><\/p>\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[38],"tags":[52,60],"class_list":["post-3499","post","type-post","status-publish","format-standard","hentry","category-guide-screaming-frog","tag-beginner-en","tag-guide"],"acf":[],"yoast_head":"\nCrawl of subfolders | Screaming Frog<\/title>\n<meta name=\"description\" content=\"Step by step tutorial in using the Seo Spider Screaming Frog to scan a subfolder or area of a website.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/screamingfrog.club\/en\/crawl-of-subfolders\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Crawl of subfolders | Screaming Frog\" \/>\n<meta property=\"og:description\" content=\"Step by step tutorial in using the Seo Spider Screaming Frog to scan a subfolder or area of a website.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/screamingfrog.club\/en\/crawl-of-subfolders\/\" \/>\n<meta property=\"og:site_name\" content=\"Screaming Frog Club\" \/>\n<meta property=\"article:published_time\" content=\"2024-03-25T10:12:58+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-07-03T07:07:19+00:00\" \/>\n<meta property=\"og:image\" content=\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/SF_Back_dark.png\" \/>\n<meta name=\"author\" content=\"raffaele.visintin@gmail.com\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"raffaele.visintin@gmail.com\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/screamingfrog.club\/en\/crawl-of-subfolders\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/screamingfrog.club\/en\/crawl-of-subfolders\/\"},\"author\":{\"name\":\"raffaele.visintin@gmail.com\",\"@id\":\"https:\/\/screamingfrog.club\/en\/#\/schema\/person\/cd9ee509ae86128e5e339f9e3de1bc73\"},\"headline\":\"Crawl of Subfolders\",\"datePublished\":\"2024-03-25T10:12:58+00:00\",\"dateModified\":\"2024-07-03T07:07:19+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/screamingfrog.club\/en\/crawl-of-subfolders\/\"},\"wordCount\":3875,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/screamingfrog.club\/en\/#organization\"},\"keywords\":[\"Beginner\",\"Guide\"],\"articleSection\":[\"Guide Screaming Frog\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/screamingfrog.club\/en\/crawl-of-subfolders\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/screamingfrog.club\/en\/crawl-of-subfolders\/\",\"url\":\"https:\/\/screamingfrog.club\/en\/crawl-of-subfolders\/\",\"name\":\"Crawl of subfolders | Screaming Frog\",\"isPartOf\":{\"@id\":\"https:\/\/screamingfrog.club\/en\/#website\"},\"datePublished\":\"2024-03-25T10:12:58+00:00\",\"dateModified\":\"2024-07-03T07:07:19+00:00\",\"description\":\"Step by step tutorial in using the Seo Spider Screaming Frog to scan a subfolder or area of a website.\",\"breadcrumb\":{\"@id\":\"https:\/\/screamingfrog.club\/en\/crawl-of-subfolders\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/screamingfrog.club\/en\/crawl-of-subfolders\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/screamingfrog.club\/en\/crawl-of-subfolders\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/screamingfrog.club\/en\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Crawl of Subfolders\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/screamingfrog.club\/en\/#website\",\"url\":\"https:\/\/screamingfrog.club\/en\/\",\"name\":\"Screaming Frog Club\",\"description\":\"Guide e Tutorial sul Seo SPider\",\"publisher\":{\"@id\":\"https:\/\/screamingfrog.club\/en\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/screamingfrog.club\/en\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/screamingfrog.club\/en\/#organization\",\"name\":\"Screaming Frog Club\",\"url\":\"https:\/\/screamingfrog.club\/en\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/screamingfrog.club\/en\/#\/schema\/logo\/image\/\",\"url\":\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/02\/SF_Club_Logo-2.png\",\"contentUrl\":\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/02\/SF_Club_Logo-2.png\",\"width\":117,\"height\":67,\"caption\":\"Screaming Frog Club\"},\"image\":{\"@id\":\"https:\/\/screamingfrog.club\/en\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/screamingfrog.club\/en\/#\/schema\/person\/cd9ee509ae86128e5e339f9e3de1bc73\",\"name\":\"raffaele.visintin@gmail.com\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/screamingfrog.club\/wp-content\/litespeed\/avatar\/5b2dd661ff57d360794386547fd7a2dd.jpg?ver=1775343410\",\"url\":\"https:\/\/screamingfrog.club\/wp-content\/litespeed\/avatar\/5b2dd661ff57d360794386547fd7a2dd.jpg?ver=1775343410\",\"contentUrl\":\"https:\/\/screamingfrog.club\/wp-content\/litespeed\/avatar\/5b2dd661ff57d360794386547fd7a2dd.jpg?ver=1775343410\",\"caption\":\"raffaele.visintin@gmail.com\"},\"sameAs\":[\"http:\/\/screamingfrog.club\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Crawl of subfolders | Screaming Frog","description":"Step by step tutorial in using the Seo Spider Screaming Frog to scan a subfolder or area of a website.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/screamingfrog.club\/en\/crawl-of-subfolders\/","og_locale":"en_US","og_type":"article","og_title":"Crawl of subfolders | Screaming Frog","og_description":"Step by step tutorial in using the Seo Spider Screaming Frog to scan a subfolder or area of a website.","og_url":"https:\/\/screamingfrog.club\/en\/crawl-of-subfolders\/","og_site_name":"Screaming Frog Club","article_published_time":"2024-03-25T10:12:58+00:00","article_modified_time":"2024-07-03T07:07:19+00:00","og_image":[{"url":"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/SF_Back_dark.png","type":"","width":"","height":""}],"author":"raffaele.visintin@gmail.com","twitter_card":"summary_large_image","twitter_misc":{"Written by":"raffaele.visintin@gmail.com","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/screamingfrog.club\/en\/crawl-of-subfolders\/#article","isPartOf":{"@id":"https:\/\/screamingfrog.club\/en\/crawl-of-subfolders\/"},"author":{"name":"raffaele.visintin@gmail.com","@id":"https:\/\/screamingfrog.club\/en\/#\/schema\/person\/cd9ee509ae86128e5e339f9e3de1bc73"},"headline":"Crawl of Subfolders","datePublished":"2024-03-25T10:12:58+00:00","dateModified":"2024-07-03T07:07:19+00:00","mainEntityOfPage":{"@id":"https:\/\/screamingfrog.club\/en\/crawl-of-subfolders\/"},"wordCount":3875,"commentCount":0,"publisher":{"@id":"https:\/\/screamingfrog.club\/en\/#organization"},"keywords":["Beginner","Guide"],"articleSection":["Guide Screaming Frog"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/screamingfrog.club\/en\/crawl-of-subfolders\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/screamingfrog.club\/en\/crawl-of-subfolders\/","url":"https:\/\/screamingfrog.club\/en\/crawl-of-subfolders\/","name":"Crawl of subfolders | Screaming Frog","isPartOf":{"@id":"https:\/\/screamingfrog.club\/en\/#website"},"datePublished":"2024-03-25T10:12:58+00:00","dateModified":"2024-07-03T07:07:19+00:00","description":"Step by step tutorial in using the Seo Spider Screaming Frog to scan a subfolder or area of a website.","breadcrumb":{"@id":"https:\/\/screamingfrog.club\/en\/crawl-of-subfolders\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/screamingfrog.club\/en\/crawl-of-subfolders\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/screamingfrog.club\/en\/crawl-of-subfolders\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/screamingfrog.club\/en\/"},{"@type":"ListItem","position":2,"name":"Crawl of Subfolders"}]},{"@type":"WebSite","@id":"https:\/\/screamingfrog.club\/en\/#website","url":"https:\/\/screamingfrog.club\/en\/","name":"Screaming Frog Club","description":"Guide e Tutorial sul Seo SPider","publisher":{"@id":"https:\/\/screamingfrog.club\/en\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/screamingfrog.club\/en\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/screamingfrog.club\/en\/#organization","name":"Screaming Frog Club","url":"https:\/\/screamingfrog.club\/en\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/screamingfrog.club\/en\/#\/schema\/logo\/image\/","url":"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/02\/SF_Club_Logo-2.png","contentUrl":"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/02\/SF_Club_Logo-2.png","width":117,"height":67,"caption":"Screaming Frog Club"},"image":{"@id":"https:\/\/screamingfrog.club\/en\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/screamingfrog.club\/en\/#\/schema\/person\/cd9ee509ae86128e5e339f9e3de1bc73","name":"raffaele.visintin@gmail.com","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/screamingfrog.club\/wp-content\/litespeed\/avatar\/5b2dd661ff57d360794386547fd7a2dd.jpg?ver=1775343410","url":"https:\/\/screamingfrog.club\/wp-content\/litespeed\/avatar\/5b2dd661ff57d360794386547fd7a2dd.jpg?ver=1775343410","contentUrl":"https:\/\/screamingfrog.club\/wp-content\/litespeed\/avatar\/5b2dd661ff57d360794386547fd7a2dd.jpg?ver=1775343410","caption":"raffaele.visintin@gmail.com"},"sameAs":["http:\/\/screamingfrog.club"]}]}},"_links":{"self":[{"href":"https:\/\/screamingfrog.club\/en\/wp-json\/wp\/v2\/posts\/3499","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/screamingfrog.club\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/screamingfrog.club\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/screamingfrog.club\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/screamingfrog.club\/en\/wp-json\/wp\/v2\/comments?post=3499"}],"version-history":[{"count":0,"href":"https:\/\/screamingfrog.club\/en\/wp-json\/wp\/v2\/posts\/3499\/revisions"}],"wp:attachment":[{"href":"https:\/\/screamingfrog.club\/en\/wp-json\/wp\/v2\/media?parent=3499"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/screamingfrog.club\/en\/wp-json\/wp\/v2\/categories?post=3499"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/screamingfrog.club\/en\/wp-json\/wp\/v2\/tags?post=3499"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}