SEO SPIDER<\/h2><\/span><\/div><\/div><\/div>Overview of the Seo spider console and main areas of interest during Seo Audits.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/SCREAMING-FROG-CONSOLE.jpeg\")
<\/a><\/span><\/div><\/div><\/div><\/span>Main menu<\/h3><\/span><\/div><\/div><\/div>It represents the heart of the console and allows you to manage the most relevant settings that will affect your Seo Audit tenico.<\/p>\n<\/div>
<\/div><\/div><\/span>File<\/h4><\/span><\/div><\/div><\/div>You can create a new project, open previous scans in “Database Storage”(Crawl) or “Memory Storage” (Open Recent) mode, manage crawl scheduling (Scheduling), and change default scan configurations by also creating multiple ready-to-use custom profiles.<\/p>\n<\/div><\/div><\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/SF-menu-principale-seo-spider.png\" "\\"SF-menu-principale-seo-spider\\"")
<\/span><\/div><\/div><\/div><\/div><\/div><\/div><\/span>Scan Mode<\/h4><\/span><\/div><\/div><\/div>It allows you to choose which scanning mode to use.<\/p>\n
\n- Spider<\/strong>: default mode for a complete scan of a Web site.<\/li>\n
- List<\/strong>: allows you to do specific crawls from lists of URLs that you can manually upload or copy into Seo Spider.<\/li>\n
- Compare<\/strong>: represents the latest feature added by Screaming Frog developers to compare two previously performed scans and analyze the differences in terms of URLs and filters (useful for staging sites vs. live sites, desktop vs. mobile version etc.).<\/li>\n
- Serp<\/strong>: allows you to preview snippets by uploading a tabulated .csv or .xls file.<\/li>\n<\/ul>\n<\/div><\/div><\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/SF-mode-seo-spider.png\")
<\/a><\/span><\/div><\/div><\/div><\/div><\/div><\/div><\/span>Configuration<\/h4><\/span><\/div><\/div><\/div>This menu item is among the most important of the Seo Spider and corresponds to the console engine putting at your disposal a whole series of customizations to perform Advanced Seo Audits and links to external services such as Google Analytics, Search Console, Lightspeed, Moz, Ahrefs.<\/p>\n
\n- Spider<\/strong>: through this option you can manage the behavior of the Spider, control which resources it scans, limit its raids, define the parameters of some filters, and decide whether to scan static HTML or also include Javascript rendering.<\/li>\n
- Content<\/strong>: you can define the area in which the Seo Spider should examine the content, set tolerability indices to consider a piece of content as duplicate, and choose various options regarding the vocabulary to be used for grammar and typing check.<\/li>\n
- Robots.txt<\/strong>: this option allows you to custom define the Robots.txt file by editing it directly in the Seo Spider or what attitude the Spider should have towards this file. In summary, you can decide whether the Seo Spider should follow the guidelines or ignore them for crawling.<\/li>\n
- URL Rewriting<\/strong>: allows you to rewrite URLs before scanning. Very useful for avoiding parameterized URLs in crawl results and other advanced best practices using RegEX syntax.<\/li>\n
- Include\/Exclude<\/strong>: These features allow you to include or exclude URLs from crawling by setting RegEX rules that you can customize based on the focus of your analysis.<\/li>\n
- Speed<\/strong>: allows you to define the scanning speed of the Spider Tool; very useful for not overloading the server of the website to be scanned.<\/li>\n
- User Agent<\/strong>: as you know Screaming Frog is a Spider emulator, with this option you can choose which User Agent to scan with choosing from GoogleBot, GoogleBot for smartphone, Baidu for Eastern market, Yandex for Russian market, Yahoo\/Bing etc. based on your market and focus.<\/li>\n
- HTTP<\/strong> Header: allows you to configure some HTTP Header options in a customized way.<\/li>\n
- Custom<\/strong>: through this item you are able to scan particular HTML elements and extract them to your liking for very custom analysis. Also very useful for defining the position of Links within the Html document.<\/li>\n
- Api Access<\/strong>: this feature allows you to set up API connections with services such as Google Analytics, Search Console, Ahrefs, Majestic, PageSpeed Insight, and Moz. Useful integrations to obtain comprehensive Seo Audits with granular data.<\/li>\n
- Authentication<\/strong>: allows you to scan a website that requires authentication.<\/li>\n
- System<\/strong>: allows you to choose the mode of storage of scan data.<\/li>\n<\/ul>\n<\/div><\/div><\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/menu-configurazioni-seo-spider-1.png\")
<\/a><\/span><\/div><\/div><\/div><\/div><\/div><\/div><\/span>Bulk Export<\/h4><\/span><\/div><\/div><\/div>(Mass Exports): the items included in this menu allow you to download almost all the most relevant data from your scans.
\nThese documents are in “.csv”, “.xlsx” and “.gsheet” formats. Below are the metrics available to you:<\/p>\n
\n- Links;<\/li>\n
- Web;<\/li>\n
- Path Type;<\/li>\n
- Security;<\/li>\n
- Response Code;<\/li>\n
- Content;<\/li>\n
- Images;<\/li>\n
- Canonicals;<\/li>\n
- Directives;<\/li>\n
- AMP;<\/li>\n
- Structured Data;<\/li>\n
- Sitemaps;<\/li>\n
- Custom Search;<\/li>\n
- Custom Extraction;<\/li>\n<\/ul>\n<\/div><\/div><\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/SF-menu-bulk-export.png\")
<\/a><\/span><\/div><\/div><\/div><\/div><\/div><\/div><\/span>Report<\/h4><\/span><\/div><\/div><\/div>This menu item allows you to find a whole series of reports that focus on redirects, canonicals, website pageviews, any content that is not considered safe, orphan page detection, structured data, and a lot of other useful information to export and manage during Seo optimization.<\/p>\n<\/div><\/div><\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/SF-menu-report.png\")
<\/a><\/span><\/div><\/div><\/div><\/div><\/div><\/div><\/span>Visualisation<\/h4><\/span><\/div><\/div><\/div>This feature allows you to visualize data regarding URLs in a very intuitive and impactful way according to different graphical modes. Views are interactive and ideal for focusing on different nodes or folders in the site structure and providing more specific and timely control.<\/p>\n<\/div><\/div><\/div>
Overview of the Seo spider console and main areas of interest during Seo Audits.<\/p>\n<\/div>
<\/a><\/span><\/div>Main menu<\/h3><\/span><\/div><\/div><\/div>It represents the heart of the console and allows you to manage the most relevant settings that will affect your Seo Audit tenico.<\/p>\n<\/div>
<\/div><\/div><\/span>File<\/h4><\/span><\/div><\/div><\/div>You can create a new project, open previous scans in “Database Storage”(Crawl) or “Memory Storage” (Open Recent) mode, manage crawl scheduling (Scheduling), and change default scan configurations by also creating multiple ready-to-use custom profiles.<\/p>\n<\/div><\/div><\/div>
<\/span><\/div><\/div><\/div><\/div><\/div><\/div><\/span>Scan Mode<\/h4><\/span><\/div><\/div><\/div>It allows you to choose which scanning mode to use.<\/p>\n
\n- Spider<\/strong>: default mode for a complete scan of a Web site.<\/li>\n
- List<\/strong>: allows you to do specific crawls from lists of URLs that you can manually upload or copy into Seo Spider.<\/li>\n
- Compare<\/strong>: represents the latest feature added by Screaming Frog developers to compare two previously performed scans and analyze the differences in terms of URLs and filters (useful for staging sites vs. live sites, desktop vs. mobile version etc.).<\/li>\n
- Serp<\/strong>: allows you to preview snippets by uploading a tabulated .csv or .xls file.<\/li>\n<\/ul>\n<\/div><\/div><\/div><\/a><\/span><\/div><\/div><\/div><\/div><\/div><\/div><\/span>
Configuration<\/h4><\/span><\/div><\/div><\/div>This menu item is among the most important of the Seo Spider and corresponds to the console engine putting at your disposal a whole series of customizations to perform Advanced Seo Audits and links to external services such as Google Analytics, Search Console, Lightspeed, Moz, Ahrefs.<\/p>\n
\n- Spider<\/strong>: through this option you can manage the behavior of the Spider, control which resources it scans, limit its raids, define the parameters of some filters, and decide whether to scan static HTML or also include Javascript rendering.<\/li>\n
- Content<\/strong>: you can define the area in which the Seo Spider should examine the content, set tolerability indices to consider a piece of content as duplicate, and choose various options regarding the vocabulary to be used for grammar and typing check.<\/li>\n
- Robots.txt<\/strong>: this option allows you to custom define the Robots.txt file by editing it directly in the Seo Spider or what attitude the Spider should have towards this file. In summary, you can decide whether the Seo Spider should follow the guidelines or ignore them for crawling.<\/li>\n
- URL Rewriting<\/strong>: allows you to rewrite URLs before scanning. Very useful for avoiding parameterized URLs in crawl results and other advanced best practices using RegEX syntax.<\/li>\n
- Include\/Exclude<\/strong>: These features allow you to include or exclude URLs from crawling by setting RegEX rules that you can customize based on the focus of your analysis.<\/li>\n
- Speed<\/strong>: allows you to define the scanning speed of the Spider Tool; very useful for not overloading the server of the website to be scanned.<\/li>\n
- User Agent<\/strong>: as you know Screaming Frog is a Spider emulator, with this option you can choose which User Agent to scan with choosing from GoogleBot, GoogleBot for smartphone, Baidu for Eastern market, Yandex for Russian market, Yahoo\/Bing etc. based on your market and focus.<\/li>\n
- HTTP<\/strong> Header: allows you to configure some HTTP Header options in a customized way.<\/li>\n
- Custom<\/strong>: through this item you are able to scan particular HTML elements and extract them to your liking for very custom analysis. Also very useful for defining the position of Links within the Html document.<\/li>\n
- Api Access<\/strong>: this feature allows you to set up API connections with services such as Google Analytics, Search Console, Ahrefs, Majestic, PageSpeed Insight, and Moz. Useful integrations to obtain comprehensive Seo Audits with granular data.<\/li>\n
- Authentication<\/strong>: allows you to scan a website that requires authentication.<\/li>\n
- System<\/strong>: allows you to choose the mode of storage of scan data.<\/li>\n<\/ul>\n<\/div><\/div><\/div><\/a><\/span><\/div><\/div><\/div><\/div><\/div><\/div><\/span>
Bulk Export<\/h4><\/span><\/div><\/div><\/div>(Mass Exports): the items included in this menu allow you to download almost all the most relevant data from your scans.
\nThese documents are in “.csv”, “.xlsx” and “.gsheet” formats. Below are the metrics available to you:<\/p>\n
\n- Links;<\/li>\n
- Web;<\/li>\n
- Path Type;<\/li>\n
- Security;<\/li>\n
- Response Code;<\/li>\n
- Content;<\/li>\n
- Images;<\/li>\n
- Canonicals;<\/li>\n
- Directives;<\/li>\n
- AMP;<\/li>\n
- Structured Data;<\/li>\n
- Sitemaps;<\/li>\n
- Custom Search;<\/li>\n
- Custom Extraction;<\/li>\n<\/ul>\n<\/div><\/div><\/div><\/a><\/span><\/div><\/div><\/div><\/div><\/div><\/div><\/span>
Report<\/h4><\/span><\/div><\/div><\/div>This menu item allows you to find a whole series of reports that focus on redirects, canonicals, website pageviews, any content that is not considered safe, orphan page detection, structured data, and a lot of other useful information to export and manage during Seo optimization.<\/p>\n<\/div><\/div><\/div>
<\/a><\/span><\/div><\/div><\/div><\/div><\/div><\/div><\/span>Visualisation<\/h4><\/span><\/div><\/div><\/div>This feature allows you to visualize data regarding URLs in a very intuitive and impactful way according to different graphical modes. Views are interactive and ideal for focusing on different nodes or folders in the site structure and providing more specific and timely control.<\/p>\n<\/div><\/div><\/div>
It represents the heart of the console and allows you to manage the most relevant settings that will affect your Seo Audit tenico.<\/p>\n<\/div>
File<\/h4><\/span><\/div><\/div><\/div>You can create a new project, open previous scans in “Database Storage”(Crawl) or “Memory Storage” (Open Recent) mode, manage crawl scheduling (Scheduling), and change default scan configurations by also creating multiple ready-to-use custom profiles.<\/p>\n<\/div><\/div><\/div>
<\/span><\/div><\/div><\/div><\/div><\/div><\/div><\/span>Scan Mode<\/h4><\/span><\/div><\/div><\/div>It allows you to choose which scanning mode to use.<\/p>\n
\n- Spider<\/strong>: default mode for a complete scan of a Web site.<\/li>\n
- List<\/strong>: allows you to do specific crawls from lists of URLs that you can manually upload or copy into Seo Spider.<\/li>\n
- Compare<\/strong>: represents the latest feature added by Screaming Frog developers to compare two previously performed scans and analyze the differences in terms of URLs and filters (useful for staging sites vs. live sites, desktop vs. mobile version etc.).<\/li>\n
- Serp<\/strong>: allows you to preview snippets by uploading a tabulated .csv or .xls file.<\/li>\n<\/ul>\n<\/div><\/div><\/div><\/a><\/span><\/div><\/div><\/div><\/div><\/div><\/div><\/span>
Configuration<\/h4><\/span><\/div><\/div><\/div>This menu item is among the most important of the Seo Spider and corresponds to the console engine putting at your disposal a whole series of customizations to perform Advanced Seo Audits and links to external services such as Google Analytics, Search Console, Lightspeed, Moz, Ahrefs.<\/p>\n
\n- Spider<\/strong>: through this option you can manage the behavior of the Spider, control which resources it scans, limit its raids, define the parameters of some filters, and decide whether to scan static HTML or also include Javascript rendering.<\/li>\n
- Content<\/strong>: you can define the area in which the Seo Spider should examine the content, set tolerability indices to consider a piece of content as duplicate, and choose various options regarding the vocabulary to be used for grammar and typing check.<\/li>\n
- Robots.txt<\/strong>: this option allows you to custom define the Robots.txt file by editing it directly in the Seo Spider or what attitude the Spider should have towards this file. In summary, you can decide whether the Seo Spider should follow the guidelines or ignore them for crawling.<\/li>\n
- URL Rewriting<\/strong>: allows you to rewrite URLs before scanning. Very useful for avoiding parameterized URLs in crawl results and other advanced best practices using RegEX syntax.<\/li>\n
- Include\/Exclude<\/strong>: These features allow you to include or exclude URLs from crawling by setting RegEX rules that you can customize based on the focus of your analysis.<\/li>\n
- Speed<\/strong>: allows you to define the scanning speed of the Spider Tool; very useful for not overloading the server of the website to be scanned.<\/li>\n
- User Agent<\/strong>: as you know Screaming Frog is a Spider emulator, with this option you can choose which User Agent to scan with choosing from GoogleBot, GoogleBot for smartphone, Baidu for Eastern market, Yandex for Russian market, Yahoo\/Bing etc. based on your market and focus.<\/li>\n
- HTTP<\/strong> Header: allows you to configure some HTTP Header options in a customized way.<\/li>\n
- Custom<\/strong>: through this item you are able to scan particular HTML elements and extract them to your liking for very custom analysis. Also very useful for defining the position of Links within the Html document.<\/li>\n
- Api Access<\/strong>: this feature allows you to set up API connections with services such as Google Analytics, Search Console, Ahrefs, Majestic, PageSpeed Insight, and Moz. Useful integrations to obtain comprehensive Seo Audits with granular data.<\/li>\n
- Authentication<\/strong>: allows you to scan a website that requires authentication.<\/li>\n
- System<\/strong>: allows you to choose the mode of storage of scan data.<\/li>\n<\/ul>\n<\/div><\/div><\/div><\/a><\/span><\/div><\/div><\/div><\/div><\/div><\/div><\/span>
Bulk Export<\/h4><\/span><\/div><\/div><\/div>(Mass Exports): the items included in this menu allow you to download almost all the most relevant data from your scans.
\nThese documents are in “.csv”, “.xlsx” and “.gsheet” formats. Below are the metrics available to you:<\/p>\n
\n- Links;<\/li>\n
- Web;<\/li>\n
- Path Type;<\/li>\n
- Security;<\/li>\n
- Response Code;<\/li>\n
- Content;<\/li>\n
- Images;<\/li>\n
- Canonicals;<\/li>\n
- Directives;<\/li>\n
- AMP;<\/li>\n
- Structured Data;<\/li>\n
- Sitemaps;<\/li>\n
- Custom Search;<\/li>\n
- Custom Extraction;<\/li>\n<\/ul>\n<\/div><\/div><\/div><\/a><\/span><\/div><\/div><\/div><\/div><\/div><\/div><\/span>
Report<\/h4><\/span><\/div><\/div><\/div>This menu item allows you to find a whole series of reports that focus on redirects, canonicals, website pageviews, any content that is not considered safe, orphan page detection, structured data, and a lot of other useful information to export and manage during Seo optimization.<\/p>\n<\/div><\/div><\/div>
<\/a><\/span><\/div><\/div><\/div><\/div><\/div><\/div><\/span>Visualisation<\/h4><\/span><\/div><\/div><\/div>This feature allows you to visualize data regarding URLs in a very intuitive and impactful way according to different graphical modes. Views are interactive and ideal for focusing on different nodes or folders in the site structure and providing more specific and timely control.<\/p>\n<\/div><\/div><\/div>
You can create a new project, open previous scans in “Database Storage”(Crawl) or “Memory Storage” (Open Recent) mode, manage crawl scheduling (Scheduling), and change default scan configurations by also creating multiple ready-to-use custom profiles.<\/p>\n<\/div><\/div><\/div>
<\/span><\/div><\/div><\/div><\/div>Scan Mode<\/h4><\/span><\/div><\/div><\/div>It allows you to choose which scanning mode to use.<\/p>\n
\n- Spider<\/strong>: default mode for a complete scan of a Web site.<\/li>\n
- List<\/strong>: allows you to do specific crawls from lists of URLs that you can manually upload or copy into Seo Spider.<\/li>\n
- Compare<\/strong>: represents the latest feature added by Screaming Frog developers to compare two previously performed scans and analyze the differences in terms of URLs and filters (useful for staging sites vs. live sites, desktop vs. mobile version etc.).<\/li>\n
- Serp<\/strong>: allows you to preview snippets by uploading a tabulated .csv or .xls file.<\/li>\n<\/ul>\n<\/div><\/div><\/div><\/a><\/span><\/div><\/div><\/div><\/div><\/div><\/div><\/span>
Configuration<\/h4><\/span><\/div><\/div><\/div>This menu item is among the most important of the Seo Spider and corresponds to the console engine putting at your disposal a whole series of customizations to perform Advanced Seo Audits and links to external services such as Google Analytics, Search Console, Lightspeed, Moz, Ahrefs.<\/p>\n
\n- Spider<\/strong>: through this option you can manage the behavior of the Spider, control which resources it scans, limit its raids, define the parameters of some filters, and decide whether to scan static HTML or also include Javascript rendering.<\/li>\n
- Content<\/strong>: you can define the area in which the Seo Spider should examine the content, set tolerability indices to consider a piece of content as duplicate, and choose various options regarding the vocabulary to be used for grammar and typing check.<\/li>\n
- Robots.txt<\/strong>: this option allows you to custom define the Robots.txt file by editing it directly in the Seo Spider or what attitude the Spider should have towards this file. In summary, you can decide whether the Seo Spider should follow the guidelines or ignore them for crawling.<\/li>\n
- URL Rewriting<\/strong>: allows you to rewrite URLs before scanning. Very useful for avoiding parameterized URLs in crawl results and other advanced best practices using RegEX syntax.<\/li>\n
- Include\/Exclude<\/strong>: These features allow you to include or exclude URLs from crawling by setting RegEX rules that you can customize based on the focus of your analysis.<\/li>\n
- Speed<\/strong>: allows you to define the scanning speed of the Spider Tool; very useful for not overloading the server of the website to be scanned.<\/li>\n
- User Agent<\/strong>: as you know Screaming Frog is a Spider emulator, with this option you can choose which User Agent to scan with choosing from GoogleBot, GoogleBot for smartphone, Baidu for Eastern market, Yandex for Russian market, Yahoo\/Bing etc. based on your market and focus.<\/li>\n
- HTTP<\/strong> Header: allows you to configure some HTTP Header options in a customized way.<\/li>\n
- Custom<\/strong>: through this item you are able to scan particular HTML elements and extract them to your liking for very custom analysis. Also very useful for defining the position of Links within the Html document.<\/li>\n
- Api Access<\/strong>: this feature allows you to set up API connections with services such as Google Analytics, Search Console, Ahrefs, Majestic, PageSpeed Insight, and Moz. Useful integrations to obtain comprehensive Seo Audits with granular data.<\/li>\n
- Authentication<\/strong>: allows you to scan a website that requires authentication.<\/li>\n
- System<\/strong>: allows you to choose the mode of storage of scan data.<\/li>\n<\/ul>\n<\/div><\/div><\/div><\/a><\/span><\/div><\/div><\/div><\/div><\/div><\/div><\/span>
Bulk Export<\/h4><\/span><\/div><\/div><\/div>(Mass Exports): the items included in this menu allow you to download almost all the most relevant data from your scans.
\nThese documents are in “.csv”, “.xlsx” and “.gsheet” formats. Below are the metrics available to you:<\/p>\n
\n- Links;<\/li>\n
- Web;<\/li>\n
- Path Type;<\/li>\n
- Security;<\/li>\n
- Response Code;<\/li>\n
- Content;<\/li>\n
- Images;<\/li>\n
- Canonicals;<\/li>\n
- Directives;<\/li>\n
- AMP;<\/li>\n
- Structured Data;<\/li>\n
- Sitemaps;<\/li>\n
- Custom Search;<\/li>\n
- Custom Extraction;<\/li>\n<\/ul>\n<\/div><\/div><\/div><\/a><\/span><\/div><\/div><\/div><\/div><\/div><\/div><\/span>
Report<\/h4><\/span><\/div><\/div><\/div>This menu item allows you to find a whole series of reports that focus on redirects, canonicals, website pageviews, any content that is not considered safe, orphan page detection, structured data, and a lot of other useful information to export and manage during Seo optimization.<\/p>\n<\/div><\/div><\/div>
<\/a><\/span><\/div><\/div><\/div><\/div><\/div><\/div><\/span>Visualisation<\/h4><\/span><\/div><\/div><\/div>This feature allows you to visualize data regarding URLs in a very intuitive and impactful way according to different graphical modes. Views are interactive and ideal for focusing on different nodes or folders in the site structure and providing more specific and timely control.<\/p>\n<\/div><\/div><\/div>
It allows you to choose which scanning mode to use.<\/p>\n
- \n
- Spider<\/strong>: default mode for a complete scan of a Web site.<\/li>\n
- List<\/strong>: allows you to do specific crawls from lists of URLs that you can manually upload or copy into Seo Spider.<\/li>\n
- Compare<\/strong>: represents the latest feature added by Screaming Frog developers to compare two previously performed scans and analyze the differences in terms of URLs and filters (useful for staging sites vs. live sites, desktop vs. mobile version etc.).<\/li>\n
- Serp<\/strong>: allows you to preview snippets by uploading a tabulated .csv or .xls file.<\/li>\n<\/ul>\n<\/div><\/div><\/div>
<\/a><\/span><\/div><\/div><\/div><\/div><\/div><\/div><\/span>Configuration<\/h4><\/span>
<\/div><\/div><\/div>This menu item is among the most important of the Seo Spider and corresponds to the console engine putting at your disposal a whole series of customizations to perform Advanced Seo Audits and links to external services such as Google Analytics, Search Console, Lightspeed, Moz, Ahrefs.<\/p>\n
- \n
- Spider<\/strong>: through this option you can manage the behavior of the Spider, control which resources it scans, limit its raids, define the parameters of some filters, and decide whether to scan static HTML or also include Javascript rendering.<\/li>\n
- Content<\/strong>: you can define the area in which the Seo Spider should examine the content, set tolerability indices to consider a piece of content as duplicate, and choose various options regarding the vocabulary to be used for grammar and typing check.<\/li>\n
- Robots.txt<\/strong>: this option allows you to custom define the Robots.txt file by editing it directly in the Seo Spider or what attitude the Spider should have towards this file. In summary, you can decide whether the Seo Spider should follow the guidelines or ignore them for crawling.<\/li>\n
- URL Rewriting<\/strong>: allows you to rewrite URLs before scanning. Very useful for avoiding parameterized URLs in crawl results and other advanced best practices using RegEX syntax.<\/li>\n
- Include\/Exclude<\/strong>: These features allow you to include or exclude URLs from crawling by setting RegEX rules that you can customize based on the focus of your analysis.<\/li>\n
- Speed<\/strong>: allows you to define the scanning speed of the Spider Tool; very useful for not overloading the server of the website to be scanned.<\/li>\n
- User Agent<\/strong>: as you know Screaming Frog is a Spider emulator, with this option you can choose which User Agent to scan with choosing from GoogleBot, GoogleBot for smartphone, Baidu for Eastern market, Yandex for Russian market, Yahoo\/Bing etc. based on your market and focus.<\/li>\n
- HTTP<\/strong> Header: allows you to configure some HTTP Header options in a customized way.<\/li>\n
- Custom<\/strong>: through this item you are able to scan particular HTML elements and extract them to your liking for very custom analysis. Also very useful for defining the position of Links within the Html document.<\/li>\n
- Api Access<\/strong>: this feature allows you to set up API connections with services such as Google Analytics, Search Console, Ahrefs, Majestic, PageSpeed Insight, and Moz. Useful integrations to obtain comprehensive Seo Audits with granular data.<\/li>\n
- Authentication<\/strong>: allows you to scan a website that requires authentication.<\/li>\n
- System<\/strong>: allows you to choose the mode of storage of scan data.<\/li>\n<\/ul>\n<\/div><\/div><\/div>
<\/a><\/span><\/div><\/div><\/div><\/div><\/div><\/div><\/span>Bulk Export<\/h4><\/span>
<\/div><\/div><\/div>(Mass Exports): the items included in this menu allow you to download almost all the most relevant data from your scans.
\nThese documents are in “.csv”, “.xlsx” and “.gsheet” formats. Below are the metrics available to you:<\/p>\n- \n
- Links;<\/li>\n
- Web;<\/li>\n
- Path Type;<\/li>\n
- Security;<\/li>\n
- Response Code;<\/li>\n
- Content;<\/li>\n
- Images;<\/li>\n
- Canonicals;<\/li>\n
- Directives;<\/li>\n
- AMP;<\/li>\n
- Structured Data;<\/li>\n
- Sitemaps;<\/li>\n
- Custom Search;<\/li>\n
- Custom Extraction;<\/li>\n<\/ul>\n<\/div><\/div><\/div><\/a><\/span><\/div><\/div><\/div><\/div><\/div><\/div><\/span>
Report<\/h4><\/span>
<\/div><\/div><\/div>This menu item allows you to find a whole series of reports that focus on redirects, canonicals, website pageviews, any content that is not considered safe, orphan page detection, structured data, and a lot of other useful information to export and manage during Seo optimization.<\/p>\n<\/div><\/div><\/div>
<\/a><\/span><\/div><\/div><\/div><\/div><\/div><\/div><\/span>Visualisation<\/h4><\/span>
<\/div><\/div><\/div>This feature allows you to visualize data regarding URLs in a very intuitive and impactful way according to different graphical modes. Views are interactive and ideal for focusing on different nodes or folders in the site structure and providing more specific and timely control.<\/p>\n<\/div><\/div><\/div>
- Content<\/strong>: you can define the area in which the Seo Spider should examine the content, set tolerability indices to consider a piece of content as duplicate, and choose various options regarding the vocabulary to be used for grammar and typing check.<\/li>\n
- List<\/strong>: allows you to do specific crawls from lists of URLs that you can manually upload or copy into Seo Spider.<\/li>\n