Database Storage Mode<\/h2><\/span><\/div><\/div><\/div>To handle scanning a site from millions of URLs the “Memory Storage” mode with storage on RAM is not the recommended choice, so the first step I recommend is to switch to “Database Storage.”<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/Configurazione-metodo-di-archiviazione-screaming-frog.png\")
<\/a><\/span><\/div>![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/database-storage-spider-tool.png\" "\\"database")
<\/span><\/div>To get the maximum performance from Seo Spider with this mode you should provide an internal SSD storage card or an external one if your system supports UASP mode.<\/p>\n<\/div>
<\/div><\/div><\/span>Memory Allocation<\/h3><\/span><\/div><\/div><\/div>Having chosen the “Database Storage” mode, the second step is to increase the allocated memory.<\/p>\n
Below is a comparison of performance based on the maximum number of URLs that can be crawled.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/tabella-con-differenze-tra-database-e-ram-mode.png\" "\\"tabella")
<\/span><\/div>These requirements are just references, always start with the idea that the more memory allocated the better the performance of the Seo Spider will be.<\/p>\n<\/div>
<\/div><\/div><\/span>Attributes to be scanned<\/h3><\/span><\/div><\/div><\/div>It is not always necessary to scan an entire site so it is a good idea to tailor the crawl to the specific focuses of the analysis in place. Remember that the more data you collect, the more intense the demand for RAM will be.<\/p>\n
To choose which metrics to scan and which not to include you can directly edit the Seo Spider settings.<\/p>\n<\/div><\/div><\/div><\/div><\/div>
Configuration > Spider > Crawl<\/span><\/p>\n<\/div><\/div><\/div><\/div><\/div>![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/Elementi-e-attributi-da-scansionare-con-seo-spider.png\")
<\/a><\/span><\/div>Depending on the type of analysis you can choose to disable scanning and storage of the following items:<\/p>\n
\n- Images*;<\/li>\n
- CSS*;<\/li>\n
- JavaScript*;<\/li>\n
- SWF*.<\/li>\n<\/ul>\n
*In terms of memory it makes a lot of sense to disable these resources but in case you use JavaScript rendering mode you will have to enable them in order not to penalize rendering.<\/p>\n
\n- External Links;<\/li>\n
- Canonicals;<\/li>\n
- Pagination;<\/li>\n
- Hreflang.<\/li>\n
- AMP.<\/li>\n<\/ul>\n
Likewise, you can disable the extraction of certain non-essential elements such as “Meta Keywords” directly from the “Spider Configuration” screen or other attributes that are not strictly necessary at the time of analysis by choosing from “Page Details” (e.g. page title, word count, headings etc.), URL details, directives or structured data.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/Elementi-da-estrarre-durante-il-crawl.png\")
<\/a><\/span><\/div>In addition to the above-mentioned elements, I recommend that you disable the function related to the position of links in Html.<\/p>\n<\/div><\/div><\/div><\/div><\/div>
To handle scanning a site from millions of URLs the “Memory Storage” mode with storage on RAM is not the recommended choice, so the first step I recommend is to switch to “Database Storage.”<\/p>\n<\/div>
<\/a><\/span><\/div><\/span><\/div>To get the maximum performance from Seo Spider with this mode you should provide an internal SSD storage card or an external one if your system supports UASP mode.<\/p>\n<\/div>
Memory Allocation<\/h3><\/span><\/div><\/div><\/div>Having chosen the “Database Storage” mode, the second step is to increase the allocated memory.<\/p>\n
Below is a comparison of performance based on the maximum number of URLs that can be crawled.<\/p>\n<\/div>
<\/span><\/div>These requirements are just references, always start with the idea that the more memory allocated the better the performance of the Seo Spider will be.<\/p>\n<\/div>
<\/div><\/div><\/span>Attributes to be scanned<\/h3><\/span><\/div><\/div><\/div>It is not always necessary to scan an entire site so it is a good idea to tailor the crawl to the specific focuses of the analysis in place. Remember that the more data you collect, the more intense the demand for RAM will be.<\/p>\n
To choose which metrics to scan and which not to include you can directly edit the Seo Spider settings.<\/p>\n<\/div><\/div><\/div><\/div><\/div>
Configuration > Spider > Crawl<\/span><\/p>\n<\/div><\/div><\/div><\/div><\/div><\/a><\/span><\/div>Depending on the type of analysis you can choose to disable scanning and storage of the following items:<\/p>\n
\n- Images*;<\/li>\n
- CSS*;<\/li>\n
- JavaScript*;<\/li>\n
- SWF*.<\/li>\n<\/ul>\n
*In terms of memory it makes a lot of sense to disable these resources but in case you use JavaScript rendering mode you will have to enable them in order not to penalize rendering.<\/p>\n
\n- External Links;<\/li>\n
- Canonicals;<\/li>\n
- Pagination;<\/li>\n
- Hreflang.<\/li>\n
- AMP.<\/li>\n<\/ul>\n
Likewise, you can disable the extraction of certain non-essential elements such as “Meta Keywords” directly from the “Spider Configuration” screen or other attributes that are not strictly necessary at the time of analysis by choosing from “Page Details” (e.g. page title, word count, headings etc.), URL details, directives or structured data.<\/p>\n<\/div>
<\/a><\/span><\/div>In addition to the above-mentioned elements, I recommend that you disable the function related to the position of links in Html.<\/p>\n<\/div><\/div><\/div><\/div><\/div>
Having chosen the “Database Storage” mode, the second step is to increase the allocated memory.<\/p>\n
Below is a comparison of performance based on the maximum number of URLs that can be crawled.<\/p>\n<\/div>
<\/span><\/div>These requirements are just references, always start with the idea that the more memory allocated the better the performance of the Seo Spider will be.<\/p>\n<\/div>
Attributes to be scanned<\/h3><\/span><\/div><\/div><\/div>It is not always necessary to scan an entire site so it is a good idea to tailor the crawl to the specific focuses of the analysis in place. Remember that the more data you collect, the more intense the demand for RAM will be.<\/p>\n
To choose which metrics to scan and which not to include you can directly edit the Seo Spider settings.<\/p>\n<\/div><\/div><\/div><\/div><\/div>
Configuration > Spider > Crawl<\/span><\/p>\n<\/div><\/div><\/div><\/div><\/div><\/a><\/span><\/div>Depending on the type of analysis you can choose to disable scanning and storage of the following items:<\/p>\n
\n- Images*;<\/li>\n
- CSS*;<\/li>\n
- JavaScript*;<\/li>\n
- SWF*.<\/li>\n<\/ul>\n
*In terms of memory it makes a lot of sense to disable these resources but in case you use JavaScript rendering mode you will have to enable them in order not to penalize rendering.<\/p>\n
\n- External Links;<\/li>\n
- Canonicals;<\/li>\n
- Pagination;<\/li>\n
- Hreflang.<\/li>\n
- AMP.<\/li>\n<\/ul>\n
Likewise, you can disable the extraction of certain non-essential elements such as “Meta Keywords” directly from the “Spider Configuration” screen or other attributes that are not strictly necessary at the time of analysis by choosing from “Page Details” (e.g. page title, word count, headings etc.), URL details, directives or structured data.<\/p>\n<\/div>
<\/a><\/span><\/div>In addition to the above-mentioned elements, I recommend that you disable the function related to the position of links in Html.<\/p>\n<\/div><\/div><\/div><\/div><\/div>
It is not always necessary to scan an entire site so it is a good idea to tailor the crawl to the specific focuses of the analysis in place. Remember that the more data you collect, the more intense the demand for RAM will be.<\/p>\n
To choose which metrics to scan and which not to include you can directly edit the Seo Spider settings.<\/p>\n<\/div><\/div><\/div><\/div><\/div>
Configuration > Spider > Crawl<\/span><\/p>\n<\/div><\/div><\/div><\/div><\/div> Depending on the type of analysis you can choose to disable scanning and storage of the following items:<\/p>\n *In terms of memory it makes a lot of sense to disable these resources but in case you use JavaScript rendering mode you will have to enable them in order not to penalize rendering.<\/p>\n Likewise, you can disable the extraction of certain non-essential elements such as “Meta Keywords” directly from the “Spider Configuration” screen or other attributes that are not strictly necessary at the time of analysis by choosing from “Page Details” (e.g. page title, word count, headings etc.), URL details, directives or structured data.<\/p>\n<\/div> In addition to the above-mentioned elements, I recommend that you disable the function related to the position of links in Html.<\/p>\n<\/div><\/div><\/div><\/div><\/div><\/a><\/span><\/div>\n
\n
<\/a><\/span><\/div>