Duplicate Content<\/h2><\/span><\/div><\/div><\/div>As repeatedly stated by Google, content is one of the most important ranking elements, and managing any duplication is one of the activities to consider in any Seo optimization.<\/p>\n
Duplicate content is one of the problems that most commonly clips the wings of a project and must be minimized so as not to waste the Crawling Budget.<\/p>\n
The goal of a Seo specialist should not be limited to understanding and optimizing identical pages but should also hone in on “similar” content that could create conditions of “Query Cannibalization” creating absolute inefficiency to Search Engine crawling.<\/p>\n
Through this tutorial, let’s see how Screaming Frog allows you to ferret out both exact duplicate content (with the same hash) and similar content that might turn out to have the same “Search Intent” in Google’s eyes.<\/p>\n
\n- 1. Enable ‘Near Duplicates’ via ‘Config > Content > Duplicates’.<\/li>\n<\/ul>\n
Screaming Frog, by default automatically identifies exact duplicate pages (duplicate pages have the same “Hash”), while to identify similar content you must enable the “Enable Near Duplicates” option. Once activated, the Seo Spider considers any document that has a minimum 90% match as a “similar” page.<\/p>\n
You can also adjust this threshold according to the refinement that best suits your project.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/funzionalita-seo-spider-per-analisi-contenuti-duplicati-.png\")
<\/a><\/span><\/div>By default the SEO Spider only checks similar or duplicate pages if they are ‘indexable’ and not ‘canonicalized’; the advice is to uncheck the ‘Only Check Indexable Pages For Duplicates’ option, as this may help you find areas of potential Crawler Budget waste.<\/p>\n
Nota Bene: I normally recommend that for new projects, where you go to manage all Key Research, you set the threshold at 50-60%, while on projects with some history at a minimum of 70% to make sure that the Search Intent is always unique and well marked.<\/p>\n
Through the use of Search Console you can periodically check the Query-Landing Page relationship and in the case of cannibalization choose which landing page is most appropriate (also checking GA data) to that specific Search Intent and renew the content of secondary pages or, in some cases remove them.<\/p>\n
\n- 2. Define the content area to be analyzed<\/li>\n<\/ul>\n
Even for this task Screaming Frog is very flexible and allows you to choose the focus to put your attention on when searching for duplicate or similar content.<\/p>\n
By default, the Seo Spider automatically excludes items from the <nav> and the
\n<footer> to focus on the main body content. However, not all websites are built using these HTML5 elements but Seo Spider allows you to ‘include’ or ‘exclude’ certain HTML tags, classes or IDs from your examination.<\/p>\n
For example, if you crawl the Screaming Frog website you will find that it has a mobile menu outside of the “nav” element but, by default, this element is still included in the content analysis. In this specific case you could define the menu class (‘mobile-menu__dropdown’) and exclude it from consideration to focus more on the main content.<\/p>\n<\/div>
As repeatedly stated by Google, content is one of the most important ranking elements, and managing any duplication is one of the activities to consider in any Seo optimization.<\/p>\n
Duplicate content is one of the problems that most commonly clips the wings of a project and must be minimized so as not to waste the Crawling Budget.<\/p>\n
The goal of a Seo specialist should not be limited to understanding and optimizing identical pages but should also hone in on “similar” content that could create conditions of “Query Cannibalization” creating absolute inefficiency to Search Engine crawling.<\/p>\n
Through this tutorial, let’s see how Screaming Frog allows you to ferret out both exact duplicate content (with the same hash) and similar content that might turn out to have the same “Search Intent” in Google’s eyes.<\/p>\n
- \n
- 1. Enable ‘Near Duplicates’ via ‘Config > Content > Duplicates’.<\/li>\n<\/ul>\n
Screaming Frog, by default automatically identifies exact duplicate pages (duplicate pages have the same “Hash”), while to identify similar content you must enable the “Enable Near Duplicates” option. Once activated, the Seo Spider considers any document that has a minimum 90% match as a “similar” page.<\/p>\n
You can also adjust this threshold according to the refinement that best suits your project.<\/p>\n<\/div>
<\/a><\/span><\/div>By default the SEO Spider only checks similar or duplicate pages if they are ‘indexable’ and not ‘canonicalized’; the advice is to uncheck the ‘Only Check Indexable Pages For Duplicates’ option, as this may help you find areas of potential Crawler Budget waste.<\/p>\n
Nota Bene: I normally recommend that for new projects, where you go to manage all Key Research, you set the threshold at 50-60%, while on projects with some history at a minimum of 70% to make sure that the Search Intent is always unique and well marked.<\/p>\n
Through the use of Search Console you can periodically check the Query-Landing Page relationship and in the case of cannibalization choose which landing page is most appropriate (also checking GA data) to that specific Search Intent and renew the content of secondary pages or, in some cases remove them.<\/p>\n
- \n
- 2. Define the content area to be analyzed<\/li>\n<\/ul>\n
Even for this task Screaming Frog is very flexible and allows you to choose the focus to put your attention on when searching for duplicate or similar content.<\/p>\n
By default, the Seo Spider automatically excludes items from the <nav> and the
\n<footer> to focus on the main body content. However, not all websites are built using these HTML5 elements but Seo Spider allows you to ‘include’ or ‘exclude’ certain HTML tags, classes or IDs from your examination.<\/p>\nFor example, if you crawl the Screaming Frog website you will find that it has a mobile menu outside of the “nav” element but, by default, this element is still included in the content analysis. In this specific case you could define the menu class (‘mobile-menu__dropdown’) and exclude it from consideration to focus more on the main content.<\/p>\n<\/div>
- 2. Define the content area to be analyzed<\/li>\n<\/ul>\n