Introduction Javascript sites<\/h2><\/span><\/div><\/div><\/div>Until a few years ago, one of the most common problems in Seo optimization was getting pages with dynamically created content indexed using Javascript. The problem was related to Search Engine Bots that were unable to scan and index these resources and only considered the content present in the source code of static HTML.<\/p>\n
Contextually to this issue, there has been an evolution of frameworks such as AngularJS, React, Vue.JS, single-page applications (SPAs) and progressive web apps (PWAs) that has led Google to evolve significantly and achieve full maturity in understanding web pages as a modern browser.<\/p>\n
However, although today Google is generally able to scan and index most JavaScript content, server-side rendering, pre-rendering or dynamic rendering is recommended rather than relying on client-side JavaScript since it is still difficult for crawlers to process JavaScript successfully or immediately.<\/p>\n
For these reasons in analysis, it is critical to understand the differences between the DOM after JavaScript has come into play and built the Web page and the static HTML response.<\/p>\n
Javascript, the seo spider and some contraindications<\/p>\n
To remedy this need, Screaming Frog comes to your aid with its “Javascript Rendering” mode, which has integrated the “Chromium” library to be able to better emulate the behavior of the Google Spider.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/javascript-crawling-SEO-SPIDER.jpeg\")
<\/a><\/span><\/div>Insight<\/strong>: Google upgraded in 2019 the Web Rendering Service (WRS) that was previously based on Chrome 41 with the latest stable version of Chrome, and this enabled it to support over a thousand additional features.<\/p>\n1. Let us consider some aspects inherent to sites in JavaScript:<\/p>\n
\n- 1. Enable “Javascript Rendering” carefully<\/li>\n<\/ul>\n
Although understanding what the differences are between the DOM and static html after Javascript execution is very important I advise against enabling the “Javascript Rendering” mode indiscriminately for every website.<\/p>\n
Crawling with JavaScript is slower and more intensive for the server, since all resources (both JavaScript, CSS, images, etc.) must be retrieved for rendering each web page. This criticality is irrelevant for small sites, but for large portals or ecommerce it could become a hindrance to your scanning in terms of time and Ram.<\/p>\n
\n- 2. Consider the principles and limitations of Javascript scanning<\/li>\n<\/ul>\n
All resources on a page (JS, CSS, images) must be available to be scanned, rendered, and indexed.<\/p>\n
Google still requires clean, unique URLs for a page, and links must be in appropriate HTML anchor tags (you can offer a static link, as well as call a JavaScript function).<\/p>\n
The Google Spider does not interact like users, for example, it does not click on resources by loading additional events after rendering (a click, hover or scroll for example).<\/p>\n
The snapshot of the rendered page is taken when it is determined that network activity has stopped, or beyond a time threshold. There is a risk that if a page takes a long time to render it may be skipped and elements may not be seen and indexed.<\/p>\n
In general Google renders all pages, however it does not queue pages for rendering if they have a ‘noindex’ in the initial HTTP response or static HTML.<\/p>\n
Finally, Google rendering is separate from indexing. Google initially scans the static HTML of a website, and defers rendering until it has resources.<\/p>\n
When beginning the development of a website knowing these conditions is vital to the success or failure of the project in terms of organic ranking.<\/p>\n<\/div>
<\/div><\/div><\/span>Server-side and Client-side Javascript<\/h3><\/span><\/div><\/div><\/div>Google advises against relying on client-side JavaScript and recommends developing with “progressive enhancement,” building the structure and navigation of the site using only HTML and then enhancing the look and feel of the site with AJAX.<\/p>\n
For this reason, if you use a JavaScript framework, Google recommends setting up server-side rendering, pre-rendering, or hybrid rendering, which can improve performance for users and Search Engine crawlers.<\/p>\n
Serving server-side rendering (SSR) and pre-rendering the JavaScript execution of the pages provides an initial rendered, scan-ready HTML version.<\/p>\n
Hybrid rendering (sometimes referred to as “isomorphic”) identifies a version of rendering that executes server-side JavaScript for initial page loading and HTML and client-side for non-critical elements and subsequent pages.<\/p>\n
Many JavaScript frameworks such as React or Angular Universal allow server-side and hybrid rendering.<\/p>\n
Alternatively, you can opt to use dynamic rendering.<\/p>\n
This can be especially useful when changes cannot be made to the front-end codebase. Dynamic rendering switches from client-side rendering for users to pre-rendered content for Search Engines.<\/p>\n<\/div>
<\/div><\/div><\/span>Google crawling times<\/h3><\/span><\/div><\/div><\/div>Although opting for the three solutions listed above solves most crawling problems, it is a good idea to consider one more aspect.<\/p>\n
Google has a two-step indexing process, in which it initially scans and indexes static HTML, and then returns later when resources are available to render the page and to scan and index the content and links in the rendered HTML.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/google-render-queue-CRAWL.jpeg\" "\\"google-render-queue-CRAWL\\"")
<\/span><\/div>The time required between crawling and rendering could also be very long (up to 7 days) and this condition is a problem for sites that rely on content that by its nature has timeliness characteristics or simply expires (e.g. news site, events etc.); while the average time (stated at the 2019 Chrome Dev Summit) is 5 seconds.<\/p>\n<\/div>
<\/div><\/div><\/span>How to identify Javascript<\/h3><\/span><\/div><\/div><\/div>Identifying a site built using a JavaScript framework can be fairly straightforward, however, diagnosing sections, pages, or just smaller elements that are dynamically adapted using JavaScript can be much more challenging. When you are getting ready to define a Seo Audit, it is a good idea to immediately define whether there is the presence of Javascript and try to compend if the site may or may not have problems during different User Agent crawls.<\/p>\n
There are several ways to know if the site is built using a JavaScript framework, let’s look at some of them:<\/p>\n
\n- 1. Identifying Javascript with Screaming Frog<\/li>\n<\/ul>\n
By default, the Seo Spider crawls using the “old AJAX crawling scheme,” which means JavaScript is disabled, and only scans Raw HTML.<\/p>\n
If the site uses only client-side JavaScript then you will get as a result of your scan only the Homepage with a 200 response with a few “JavaScript and CSS” files.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/javascript-siti-web-crawling-without-javascript.jpeg\")
<\/a><\/span><\/div>As you can also see the “Outlinks” Tab does not have hyperlinks since they are not rendered in the raw html and cannot be seen by the Seo Spider.<\/p>\n
<\/div>\n<\/div>![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/outlinks-scheda-no-hyperlinks.jpeg\")
<\/a><\/span><\/div>While this is often a sign that the Web site is using a JavaScript framework, with client-side rendering-it doesn’t tell you if other JavaScript dependencies also exist on the site.<\/p>\n
For example, a website might only have JavaScript to load products on category pages, or update title elements.<\/p>\n
How then is it easy to find them?<\/p>\n
To identify JavaScript more efficiently, you need to switch to JavaScript rendering mode (“Config > Spider > Rendering”) and scan the site, or a sample of templates from around the site.<\/p>\n
The SEO Spider will scan both the original and rendered HTML to identify pages that have content or links available only client-side and flag other key dependencies.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/filtri-della-scheda-javascript-del-seo-spider.png\")
<\/a><\/span><\/div>To guide you through the common problems of sites with client-side JavaScript, you have a comprehensive list of filters that will guide you in defining your Seo Audit.<\/p>\n
The Seo Spider will alert you if there is content that is only in the HTML rendered after the JavaScript comes into play allowing you a complete view of the behavior of the website.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/javascript-contenuto.jpeg\")
<\/a><\/span><\/div>In this example, 100% of the content is only in the rendered HTML since the site relies entirely on JavaScript.<\/p>\n
In this case, the examination focused on the number of words found in the content of the pages, but you can also find, for example, page references with links found only in the rendered HTML and remedy them by following Google’s directions.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/javascript-links-sito-web.jpeg\")
<\/a><\/span><\/div>This Screaming Frog feature is also very useful for checking whether or not meta tags or other Seo elements are present in static html or are only served and\/or modified once the JavaScript has been executed.<\/p>\n
\n- 2. Disable Javascript in the browser<\/li>\n<\/ul>\n
An alternative way to understand the nature of Web sites is to disable Javascript directly in the browser and reload the page. If it will show up white, there is no doubt that the website was designed with Javascript.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/disabilitare-javascript-devtools.jpeg\")
<\/a><\/span><\/div>\n- 3. Search for JavaScript directly in the source code<\/li>\n<\/ul>\n
The third solution is to right-click on the Web site and view the source code by examining the Html content, whether there is text, there are links or signs of the libraries of the various JS frameworks.<\/p>\n
By following this process you are seeing code before it is processed by the browser that will match what the Seo Spider scans, when it is not in JavaScript rendering mode.<\/p>\n
If you try to search for some element but get no occurrences within the source code, then presumably they are dynamically generated in the DOM and will be viewable only in the rendered code.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/source-code-sito-web.jpeg\")
<\/a><\/span><\/div>In this case the <body tag> is completely empty, a clear signal that the site has JS.<\/p>\n
\n- 4. Do an Audit of the rendered code<\/li>\n<\/ul>\n
The first question to ask yourself should be: how different is the rendered code from the static HTML source? To find out just right-click and use “inspect element” in Chrome to see the rendered HTML. You can often see the name of the JS Framework in the rendered code, such as ‘React’ in the example below.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/rendered-html-source-sito-web.jpeg\")
<\/a><\/span><\/div>You will find that the content and hyperlinks are in the rendered code, but not in the original HTML source code. This is what the Seo Spider will see, when it is in JavaScript rendering mode.<\/p>\n
By clicking on the opening HTML element, then “copy > outerHTML” you can compare the rendered source code with the original.<\/p>\n<\/div>
<\/div><\/div><\/span>Javascript Crawl with Screaming Frog<\/h3><\/span><\/div><\/div><\/div>Having defined the presence of JS and understood the pitfalls you can begin your Seo Audit with Screaming Frog by configuring it in JavaScript rendering mode. This allows you to scan dynamic, JavaScript-rich websites and frameworks, such as Angular, React, and Vue.js.<\/p>\n
\n- 1. To scan a JavaScript website, open Seo Spider, click on ‘Configuration > Spider > Rendering’ and change ‘Rendering’ to ‘JavaScript’.<\/li>\n<\/ul>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/rendering-js-con-screaming-frog.png\")
<\/a><\/span><\/div>\n- 2. Configure User-Agent & Window Size<\/li>\n<\/ul>\n
Configure the user-agent<\/p>\n
Configuration > HTTP Header > User-Agent<\/p>\n
And the size of the window<\/p>\n
Configuration > Spider > Rendering<\/p>\n
By default, the viewport is set to “Google Mobile: Smartphone” following the “Mobile Index First” orientation of the Search Engine.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/rendering-simulato-di-google-smartphone.png\" "\\"rendering")
<\/span><\/div>\n- 3. Check resources and external links<\/li>\n<\/ul>\n
Make sure resources such as images, CSS, and JS are checked in the “Configuration > Spider.” If the resources are on a different subdomain, or on a separate “root domain”, then also enable “check external links”, otherwise they will not be scanned and thus rendered.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/configurazione-crawl-pwe-aiti-Js.png\")
<\/a><\/span><\/div>\n- 4. Run the scan<\/li>\n<\/ul>\n<\/div>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/crawl-javascript-sito-web.jpeg\")
<\/a><\/span><\/div>\n- 5. See the Javascript tab<\/li>\n<\/ul>\n
With the JavaScript tab you have 15 filters to help you consider JavaScript dependencies and isolate common problems.<\/p>\n<\/div>
<\/span><\/div>\n- Pages with Blocked<\/strong> Resources: identifies all pages with resources (such as images, JavaScript and CSS) that are blocked by robots.txt. This condition can be a problem because Search Engines may not be able to access critical resources in order to render pages accurately. I recommend that you update the robots.txt file to allow all critical resources to be scanned and used to render site content. Resources that are not critical (e.g., Google Maps embed) can be ignored.<\/li>\n
- Contains JavaScript Links <\/strong>: the filter displays pages that contain hyperlinks that are discovered in the rendered HTML only after JavaScript is executed. These hyperlinks are not in raw HTML. It is a good idea to consider including important server-side links in static HTML.<\/li>\n
- Contains JavaScript Content<\/strong>: shows all pages that contain body text only in the HTML rendered after JavaScript is executed. Google is able to render pages and see only client-side content, consider including important server-side content in the raw HTML.<\/li>\n
- Noindex Only in Original HTML<\/strong>: all pages that contain a noindex tag in static HTML but not in rendered HTML. When Googlebot encounters a noindex tag, it skips rendering and executing JavaScript. Since Googlebot skips executing JavaScript, using JavaScript to remove the ‘noindex’ in the rendered HTML will not work.<\/li>\n
- Nofollow Only in Original HTML<\/strong>: identifies pages that contain a nofollow attribute in static HTML, and not in rendered HTML. This means that any hyperlinks in the raw HTML before JavaScript is executed will not be followed. Remove ‘nofollow’ if links should be followed, scanned, and indexed.<\/li>\n
- Canonical<\/strong> Only in Rendered HTML<\/strong>: returns all pages that contain a canonical only tag in the rendered HTML after JavaScript execution. I recommend that you include a canonical link in the static HTML (or HTTP header) to ensure that Google can see it and avoid relying only on the canonical in the rendered HTML.<\/li>\n
- Canonical Mismatch<\/strong>:identifies pages that contain a different canonical link in static HTML than in rendered HTML after JavaScript execution. This condition can cause mixed signals and can lead to undesirable behavior by the Search Engine.<\/li>\n
- Page Title Only in Rendered HTML<\/strong>: returns pages that contain a page title only in the rendered HTML after JavaScript is executed. This means that a Search Engine must render the page to see them. Consider including the important server-side content in the raw HTML.<\/li>\n
- Page Title Updated by JavaScript<\/strong>: identifies pages that have page titles modified by JavaScript. This means that the page title in static HTML is different from the page title in rendered HTML. Consider including important server-side content in static HTML.<\/li>\n
- Meta Description<\/strong> Only in Rendered<\/strong> HTML: returns pages that contain a meta description only in the rendered HTML after JavaScript execution.<\/li>\n
- Meta Description Updated by JavaScript<\/strong>: all pages that have meta descriptions that are modified by JavaScript. This means that the meta description in static HTML is different from the meta description in rendered HTML.<\/li>\n
- H1 <\/strong> Only in Rendered HTML <\/strong>: pages that contain an h1 only in the rendered HTML after JavaScript execution. This means that a Search Engine must render the page to consider the heading. Consider including server-side headings in static HTML.<\/li>\n
- H1 Updated by JavaScript<\/strong>: pages that have h1s modified by JavaScript. This means that the h1 in the raw HTML is different from the h1 in the rendered HTML.<\/li>\n
- Uses Old AJAX Crawling Scheme URL<\/strong>s: identifies URLs that are still using the old AJAX crawling scheme (a URL containing a hash fragment #!) that was officially deprecated as of October 2015. Consider server-side rendering or pre-rendering where possible, and dynamic rendering as an alternative solution.<\/li>\n<\/ul>\n
\n- Uses Old AJAX Crawling Scheme Meta Fragment Tag<\/strong>: includes URLs that include a meta fragment tag indicating that the page is still using the old AJAX crawling scheme that was officially deprecated as of October 2015. The advice is to update URLs to follow the evolution of JavaScript on current Web sites. Consider server-side rendering or pre-rendering where possible, and dynamic rendering as a workaround solution. If the site still has the old meta fragment tag in error, then this should be removed.<\/li>\n<\/ul>\n
\n- 6. Monitor blocked resources<\/li>\n<\/ul>\n
Keep an eye on anything that appears under the “Pages with blocked resources” filter in the “Javascript” tab<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/pagine-blocked-resources.jpeg\")
<\/a><\/span><\/div>Blocked resources can also be viewed for each page within the ‘Rendered Page’ tab, adjacent to the rendered screen in the bottom panel of the window. In severe cases, if a JavaScript site completely blocks JS resources, then the site simply will not be scanned.<\/p>\n
<\/div>\n<\/div>![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/blocked-resources-small-anteprima.jpeg\")
<\/a><\/span><\/div>Any blocked resources you can also see in the “Response Code” tab.<\/p>\n<\/div><\/div><\/div><\/div><\/div>
Response Codes > Blocked Resource<\/span><\/p>\n<\/div><\/div><\/div><\/div><\/div>Blocked pages and individual resources can also be bulk exported with the report “Bulk Export > Response Code > Blocked Resource Inlinks to be unblocked and optimized.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/blocked-resources-bulk-export-report.jpeg\" "\\"blocked-resources-bulk-export-report\\"")
<\/span><\/div>\n- 7. View rendered pages<\/li>\n<\/ul>\n
With Screaming Frog, it is possible to display the crawled rendered page in the ‘Rendered Page’ tab that dynamically appears at the bottom of the user interface when crawling in JavaScript rendering mode.<\/p>\n<\/div>
![](\"http:\/\/screamingfrog.club\/wp-content\/uploads\/2024\/03\/rendered-page-scheda.jpeg\")
<\/a><\/span><\/div>In case the Rendering of pages does not load consider acting on the AJAX timeout time directly in the Seo Spider configuration.<\/p>\n
Viewing the rendered page is vital when analyzing what a modern search bot is able to see and is especially useful when reviewing in staging, where Google’s native “Fetch & Render” functionality cannot be used through Search Console.<\/p>\n
If you have adjusted the user-agent and viewport on Googlebot Smartphone, you can see exactly how each page is rendered on mobile, for example.<\/p>\n
\n- 8. Compare raw and rendered HTML.<\/li>\n<\/ul>\n
You may want to store and display HTML and rendered HTML within the Seo Spider when working with JavaScript.<\/p>\n
Configuration > Spider > Extraction > Store Html\/Render Html<\/p>\n
By checking the appropriate “Store HTML” and “Store HTML Rendered” options you can have an easy comparison between the two versions and be able to compare the differences to be sure that critical content or links are present in the DOM.<\/p>\n<\/div>
Until a few years ago, one of the most common problems in Seo optimization was getting pages with dynamically created content indexed using Javascript. The problem was related to Search Engine Bots that were unable to scan and index these resources and only considered the content present in the source code of static HTML.<\/p>\n
Contextually to this issue, there has been an evolution of frameworks such as AngularJS, React, Vue.JS, single-page applications (SPAs) and progressive web apps (PWAs) that has led Google to evolve significantly and achieve full maturity in understanding web pages as a modern browser.<\/p>\n
However, although today Google is generally able to scan and index most JavaScript content, server-side rendering, pre-rendering or dynamic rendering is recommended rather than relying on client-side JavaScript since it is still difficult for crawlers to process JavaScript successfully or immediately.<\/p>\n
For these reasons in analysis, it is critical to understand the differences between the DOM after JavaScript has come into play and built the Web page and the static HTML response.<\/p>\n
Javascript, the seo spider and some contraindications<\/p>\n
To remedy this need, Screaming Frog comes to your aid with its “Javascript Rendering” mode, which has integrated the “Chromium” library to be able to better emulate the behavior of the Google Spider.<\/p>\n<\/div>
<\/a><\/span><\/div>Insight<\/strong>: Google upgraded in 2019 the Web Rendering Service (WRS) that was previously based on Chrome 41 with the latest stable version of Chrome, and this enabled it to support over a thousand additional features.<\/p>\n 1. Let us consider some aspects inherent to sites in JavaScript:<\/p>\n Although understanding what the differences are between the DOM and static html after Javascript execution is very important I advise against enabling the “Javascript Rendering” mode indiscriminately for every website.<\/p>\n Crawling with JavaScript is slower and more intensive for the server, since all resources (both JavaScript, CSS, images, etc.) must be retrieved for rendering each web page. This criticality is irrelevant for small sites, but for large portals or ecommerce it could become a hindrance to your scanning in terms of time and Ram.<\/p>\n All resources on a page (JS, CSS, images) must be available to be scanned, rendered, and indexed.<\/p>\n Google still requires clean, unique URLs for a page, and links must be in appropriate HTML anchor tags (you can offer a static link, as well as call a JavaScript function).<\/p>\n The Google Spider does not interact like users, for example, it does not click on resources by loading additional events after rendering (a click, hover or scroll for example).<\/p>\n The snapshot of the rendered page is taken when it is determined that network activity has stopped, or beyond a time threshold. There is a risk that if a page takes a long time to render it may be skipped and elements may not be seen and indexed.<\/p>\n In general Google renders all pages, however it does not queue pages for rendering if they have a ‘noindex’ in the initial HTTP response or static HTML.<\/p>\n Finally, Google rendering is separate from indexing. Google initially scans the static HTML of a website, and defers rendering until it has resources.<\/p>\n When beginning the development of a website knowing these conditions is vital to the success or failure of the project in terms of organic ranking.<\/p>\n<\/div> Google advises against relying on client-side JavaScript and recommends developing with “progressive enhancement,” building the structure and navigation of the site using only HTML and then enhancing the look and feel of the site with AJAX.<\/p>\n For this reason, if you use a JavaScript framework, Google recommends setting up server-side rendering, pre-rendering, or hybrid rendering, which can improve performance for users and Search Engine crawlers.<\/p>\n Serving server-side rendering (SSR) and pre-rendering the JavaScript execution of the pages provides an initial rendered, scan-ready HTML version.<\/p>\n Hybrid rendering (sometimes referred to as “isomorphic”) identifies a version of rendering that executes server-side JavaScript for initial page loading and HTML and client-side for non-critical elements and subsequent pages.<\/p>\n Many JavaScript frameworks such as React or Angular Universal allow server-side and hybrid rendering.<\/p>\n Alternatively, you can opt to use dynamic rendering.<\/p>\n This can be especially useful when changes cannot be made to the front-end codebase. Dynamic rendering switches from client-side rendering for users to pre-rendered content for Search Engines.<\/p>\n<\/div> Although opting for the three solutions listed above solves most crawling problems, it is a good idea to consider one more aspect.<\/p>\n Google has a two-step indexing process, in which it initially scans and indexes static HTML, and then returns later when resources are available to render the page and to scan and index the content and links in the rendered HTML.<\/p>\n<\/div> The time required between crawling and rendering could also be very long (up to 7 days) and this condition is a problem for sites that rely on content that by its nature has timeliness characteristics or simply expires (e.g. news site, events etc.); while the average time (stated at the 2019 Chrome Dev Summit) is 5 seconds.<\/p>\n<\/div> Identifying a site built using a JavaScript framework can be fairly straightforward, however, diagnosing sections, pages, or just smaller elements that are dynamically adapted using JavaScript can be much more challenging. When you are getting ready to define a Seo Audit, it is a good idea to immediately define whether there is the presence of Javascript and try to compend if the site may or may not have problems during different User Agent crawls.<\/p>\n There are several ways to know if the site is built using a JavaScript framework, let’s look at some of them:<\/p>\n By default, the Seo Spider crawls using the “old AJAX crawling scheme,” which means JavaScript is disabled, and only scans Raw HTML.<\/p>\n If the site uses only client-side JavaScript then you will get as a result of your scan only the Homepage with a 200 response with a few “JavaScript and CSS” files.<\/p>\n<\/div> As you can also see the “Outlinks” Tab does not have hyperlinks since they are not rendered in the raw html and cannot be seen by the Seo Spider.<\/p>\n While this is often a sign that the Web site is using a JavaScript framework, with client-side rendering-it doesn’t tell you if other JavaScript dependencies also exist on the site.<\/p>\n For example, a website might only have JavaScript to load products on category pages, or update title elements.<\/p>\n How then is it easy to find them?<\/p>\n To identify JavaScript more efficiently, you need to switch to JavaScript rendering mode (“Config > Spider > Rendering”) and scan the site, or a sample of templates from around the site.<\/p>\n The SEO Spider will scan both the original and rendered HTML to identify pages that have content or links available only client-side and flag other key dependencies.<\/p>\n<\/div> To guide you through the common problems of sites with client-side JavaScript, you have a comprehensive list of filters that will guide you in defining your Seo Audit.<\/p>\n The Seo Spider will alert you if there is content that is only in the HTML rendered after the JavaScript comes into play allowing you a complete view of the behavior of the website.<\/p>\n<\/div> In this example, 100% of the content is only in the rendered HTML since the site relies entirely on JavaScript.<\/p>\n In this case, the examination focused on the number of words found in the content of the pages, but you can also find, for example, page references with links found only in the rendered HTML and remedy them by following Google’s directions.<\/p>\n<\/div> This Screaming Frog feature is also very useful for checking whether or not meta tags or other Seo elements are present in static html or are only served and\/or modified once the JavaScript has been executed.<\/p>\n An alternative way to understand the nature of Web sites is to disable Javascript directly in the browser and reload the page. If it will show up white, there is no doubt that the website was designed with Javascript.<\/p>\n<\/div> The third solution is to right-click on the Web site and view the source code by examining the Html content, whether there is text, there are links or signs of the libraries of the various JS frameworks.<\/p>\n By following this process you are seeing code before it is processed by the browser that will match what the Seo Spider scans, when it is not in JavaScript rendering mode.<\/p>\n If you try to search for some element but get no occurrences within the source code, then presumably they are dynamically generated in the DOM and will be viewable only in the rendered code.<\/p>\n<\/div> In this case the <body tag> is completely empty, a clear signal that the site has JS.<\/p>\n The first question to ask yourself should be: how different is the rendered code from the static HTML source? To find out just right-click and use “inspect element” in Chrome to see the rendered HTML. You can often see the name of the JS Framework in the rendered code, such as ‘React’ in the example below.<\/p>\n<\/div> You will find that the content and hyperlinks are in the rendered code, but not in the original HTML source code. This is what the Seo Spider will see, when it is in JavaScript rendering mode.<\/p>\n By clicking on the opening HTML element, then “copy > outerHTML” you can compare the rendered source code with the original.<\/p>\n<\/div> Having defined the presence of JS and understood the pitfalls you can begin your Seo Audit with Screaming Frog by configuring it in JavaScript rendering mode. This allows you to scan dynamic, JavaScript-rich websites and frameworks, such as Angular, React, and Vue.js.<\/p>\n Configure the user-agent<\/p>\n Configuration > HTTP Header > User-Agent<\/p>\n And the size of the window<\/p>\n Configuration > Spider > Rendering<\/p>\n By default, the viewport is set to “Google Mobile: Smartphone” following the “Mobile Index First” orientation of the Search Engine.<\/p>\n<\/div> Make sure resources such as images, CSS, and JS are checked in the “Configuration > Spider.” If the resources are on a different subdomain, or on a separate “root domain”, then also enable “check external links”, otherwise they will not be scanned and thus rendered.<\/p>\n<\/div> With the JavaScript tab you have 15 filters to help you consider JavaScript dependencies and isolate common problems.<\/p>\n<\/div> Keep an eye on anything that appears under the “Pages with blocked resources” filter in the “Javascript” tab<\/p>\n<\/div> Blocked resources can also be viewed for each page within the ‘Rendered Page’ tab, adjacent to the rendered screen in the bottom panel of the window. In severe cases, if a JavaScript site completely blocks JS resources, then the site simply will not be scanned.<\/p>\n Any blocked resources you can also see in the “Response Code” tab.<\/p>\n<\/div><\/div><\/div><\/div><\/div> Response Codes > Blocked Resource<\/span><\/p>\n<\/div><\/div><\/div><\/div><\/div> Blocked pages and individual resources can also be bulk exported with the report “Bulk Export > Response Code > Blocked Resource Inlinks to be unblocked and optimized.<\/p>\n<\/div> With Screaming Frog, it is possible to display the crawled rendered page in the ‘Rendered Page’ tab that dynamically appears at the bottom of the user interface when crawling in JavaScript rendering mode.<\/p>\n<\/div> In case the Rendering of pages does not load consider acting on the AJAX timeout time directly in the Seo Spider configuration.<\/p>\n Viewing the rendered page is vital when analyzing what a modern search bot is able to see and is especially useful when reviewing in staging, where Google’s native “Fetch & Render” functionality cannot be used through Search Console.<\/p>\n If you have adjusted the user-agent and viewport on Googlebot Smartphone, you can see exactly how each page is rendered on mobile, for example.<\/p>\n You may want to store and display HTML and rendered HTML within the Seo Spider when working with JavaScript.<\/p>\n Configuration > Spider > Extraction > Store Html\/Render Html<\/p>\n By checking the appropriate “Store HTML” and “Store HTML Rendered” options you can have an easy comparison between the two versions and be able to compare the differences to be sure that critical content or links are present in the DOM.<\/p>\n<\/div>\n
\n
Server-side and Client-side Javascript<\/h3><\/span>
Google crawling times<\/h3><\/span>
<\/span><\/div>How to identify Javascript<\/h3><\/span>
\n
<\/a><\/span><\/div><\/a><\/span><\/div><\/a><\/span><\/div><\/a><\/span><\/div><\/a><\/span><\/div>\n
<\/a><\/span><\/div>\n
<\/a><\/span><\/div>\n
<\/a><\/span><\/div>Javascript Crawl with Screaming Frog<\/h3><\/span>
\n
<\/a><\/span><\/div>\n
<\/span><\/div>\n
<\/a><\/span><\/div>\n
<\/a><\/span><\/div>\n
<\/span><\/div>\n
\n
\n
<\/a><\/span><\/div><\/a><\/span><\/div><\/span><\/div>\n
<\/a><\/span><\/div>\n