Custom Javascript

Insight into the new Custom javascript function of Screaming Frog

Custom JS Configuration

Custom JavaScript is a new feature of Screaming Frog in its version 20 and allows you to execute JavaScript code on every URL parsed with the SEO Spider.
Through this feature it is possible to extract all kinds of useful information from a web page that may not already be available in Spider SEO, as well as communicate with APIs such as OpenAI’s ChatGPT, local LLMs or other libraries.

To configure the Seo Spider with this feature simply click on “Configure > Custom > JavaScript.” Then simply click “Add” to start configuring a new snippet, or “Add from library” to choose an existing snippet pre-configured by Screaming Frog staff.

The library includes sample snippets to perform various functions and are a veritable reservoir of inspiration for increasingly granular and comprehensive analyses:

  • Analysis of the sentiment, search intent, or language of the page content.
  • Alternative text generation for images.
  • Activation of mouseover events.
  • Scrolling of a page (to analyze some set-ups with infinite scroll).
  • Extraction of “embedded” content from page content.
  • Downloading and saving various content locally such as images.

The library is absolutely outstanding for starting to manipulate JS and, through to reviewing them customize them uniquely for your project.

This feature also allows you to set a content type filter that will allow the Custom JavaScript Snippet to perform the action only for certain types of content and not waste OpenAi’s API.

The results will then be available in the dedicated custom JavaScript tab.

You will need to buy credits to use the OpenAi API; it is not enough to use the Free version of Chat GPT.

Type of Snippet

Analyzing the snippets we notice that there are 2 types: Extractions and Actions.

Extraction Snippets

Snippets of type Extraction return a value or list of values (numbers or strings) by displaying the “Value” as columns in the“Custom JavaScript” tab.
Each value, in a list of values will be mapped across columns in the tab.
When processing these snippets, the page will stop loading all resources and start executing the.

The SEO Spider will not complete the analysis of the page until the snippet is completed. In some cases snippets with too long an execution may expire (Timeout) and the crawl of the page will not be completed.
This type of snippets allows you to download URLs and write to a text file. Snippets are available in the library to download all images from a web page add attributes of a web page to a CSV file.

Action Snippets

Action-type snippets do not return data; they only execute actions. For example, there is a snippet that simulates the scrolling of a page by allowing “lazy loading” images to be analyzed.
The major difference from “Extraction Snippets” lies in the fact that when they are executed, the page will continue to load resources.

However, you will need to specify a timeout value in seconds for this processing and when this expires the SEO Spider will complete the page analysis.

Behavior Snippets

  • The Seo Spider allows multiple snippets to be executed simultaneously. However, it is important to be aware that the scanning speed will be affected by the number and type of snippets you run. After debugging it these days I notice how too many simultaneous requests make Screaming Frog “crash.” For testing, I started about ten.
  • In the case of multiple snippets,the “Actions” are executed before the “Extractions”.
  • In the case of Action snippets with different timeout values, Screaming Frog will use the maximum timeout value configured among the different snippets.
  • Extraction Snippets stop all page loads, so no further requests will be made. “Action” does not have this limitation, but a timeout value must be set.
  • Snippets have access to Chrome’s Console Utilities API. This allows it to use methods such as getEventListeners(), which are not accessible by normal JavaScript processing on a web page.

Extraction Snippets API

Extraction Snippets interact with the Seo Spider using the seoSpider object, which is an instance of the SEOSpider class documented below. In the most basic form, it is used as follows:

Copy to Clipboard

You can also send data to the SEO Spider with a“Promise.” The SEO Spider waits for the completion of the same. This allows you to perform asynchronous work such as fetch requests before returning data to the SEO Spider. For example:

Copy to Clipboard

As with all the examples just mentioned, the Promise also requires calling the “return” statement to terminate the execution of the function. This is because all the code in the snippet is implicitly wrapped by the SEO Spider in an immediately invoked function expression (IIFE). This scenario is necessary to avoid conflicts in the “global namespace of JavaScript” during snippet execution. If you don’t, the SEO Spider cannot receive any data.

The example below shows how your fragment JavaScript code is implicitly included in an IIFE. It also shows how the seoSpider instance is created for you just before your code is entered.

Copy to Clipboard

Extraction Snippets API

This class provides methods for sending data to the SEO Spider. Remember: do not call “new” on this class, because an instance is already provided to you called seoSpider.

Copy to Clipboard

It returns the data provided to the SEO Spider to display in the custom JavaScript tab. The “date” parameter can be a string or number, or a list of strings or numbers. If “date” represents a list, then each item in the list will be shown in a separate column in the tab.

Copy to Clipboard
Copy to Clipboard

Return any error messages to the SEO Spider. These messages will appear in a column on the custom JS tab. The result will be a string.

Copy to Clipboard

The next method includes saveText(text, saveFilePath, shouldAppend)

Copy to Clipboard

Next method loadScript(src) → {Promise}. Type: String
Loads external scripts to be used by the snippet. The script is loaded asynchronously. Write your code inside the ‘then’ clause, as shown in the example below.

Copy to Clipboard

Snippet Sharing

The snippets will be saved in your user library, and then you can export/import the library as JSON to share it very easily.

Seo Spider Tab