Screaming Frog Version 20

Discover what's new in Screaming Frog's new version 20

Version 20 – Cracker

Version 20.2 called “cracker” corresponds to an outstanding “Major Update” of Seo Spider with new features related to artificial intelligence, improvement of internal links, N-gram analysis and much more that I will go on to define in the next paragraphs.

Custom Javascript Snippets

From this version you can run custom JavaScript while crawling. This means you can manipulate pages or extract data, as well as communicate with OpenAI’s ChatGPT API, local LLMs or other libraries.

To use the feature simply go to “Configure > Custom > Custom JavaScript” and click on “Add” to configure your custom JS snippet, or choose “Add from library” to select one of the preset and really very useful snippets prepared by the Screaming Frog staff.

To use this feature you will have to set the “JavaScript Rendering” mode from “Configuration > Spider > Rendering” before crawling, and you can see the results in the new custom JavaScript tab.

Crawl with ChatGPT

Among the options in the new custom JS you will be able to select the “Template (ChatGPT)” snippet, open it in the JavaScript editor, add your OpenAI API key, and define the prompt to query whatever you want on a page during crawling.

At the top of each template, you will appreciate a comment explaining how to adjust the snippet. You will also be able to debug in the JavaScript editor dialog box on the right before crawling.

This feature provides enormous versatility, and you can adjust the OpenAI model used, the specific content analyzed, and more. This will be able to help you perform rather detailed tasks, such as generating alternative text for images on the fly.

Once you have finished testing and debugging and configured your snippets, they will be saved in your user library and you can export/import the library in JSON format and share them very easily.

Mobile Usability

Another really useful new feature is large-scale mobile usability testing through Lighthouse integration (even locally).

In the top window of the SEO Spider you will find a new “Mobile” tab with respective filters that remain in line with the directions already in Google Search Console.

Filters available:

  • Viewport not set;
  • Target Size;
  • Content not size Correctly;
  • Contains Unsupported Plugins;
  • Mobile Alternate Links.

This opportunity requires connection via API to PSI (Config > API Access>PSI).

You can find the details of this analysis at the bottom of the Seo Spider in the “Lighthouse” tab.

To export the data you simply export the report > Mobile from the main menu.

N-gram Analysis

Big news of Seo Spider at “Content ” and Internal Linking level thanks to the new“N-gram” function.

From now on you can analyze the frequency of recurring phrases using n-gram analysis of pages in a crawl, or aggregated over a selection of pages on a website.

To enable this feature, “Store HTML / Store Rendered HTML” must be enabled in “Configuration > Spider > Extraction.” The “n-grams” can then be displayed in the new dedicated bottom tab.

IMPORTANT: This feature should not be seen as a pro-keyword crusade (deprecated) but as a tool to support in defining onpage semantics and ensure the Search Intent you want to define.
Through the study of N-grams, it will be possible to enhance strategic keywords and improve internal linking of the website.

For example, by analyzing the top competitors you could also find and define the N-grams of your website and through the Seo spider redefine the internal linking.

Anchor text Aggregates

Integration of new “Anchor Text” filters in the “Inlinks” and “Outlinks” tabs. These filters will allow you to aggregately display anchor text to a URL or Url selection for your analysis.

You will also be able to filter out self-referenced links and “nofollow” links for greater clarity of analysis (for both anchors and links).

It will then be possible to click on the anchor text to see exactly what pages it is on and discover classic link details.

Local Lighthouse Integration

The last very significant new feature is the ability to run Lighthouse locally during crawling to retrieve PageSpeed data.

You will simply select the source as “Local” via “Configuration > API Access > PSI.”

Of course, it will still be possible to connect through the PSI API to collect data externally, which may include CrUX field data. Through Lighthouse “Local” you will not be able to process such data but it will still be extremely useful when a site is under development and requires authentication for access, or you wish to check a large number of URLs.

Seo Spider Tab