Frequently asked questions.
Answers to the questions I get most often about Screaming Frog SEO Spider, the Club’s guides and training.
No question found: write to me directly.
The Club.
What Screaming Frog Club is, who runs it and how the guides are made.
What is Screaming Frog Club?
It’s a hub dedicated to Screaming Frog SEO Spider: step-by-step guides, video tutorials, an overview of every tab, the news in each release and an SEO Wiki on technical SEO. Every piece starts from a real problem met in a crawl and solves it inside the tool, with menu paths and annotated screenshots.
Is Screaming Frog Club the official Screaming Frog website?
No. The Club is an independent project, not affiliated with Screaming Frog Ltd. It’s run by Raffaele Visintin, technical SEO consultant and official SEO Spider beta tester. For buying licences, downloads and software support, the official site remains the reference.
Are the guides updated to the latest SEO Spider version?
Yes, the guides are reviewed at every major release: screenshots retaken on the current version, menu paths checked and new features added. Each guide shows its last update date at the top, and the news in each version is collected in Screaming Frog Updates.
Which languages are the guides and videos available in?
The guides are available in Italian, English, French and Spanish: switch language from the selector at the top right. The video tutorials are recorded in Italian, and the translated versions flag this next to each video.
Getting started with SEO Spider.
What the tool does, free or licensed version, and where to start.
What is Screaming Frog SEO Spider and what is it for?
It’s a desktop crawler that scans a website the way a search engine would and collects the technical SEO data for every URL: status codes, titles and meta descriptions, headings, canonicals, directives, internal links, images, structured data and much more. It helps you find errors, redirects, duplicate content and architecture issues before Google does.
What’s the difference between the free and the licensed version?
The free version crawls up to 500 URLs per crawl, with no configuration options and no way to save and reopen crawls: fine for small sites or for learning. A licence removes the URL limit and unlocks everything else: full configuration, JavaScript rendering, custom extraction, Google and AI API integrations, scheduled crawls and crawl comparison.
I’ve never used Screaming Frog: where do I start?
Start with First steps with SEO Spider: installation, your first crawl and reading the results. Then move on to the Overview to understand what each tab shows, and only then to the guides on advanced features. Every list can be filtered by level: Beginner, Intermediate or Advanced.
Licence and installation.
Licence key, price and renewal, VAT, which version to download and where to run it.
Why is my licence key invalid?
Nine times out of ten it’s the username. You need the licence username, which isn’t your email and isn’t always the same as your account username: it’s all lowercase. Copy and paste the username and key from your account area (don’t type them) into 1File›2Settings›3Licence (on Mac, in the Screaming Frog SEO Spider menu), and make sure you haven’t swapped a Log File Analyser licence for an SEO Spider one. If it still fails, contact Screaming Frog support.
How many users can use one licence? Can I install it on several computers?
A licence is for a single user: the same person can use it on more than one device, say an office PC and a laptop, but it can’t be shared between colleagues. The licensing system blocks keys used by several people, so a team needs one licence each. To work together on crawls, see Screaming Frog in a team.
How much does Screaming Frog SEO Spider cost and what happens when the licence expires?
As of October 2026 the price on the official site is £199 a year per user ($279 or €245). Auto-renewal is not on by default: you choose it at purchase. When the licence expires, SEO Spider returns to the free version: a 500 URL limit, no configuration, and saved crawls you can’t open until you renew.
I’m a business in the European Union: can I buy without paying VAT?
Yes, if you’re a business in an EU country (outside the UK) with a valid EU VAT number. Enter it in the “Tax ID” field at checkout: it’s checked against VIES and, if valid, VAT is removed and you account for it under the reverse charge. Double-check it before paying: Screaming Frog can’t refund VAT after the purchase.
Which version should I download for my Mac: Apple Silicon or Intel?
It depends on the processor: open 1Apple›2About This Mac. If the Chip line says Apple M1, M2, M3 or later, download the Apple Silicon version (aarch64), native and faster, introduced with version 17; if it says Intel, download the Intel version (x86_64). macOS 11 Big Sur or later is required.
Which systems does it run on? Is there a server, cloud or Chromebook version?
It runs on Windows, macOS and Linux (Ubuntu and Fedora packages), always 64-bit. There’s no SaaS version, but every installation is already a server version: you can run it on a dedicated server or on a cloud virtual machine, even without an interface from the command line. There’s no Chromebook version: on recent models you can install the Ubuntu build through ChromeOS Linux support, but limited RAM restricts crawls.
Hardware, memory and blocks.
Crawling millions of URLs, memory, external drives and sites that block the spider.
What hardware do I need, and how do I crawl millions of URLs without crashing the tool?
For large sites switch to Database Storage in 1File›2Settings›3Storage Mode and use an SSD: 500 GB is enough, 1 TB if you crawl often. For millions of URLs, 16 GB of RAM is recommended. Then limit the crawl to what you really need with excludes, includes and sampling: the full procedure is in Crawling large sites.
How do I increase allocated memory, and why do I get “JVM failing to allocate memory”?
Memory is set in 1File›2Settings›3Memory Allocation (on Mac in the Screaming Frog SEO Spider menu) and needs a restart: the default is 2 GB; in Database Storage about 4 GB is enough for up to 2 million URLs and about 10 GB for 5 million. The JVM error appears when you ask for more memory than the system has free: lower the allocation, close other apps, avoid several open instances or switch to Database Storage.
Can I use an external SSD for Database Storage if my internal disk is full?
Yes. Plug it into a USB 3.0 port (ideally with UASP support), set the database folder in 1File›2Settings›3Storage Mode, then restart. On Mac the drive must not be formatted as ExFAT or FAT. A slow drive slows the whole crawl, so avoid USB sticks and spinning hard disks.
Why does my crawl stop at the first URL?
Check the status code of the first URL in the Internal tab. The most common causes are: a server block (403, 429 or status code 0 for timeouts and refused connections), a robots.txt blocking the spider, a redirect to another subdomain treated as external, links generated only by JavaScript, or a meta nofollow. Causes and fixes, case by case, are in SEO Spider crawl issues and in the Status Code tab.
How do I crawl a site protected by Cloudflare without being blocked?
The block usually shows up as a 403 or as a JavaScript challenge the spider can’t pass. The clean fix is asking whoever manages the site to add your IP and the “Screaming Frog SEO Spider” user agent to an allowlist: in Cloudflare, with a custom rule on “IP Source Address” and the “Skip” action. Otherwise lower the crawl speed or try a browser user agent. The Shopify case, with the crawler’s HTTP signatures, is in Crawl issues.
How do I crawl a site that redirects based on location (Geo-IP)?
If the site sends you to your own country’s version, you need to look like a visitor from the country you’re auditing: set a proxy or VPN with an IP from that country in 1File›2Settings›3Proxy and, if the site also uses the browser language, send the right Accept-Language header from 1Configuration›2HTTP Header. Even better, ask the developers to exclude your IP or the spider’s user agent from the redirect during the audit.
Crawl configuration.
What to tell the spider before pressing Start: scope, excludes, resources, environments and automation.
How do I crawl only one subfolder of a site?
Start the crawl from the subfolder URL with a trailing slash, for example /blog/: SEO Spider stays inside that folder. For more precise rules use 1Configuration›2Include with a regex such as .*/blog/.*. Every step is in Crawling subfolders.
How do I exclude URLs with parameters using regex?
In 1Configuration›2Exclude enter one regex per line: .*\?.* excludes every URL with parameters, .*sort=.* only those with a specific parameter. If you want to crawl those pages but without the parameters, remove them with 1Configuration›2URL Rewriting: I explain it in Rewrite URL.
Can I crawl a staging site or a password-protected site?
Yes: SEO Spider supports both HTTP authentication and form-based logins. Read Crawling a password-protected site for the setup and Crawling staging sites for the checks to run before go-live.
How do I crawl only the HTML, without images, CSS and JavaScript?
In 1Configuration›2Spider›3Crawl untick Images, CSS, JavaScript and the other resource types: the crawl becomes much faster and lighter. Just be careful if you use JavaScript rendering: in that case CSS and JavaScript are needed to build the page.
Can I crawl a site built with JavaScript?
Yes. With JavaScript rendering on, SEO Spider uses Chromium to run the scripts and analyse the rendered DOM, as Google does. Comparing the original and rendered HTML shows straight away what depends on JavaScript: every step is in Analysing JavaScript sites.
How do I run a parity audit between staging and production, or mobile and desktop?
Crawl both versions with the same configuration (changing domain, or the mobile and desktop user agent) and compare them in Compare mode, using URL Mapping to match the URLs of the two environments. See SEO parity audit and Comparing data from 2 crawls.
How do I use Screaming Frog from the command line (CLI) to automate crawls?
SEO Spider can also start without an interface, from the terminal, with options such as --crawl, --headless, --save-crawl, --output-folder and --export-tabs: that’s how you run it on a server or inside a script. If scheduling is enough, 1File›2Scheduling does the same from the interface, with automatic exports to Google Drive and Looker Studio too: see Scheduled crawls and Automated crawl audits.
Everyday SEO audits.
Broken links, redirects, orphan pages, content, images, hreflang, comparisons and logs.
How do I find broken links (404) and export the pages they’re on?
In the Response Codes tab filter Client Error (4xx): you see the broken URLs and, in the Inlinks panel, the pages linking to them. To get everything in one file use 1Bulk Export›2Response Codes›3Client Error (4xx) Inlinks: for each broken link you get the source page, anchor text and position. See Broken links.
How do I check 301 redirects in bulk during a migration?
Switch to 1Mode›2List, paste or upload the old URLs and enable Always Follow Redirects in 1Configuration›2Spider›3Advanced. When the crawl ends export 1Reports›2Redirects›3All Redirects: for each URL you see the final destination, status code and chains. See List mode and SEO migration.
How do I find orphan pages by combining the XML sitemap, Analytics and Search Console?
Connect the XML sitemap in 1Configuration›2Spider›3Crawl and GA4 and Search Console in 1Configuration›2API Access, enabling the option to crawl discovered URLs. When the crawl ends run 1Crawl Analysis›2Start and open 1Reports›2Orphan Pages: these are pages in the sitemap or with traffic but no internal links. See Orphan pages.
How do I find duplicate content, thin content and hard-to-read text?
The Content tab has Exact Duplicates, Near Duplicates (enabled in 1Configuration›2Content›3Duplicates, they need Crawl Analysis), Low Content Pages for pages under 200 words, and readability filters based on the Flesch score. See Duplicate content analysis and the Content tab.
How do I find images over 100 KB or without alt text?
In the Images tab use the Over 100 KB, Missing Alt Text and Missing Alt Attribute filters. The size threshold can be changed in 1Configuration›2Spider›3Preferences. See the Images tab and Crawling alt text.
How do I audit hreflang and generate an XML sitemap per language?
Enable hreflang crawling in 1Configuration›2Spider›3Crawl: the Hreflang tab flags missing return links, inconsistent languages, non-canonical or non-200 URLs, and the reports in 1Reports›2Hreflang export them. For a single-language sitemap crawl its folder and generate it from Sitemaps › XML Sitemap, including hreflang. See the Hreflang tab and the Sitemap tab.
How do I compare two crawls of the same site?
In Compare mode you load two saved crawls and SEO Spider shows what changed: new, removed and modified URLs, filter by filter. It’s the fastest way to check a release or a migration: it’s all in Comparing data from 2 crawls.
Why do I need log files if I already have a crawl?
A crawl simulates what Google could see; server logs show what it actually visited, how often and with which response. Cross-referencing them reveals wasted crawl budget, orphan pages and fake bots. I cover it in Log files and SEO, with Screaming Frog’s Log File Analyser.
Extraction, APIs and AI.
Scraping, invalid head, Google APIs, AI prompts, embeddings and the MCP server.
How do I use custom search and custom extraction for web scraping, including People Also Ask boxes?
Custom search looks for a text or regex in the code; custom extraction pulls a precise value with XPath, CSS Path or regex. For People Also Ask questions you can extract them from a list of SERP URLs in List mode, but Google limits automated requests: work on few URLs at low speed. Practical examples in Web scraping and What is web scraping.
How do I debug invalid HTML elements inside the head?
An element that isn’t allowed in the head, such as a div or an iframe, makes Googlebot close the head early, so it may ignore the meta tags that follow (canonical, hreflang, robots). SEO Spider flags them in the Validation tab with the Invalid HTML Elements In <head> filter. The full method is in Debugging invalid HTML in the head.
How do I automate the URL Inspection API and fix GSC and GA4 connection errors?
Connect Search Console in 1Configuration›2API Access›3Google Search Console and enable URL Inspection: it checks indexing status, Google-selected canonical and mobile usability, up to 2,000 URLs a day per property. Connection errors almost always come from the wrong account or property: sign in with the Google account that has access to the property, pick the right one (domain or URL prefix) and, for GA4, the correct property and data stream. See URL Inspection API and the API Integrations tab.
How do I use the built-in AI prompts and Custom JavaScript snippets to simulate interactions?
Since version 21 you connect OpenAI, Gemini and Ollama (and later Anthropic) in 1Configuration›2API Access and run prompts on each page’s data: alt text, classifications, summaries. With 1Configuration›2Custom›3Custom JavaScript you run scripts in the browser during the crawl, to scroll, click or open elements, also from the library of ready-made snippets: see Custom JavaScript.
How do I use embeddings to find similar pages and outliers, and to map redirects?
Enable embeddings in 1Configuration›2Content›3Embeddings by connecting an AI provider: the Content tab gets the Semantically Similar and Low Relevance Content filters, and 1Visualisations›2Content Cluster Diagram shows the topic clusters. The same vectors help match old URLs to the most similar new ones in a migration. Full guide in Embeddings SEO with Screaming Frog.
Can I connect Screaming Frog to an AI agent like Claude via MCP?
Yes. Since version 24.0 SEO Spider includes an official MCP (Model Context Protocol) server: from Claude Desktop, LM Studio or another compatible client you can start crawls, load saved crawls, generate reports and exports and process the data in natural language. You need a licence; in STDIO mode Claude launches SEO Spider in the background, in Streamable HTTP mode you open it yourself and see everything in the interface. Setup, prompts and security in the Screaming Frog MCP Server guide.
Training and consulting.
Courses, audits, the Kill SEO ebook and how to contribute to the Club.
Do you run Screaming Frog training courses?
Yes, both for individual professionals and in-house for teams and agencies, online or in person. Programmes and dates are on raffaelevisintin.it.
Can I ask for a technical SEO audit of my site?
Of course. Tell me about your project from the About me page or on LinkedIn: technical audits, migrations, log analysis and strategies for AI answer engines are my everyday work.
How do I download the Kill SEO ebook?
Kill SEO, Vol. 1 is a Screaming Frog guide with a foreword by Dan Sharp, Screaming Frog’s founder. It’s free and written in Italian: download the PDF from the green banner at the top of every Club page.
Can I suggest a guide, propose a collaboration or report a mistake?
Yes, and I’d be glad you did. For content proposals and collaborations see the Partnership page; for a mistake in a guide, just send me a message with the page link.
Didn’t find the answer you were looking for?
Tell me what you need: a doubt about SEO Spider, a crawl that doesn’t add up or a project to work on together. I’ll reply personally.
Author.Raffaele Visintin.Search & Strategic Marketing Advisor