Cookie SEO Audit.

Cookie analysis

With privacy laws growing in importance and constantly evolving, it’s increasingly important to understand which cookies your website sets and why, so you stay compliant and avoid fines. With Screaming Frog you can identify the cookies set automatically, without the user’s explicit consent, across the whole project.

Setting up SEO Spider

Cookie extraction

First, enable cookie extraction in 1Configuration›2Spider›3Extraction by selecting Cookies under URL Details. SEO Spider will then store every cookie found during the crawl.

123

1ExtractionThe Spider › Extraction section of the configuration.2URL DetailsThe data stored for each crawled URL.3CookiesStores the cookies each page sets.

When you choose to store cookies, SEO Spider’s automatic exclusion of Google Analytics tracking tags is disabled, to give an accurate view of every cookie set: the crawl will, however, pollute your GA data.

JavaScript rendering

Many cookies are set by JavaScript, and SEO Spider only sees them if it renders the pages. So set rendering to JavaScript in 1Configuration›2Spider›3Rendering.

123

1RenderingThe Spider › Rendering section of the configuration.2JavaScriptThe rendering engine, headless Chrome.3Window SizeGooglebot Smartphone by default, or desktop.

The default window size is Googlebot Smartphone, but you can switch to desktop if the site sets cookies differently on the two versions.

Setting the user agent

To best simulate a real user rather than a bot, set the user agent to Chrome in 1Configuration›2User-Agent.

12

1Preset User-AgentsThe menu with preset bots and browsers.2ChromeA desktop browser user agent.

Handling robots.txt

Click 1Configuration›2robots.txt and select Ignore robots.txt or Ignore robots.txt but report status. SEO Spider will then load every resource, and its cookies, without any being blocked during the crawl.

12

1robots.txt modeRespect, Ignore or Ignore but report status.2Ignore robots.txtAlso loads resources blocked by robots.txt.

Analysing cookies

Cookies per URL

Start the crawl. When it finishes, you’ll see how many cookies were found for each URL in the Internal tab, Cookies column.

12

1InternalThe tab with every internal URL crawled.2CookiesThe number of cookies set by each URL.

Cookies tab

Now you can see which cookies were found during the crawl: click one or more URLs in the top window and you’ll find all the details in the Cookies tab of the bottom pane.

123

1Selected URLThe page whose cookies you want to see.2Cookie detailsName, value, domain and attributes of each cookie.3CookiesThe bottom pane tab dedicated to cookies.

Here’s an example of the Cookies tab filled with details.

1234

1Cookie NameThe cookie’s name.2DomainThe domain setting the cookie, first or third party.3Secure and HttpOnlyThe cookie’s security attributes.4CookiesThe tab in the bottom pane.

The available columns are:

• Cookie Type: where the cookie was found (“HTTP”, “On-Page” etc.).
• Cookie Name: the cookie’s name.
• Cookie Value: the cookie’s value.
• Domain: the domain that set the cookie, first or third party.
• Expiration Time: when the cookie expires.
• Secure: details of the cookie’s Secure attribute.
• HttpOnly: details of the cookie’s HttpOnly attribute.
• Address: the URL where the cookie was found.

Exporting cookies

Cookie report

You can export a very valuable aggregated summary of the cookies found:

123

1ReportsScreaming Frog’s reports menu.2CookiesThe group of cookie reports.3Cookie SummaryThe aggregated summary of unique cookies.

The report gives an aggregated view of the unique cookies found during a crawl, based on name, domain, expiry, and Secure and HttpOnly values. For each unique cookie you also see the number of URLs that set it.

12

1Cookie NameEach unique cookie found in the crawl.2OccurrencesHow many URLs set the cookie.

Bulk export

For a more detailed analysis, use 1Bulk Export›2Web›3All Cookies to get the cookies and their attributes for every single URL.

12

1Bulk ExportThe bulk export menu.2All CookiesThe export of every cookie per URL.

Authentication: forms based

To be sure you find every cookie on a website, it’s worth checking them after accepting the consent banner too.

You can do this with Screaming Frog’s Forms Based authentication: accept the cookie pop-up in the built-in browser and SEO Spider crawls as if you’d accepted it. Go to 1Configuration›2Authentication›3Forms Based, click Add, enter the website URL and click OK.

12

1I AcceptAccepts the consent banner as a user would.2OKSaves the session with the accepted cookies.

By following these steps you can compare the cookies found with and without accepting the banner.

Cookie audit video

Video tutorialScreaming Frog tutorial: how to identify and analyse cookiesWatch the video5:02