Cookies Audit Seo

Find out how to analyse cookies and monitor reports with Screaming Frog.

Cookies Analysis

Given the importance and evolution of privacy laws, it has become increasingly important to understand the presence and purpose of cookies on our website so that we are compliant and do not incur penalties. With Screaming Frog you have the ability to identify cookies that are set automatically without explicit user consent on your entire project.

  • 1. Enable the ‘Cookies’ option for extraction with Screaming Frog.

The first action is to enable cookie extraction on your website through the “Config > Spider > Extraction” option and select “Cookies” under “URL Details.” In this way, the Seo Spider saves all cookies discovered during the scan.

When you choose to store cookies, the automatic exclusion performed by the SEO Spider for Google Analytics tracking tags is disabled to provide an accurate view of all cookies issued while still going to dirty GA data.

Rendering JS

When you choose to store cookies, the automatic exclusion performed by the SEO Spider for Google Analytics tracking tags is disabled to provide an accurate view of all cookies issued while still going to dirty GA data.

The default window size is automatically set on Googlebot Smartphone, but this can be adjusted on desktop if there are differences in the way the site issues cookies.

Set User-Agent

To best simulate browsing a user and not a Bot edit the user-agent on Chrome via “Config > User-agent”.

Robots.txt management

  • 4. Set the “Ignore robots.txt” option.

Click on “Configuration > robots.txt ” and select “Ignore robots.txt” or “Ignore robots.txt but report status.” This operation allows you to load all resources and respective cookies and prevent some resources from being blocked during scanning.

  • Crawl the website
  • Displays the number of cookies for each URL

After the scan you are able to see how many cookies were discovered for each URL scanned. This index can be found in the “Internal” tab in the “cookies” column.

Cookie Tab

  • 7. View Cookies Tab

Now all you have to do is see what cookies are discovered during the crawl; you just click on one or more URLs in the upper window to find all the details in the lower panel of the “Cookies” tab.

Below is an example of the “Cookies” tab populated with details.

The columns available to you are as follows:

  • Cookie Type: identifies where the cookie was discovered (“HTTP”, “On-Page” etc).
  • Cookie Name: collects the name of the cookie.
  • Cookie Value: defines the value of the cookie.
  • Domain: shows the domain that issued the first- or third-party cookie.
  • Expiration Time: indicates the expiration time of the cookie.
  • Secure: displays the details of the security attribute of the cookie.
  • HttpOnly: shows the details of the HttpOnly attribute of the cookie.
  • Address: displays the URL on which the cookie was found.
  • 8. Export aggregate cookie report

Cookie Report

You are now able to export a very valuable aggregate summary of discovered cookies.

Report > Cookie > Cookie Summary

The report displays an aggregated view of unique cookies discovered during a scan, considering their name, domain, expiration, security values, and HttpOnly. The number of URLs on which each unique cookie was issued is also displayed.

  • 9. Export all cookie details in bulk

For a more specific analysis you can click on “Bulk Export-Export > Web > All Cookies” to obtain for each individual URL the cookies and related attributes.

To be sure to find out all the cookies on a website, it is advisable to check them after accepting the “Consent Banner”.

Authentication: Form Based

To be sure to find out all the cookies on a website, it is advisable to check them after accepting the “Consent Banner”.

To get around this critical issue you can use Screaming Frog’s ‘Forms Based’ authentication and accept the pop-up about cookies by scanning as if you had accepted it.

“Config > Authentication > Forms Based,” then click on “Add,” enter the website URL, and click ‘OK’.

By following the steps listed above you are able to compare cookies discovered with and without acceptance.

Cookie Audit Video

Seo Spider Tab