Custom Search.

Scraping source code

SEO Spider lets you check for any element in the source code of a website’s pages with the “Custom Search” feature. By default it searches the raw HTML, while switching to “JavaScript rendering” mode lets you search the rendered data too. You can set up to 100 custom searches, which return the URLs that contain or don’t contain the input you’re looking for.

Setting up Custom Search

The feature is in 1Configuration›2Custom›3Custom Search.

12

1ConfigurationThe configuration menu.2Custom › Custom SearchThe item for custom searches.

To use Custom Search, click Add, name the filter, choose “Contains” or “Does Not Contain”, pick text or regex search, enter the input to look for and choose where to search (HTML, page text, a specific element, XPath etc.).

12345

1Custom SearchThe configuration section.2Contains and TextContains or does not contain, text or regex search.3QueryThe text or regex to look for.4Where to searchHTML, Page Text, Content Area, XPath and the other areas.5AddAdds a new search, up to 100.

In the next example we set up a custom search with two goals:

1. finding an e-commerce store’s unavailable products by searching for “Out of Stock” on product pages (Contains).
2. finding the pages missing the Google Tag Manager script (Does Not Contain).

12

1Out of StockContains: out-of-stock product pages.2GTMDoes Not Contain: pages without the Tag Manager script.

To collect the results, click “OK” and run the crawl.

Reviewing and exporting the results

Once the crawl runs, the data is filled in in real time and split into columns in the 1Custom Search tab.

You can isolate one search or the other with the drop-down filter on the left of the top window.

123

1Custom SearchThe tab with the search results.2Search filterIsolates one search at a time.3Overview sidebarThe same searches with the number of URLs.

For convenience you can also check the same results in the sidebar under “Custom Search”.

12

1Out of Stock (Contains)No out-of-stock pages.2GTM (Does Not Contain)8 URLs without Tag Manager, 88.89%.

To complete the analysis, export the results with the tab’s “Export” button, or use 1Bulk Export›2Custom Search›3All Inlinks to get every inlink for your custom search.

12

1Bulk ExportThe bulk export menu.2Custom Search › All InlinksThe inlinks of every search, or one filter at a time.

Best practices for SEO audits

Now that you’ve seen how versatile Custom Search is, here are some best practices that are very useful in everyday SEO audits.

Misspellings

A first very interesting use in “Text” mode is finding misspellings, acronyms and so on, taking letter case into account (regex is case sensitive by default) by enabling the relevant option.

1234

1Custom SearchThe configuration section.2Text to findScreaming frog with a lowercase f.3Content AreaSearches only the main content.4Case SensitiveDistinguishes upper and lower case.

Searching for exact-match queries

Another use of Custom Search in regex mode is searching for specific queries (exact match). This lets you spot cannibalisation issues with strategic keywords used on several landing pages, which could undermine their search intent on Google.

\bquery\b

You can extend the search to several words in the same Custom Search by separating the regex with “|” (the pipe symbol). This lets you assess keyword density in the text and avoid keyword stuffing penalties.

\bquery1\b|\bquery2\b|\bquery3\b

In the next example we searched for the queries “natural”, “organic” and “free”.

123

1OccurrencesHow many times the words appear on each page.2RegexThe words separated by the pipe.3Page TextSearches only the page text.

The crawl shows 5 occurrences of the three words in total on the first page.

Custom Search options

Custom Search offers 7 options to limit the search area:

• HTML: the whole HTML code.
• Page Text: only the page text, excluding all HTML elements.
• Page Text No Anchors: the page text, excluding any HTML and any text inside HTML anchor tags (also known as A elements). Useful when searching for words that also appear in link text in menus.
• HTML Head: the search is limited to the HTML head.
• HTML Body: the search covers the text and HTML elements of the body tag.
• XPath: lets you specify an XPath to define where in the HTML to search. For example, to limit the search to the text in h3 headings, enter //h3.
• Content Area: the search runs within the content area, the body element by default.

For example, we could check whether the name Screaming Frog is always written correctly in blog posts or has misspellings. Specifically, we asked SEO Spider whether the tool’s company name is written correctly or with a lowercase letter at the start of the second word: “Screaming frog”.

1234

1FilterThe Screaming frog search (Contains).2OccurrencesHow many times it appears on the page.3Text to findScreaming frog with a lowercase f.4Page TextSearches the text, excluding HTML.

Note: the custom search settings screen has a single-line input field but, by clicking the two small arrows, you can expand it to enter several lines of code or whole sentences.

1234

1Search 1: GTMThe search result for each page.2HTML HeadSearches for the script only in the head.3Multi-line fieldThe full Google Tag Manager script.4Case SensitiveDistinguishes upper and lower case.

Video tutorial

Video tutorialCustom Search tutorial | Screaming FrogWatch the video3:43