Learn · Technical SEO · Beginner

How to check pages built from a data feed against Google’s guidance

Take a sample of the pages a feed has produced, see what each one adds beyond the source, and decide whether to improve, merge or exclude it.

Published
Last fact checked
Time
45 minutes to do

What a page built from a data feed is

Some websites make one page for every record in a list: a product, a property, an event. The list often comes from a data feed, a regular supply of records from a database or from another organisation. This guide shows you how to check whether the pages built from a feed give a visitor something of their own.

It is not a test Google runs for you and it does not give a score. It is a way of putting Google's published questions to a small sample of your own pages, so you can decide what to do with the weak ones.

Why Google looks at what a feed page adds

Google's spam policy says scaled content abuse is when many pages are generated mainly to manipulate search rankings and not to help users. Among its examples is scraping feeds or other content to generate many pages where little value is provided to users. For content like that on your own site, it says to exclude it from Search. Read Spam Policies for Google Web Search. It gives no page count at which a set becomes a problem, so the check has to be about what each page gives a visitor.

What you need to check pages built from a data feed

  • A list of the page addresses that come from the feed, for example from your sitemap or an archive page
  • A web browser, and room to open two pages side by side
  • Someone who can edit the page template, if you decide to change anything

Check pages built from a data feed, step by step

  1. 1

    List the pages that come from the feed

    Gather the addresses of every page built from the feed and count them. Your sitemap, if you have one, is a quick source. Note which template they share.

    You will know it worked when You have a list of addresses and a number.

  2. 2

    Pick a sample across the set

    Choose five or six pages: a popular record, a rarely visited one, and some in between. The weak pages hide among the records nobody looks at.

    You will know it worked when Your sample includes at least two records you have never opened before.

  3. 3

    Mark what comes straight from the feed

    On each sampled page, write down which pieces are the feed's own fields, such as a name, a number or a date. Everything else is what your page adds.

    You will know it worked when Each page has two short lists: from the feed, and not from the feed.

  4. 4

    Put Google's questions to each page

    Google's page on creating helpful, reliable, people-first content asks whether the content provides original information or analysis, whether it avoids simply copying or rewriting other sources and adds substantial value, and whether it offers substantial value compared to other pages in search results. It also asks whether content is mass-produced so that individual pages do not get much attention or care. Answer yes, no or unsure for each sampled page.

    You will know it worked when Each sampled page has an answer and a one-line reason for each question.

  5. 5

    Compare two neighbouring pages

    Open two records side by side, including their titles and opening lines. If the only difference is the values in the fields, ask whether both need their own page.

    You will know it worked when You can say, in one sentence, what makes the two pages different besides their data.

  6. 6

    Check that the pages can be found by links

    Google says it can generally crawl a link only if it is an <a> element with an href attribute, and that it finds new pages through links. From your archive or list page, click through to each sampled page, then view the page source and look for that kind of link. See SEO Link Best Practices for Google.

    You will know it worked when Every sampled page can be reached by clicking ordinary links, and the source shows an anchor with an href for each.

  7. 7

    Decide what to do with each weak page

    There are three options: add material of its own, fold several records into one fuller page, or exclude the page from Search. To exclude it, add a noindex rule as a meta tag in the page head or as an HTTP response header. Google says the page must not be blocked by robots.txt, or its crawler never sees the rule. See Block Search Indexing with noindex.

    You will know it worked when Each weak page has a written decision. For any page you excluded, view its source to see the noindex tag, and open /robots.txt to confirm its folder is not blocked.

Common mistakes when you check pages built from a data feed

  • Assuming that crediting the source is enough. Google says giving credit to other sources does not replace the need for original effort.
  • Adding noindex to a page that robots.txt also blocks. Google then never sees the rule, and the page can still appear in results.
  • Judging the set by its best page and never opening the least popular records.
  • Rewording the feed's text with synonyms and counting it as new material. Google lists synonymising among the automated transformations in its scraping examples.

Terms used when you check pages built from a data feed

Data feed
A regular supply of records from a database or another organisation, which a site may turn into pages.
Scaled content abuse
In Google's spam policies, many pages generated mainly to manipulate rankings and not to help users, no matter how they are created.
noindex
A rule, in a meta tag or HTTP header, asking search engines to leave a page out of their results.

Sources for checking pages built from a data feed

Pages built from a feed and unsure which to keep?

Tell Dean which feed the pages come from and how many there are. You get a reply from Dean on what to check first, not a sales sequence.