Technical SEO · Analysis

Visitor-submitted questions and stories: what Google says about user-generated spam

Any form that publishes what visitors type is a place spam can arrive. Google lists what it counts as user-generated spam and the deterrents it suggests, from moderation to noindex for new contributors.

Published
Updated
Last fact checked
Read
6 min
New to this?How to keep spam out of pages that visitors can submit toLearn it step by step: what it is, why it matters and how to do it on your own site. Beginner45 minutesOpen the beginner guide →

Key findings

  • Google defines user-generated spam as spammy content added by users through a channel intended for user content, and says site owners are often unaware of it.
  • Its prevention guidance lists a clear abuse policy, reports from trusted users, noindex for posts from new users without reputation, nofollow or ugc on links in untrusted content, manual approval, blocklists and human verification.
  • noindex only works if the page is not blocked by robots.txt, because the crawler must be able to see the rule.

Anywhere a site publishes what visitors type, spam can arrive: a comment form, a question box, a story submission. Google treats this as the site owner’s problem to prevent, and it is specific about how. This article sets out what Google says about user-generated spam, and where its guidance gives options rather than requirements.

It came out of a build, ManFlu, which takes visitor questions and stories and holds off-topic questions and every submitted story for moderation before they appear. Where I give my own reading, I say so.

What Google counts as user-generated spam

Google’s Spam Policies for Google Web Search define user-generated spam as spammy content added to a site by users through a channel intended for user content. It adds that site owners are often unaware of it. The examples it gives are spammy accounts on hosting services anyone can register for, spammy posts on forum threads, comment spam on blogs and spammy files uploaded to file hosting platforms. A question box or story form is a channel of the same kind.

The page Prevent User-Generated Spam on Your Site says spammers take advantage of open comment forms and other inputs to put spammy content on an unsuspecting site. It also says preventing abuse is usually not hard, and that even simple deterrents, such as an unusual challenge a user has to complete before interacting, may discourage spammers. It gives no thresholds and no required set of measures. It lists what a site can do.

The deterrents Google lists

  • Say that spam is not allowed. Publish a clear abuse policy and communicate it, for example at sign-up, and let trusted users report content they consider spammy.
  • Look for spam patterns. Keep a record of sign-ups and interactions, and look at form completion time, the number of requests from the same IP address range, user agents and the values submitted. Google says these signals can feed a user reputation system.
  • Noindex content from new users. Many comment spammers want their content in search engines, so Google suggests a noindex robots meta tag on posts from new users with no reputation. Indexing can be allowed later, once they have earned it.
  • Qualify links. Add nofollow or ugc to links in untrusted content.
  • Approve manually. Moderation of suspicious interactions can decrease spam considerably. Google notes it adds overhead.
  • Use a blocklist. Once you find one spammy profile, make it simple to remove others, for example by banning an IP address that produces several.
  • Block automated submissions. Use reCAPTCHA or similar verification so that only human submissions get through.
  • Monitor. Watch for redirects, large numbers of ad sections, spammy keywords and large sections of encoded JavaScript. Use the site: operator or Google Alerts to detect problems, and watch server logs for sudden traffic spikes. Google also suggests confidence checks: if you mainly target one country, how likely is a sudden flood of interactions from another?

The figure below shows an invented submissions page twice: once open to anyone, once with these deterrents in place.

A page that takes visitor submissions, open and protected

Switch between an open submissions page and one with Google's deterrents in place. Hover, focus or tap a numbered marker to see what Google says about that detail.

Open to anyone

Share your story no rules shown, no way to report a post

Note 1

Form: name, story, link, Send

anything is accepted from anyone
Note 2

No record kept of when or from where posts arrive

Note 3

New post: "Cheap watches, click here" by a first-time visitor

live at once
Note 4

Posts from new visitors can be indexed

Note 5

Links in posts: plain links

Note 6

Held pages blocked in robots.txt

the noindex rule can never be seen
Note 7

Nobody checks the site: search or the server log

Note 8
  1. No policy, no reports. Google says to publish a clear abuse policy and tell users, and to let trusted users report content they consider spammy. Prevent User-Generated Spam on Your Site
  2. No challenge on the form. Google says even an unusual challenge before users interact may discourage spammers. It also suggests reCAPTCHA or similar verification to allow only human submissions. Prevent User-Generated Spam on Your Site
  3. No record of patterns. Google says to keep a record of sign-ups and interactions and look for spam patterns: form completion time, requests from the same IP address range, user agents and the values submitted. Prevent User-Generated Spam on Your Site
  4. Posts go live instantly. Google says manual approval for certain interactions can decrease spam considerably, by stopping spammers from instantly creating content. Prevent User-Generated Spam on Your Site
  5. New contributors indexable at once. Many comment spammers want their content in search engines. Google suggests noindex on posts from new users with no reputation. Prevent User-Generated Spam on Your Site
  6. Links not qualified. Spammers are often motivated by leaving a link to their site. Google suggests nofollow or ugc on all links in untrusted content. Prevent User-Generated Spam on Your Site
  7. Noindex behind a robots.txt block. For noindex to work, the page must not be blocked by robots.txt. If it is, the crawler never sees the rule and the page can still appear in search results. Block Search Indexing with noindex
  8. No monitoring. Google lists monitoring as one of its steps, including the site: search operator and watching server log files for sudden traffic spikes. Prevent User-Generated Spam on Your Site

Protected

Share your story Spam is not allowed. Report a post.

Note 1

Form: name, story, link, a challenge, Send

only a person gets through
Note 2

Kept: form completion time, IP range, values submitted

Note 3

New post: "Cheap watches, click here" by a first-time visitor

held until a person approves it
Note 4

New visitors' posts: noindex until they earn a reputation

Note 5

Links in posts: rel="ugc"

Note 6

Held pages open to crawlers, with noindex

the rule can be seen
Note 7

Regular checks: site: search, server log spikes

Note 8
  1. A policy and a way to report. A clear abuse policy, communicated to users, and a way for trusted users to report spammy content. Prevent User-Generated Spam on Your Site
  2. A challenge before sending. A challenge or similar verification, so that only human submissions get through. Prevent User-Generated Spam on Your Site
  3. Signals recorded. Google says these signals may help create a user reputation system, which can also help identify spammers. Prevent User-Generated Spam on Your Site
  4. Held for approval. Moderation adds overhead to daily workflows, Google says, but it is a very effective way of fighting spam. Prevent User-Generated Spam on Your Site
  5. Noindex until reputation is earned. Once the user gains reputation, their content can be allowed to be indexed. Google says this will greatly demotivate spammers. Prevent User-Generated Spam on Your Site
  6. Links marked ugc. Google recommends the ugc value for user-generated links. Links marked this way will generally not be followed. Qualify Outbound Links for SEO
  7. Noindex on a crawlable page. When Googlebot crawls the page and extracts the rule, Google drops the page entirely from Search results. Block Search Indexing with noindex
  8. Regular checks. The site: operator or Google Alerts can help detect problems, and server logs can show sudden traffic spikes. Prevent User-Generated Spam on Your Site
An invented page for illustration. Google lists these deterrents as options and gives no required set or thresholds. It says even simple deterrents may discourage spammers.

On links, Qualify Outbound Links for SEO recommends marking user-generated content links, such as those in comments and forum posts, with rel=”ugc”. It says you might remove the attribute from links posted by members who have consistently made high-quality contributions over time. It also says links marked with these attributes will generally not be followed, and that the linked pages may still be found by other means. My article on what you vouch for with outbound links covers the values in full.

How noindex works, and the trap in it

Google’s Block Search Indexing with noindex says the rule can be a meta tag or an HTTP response header. Once Googlebot crawls the page and extracts the rule, Google drops the page entirely from Search results. The trap is in the next part. For noindex to work, the page must not be blocked by robots.txt. A crawler that cannot fetch the page never sees the rule, and the page can still appear in results if other pages link to it. Specifying noindex in robots.txt is not supported.

So for content you are holding back from search, the documented route is noindex on a page the crawler can reach. My reading is that anything not yet reviewed should also be hidden from visitors, but Google does not say that. The anti-abuse page also mentions blocking crawling in robots.txt, for user-created pages meant for direct sharing rather than search. That is a different case from a page you want Google to see and drop.

Google’s Q&A markup guidelines (Schema for Q&A Pages (QAPage)) say question and answer content may not appear as a rich result if it contains obscene, profane, sexually explicit, graphically violent or hateful language, or promotes dangerous or illegal activities. I cover those in my piece on QAPage markup. My reading: a filter on what visitors submit protects the page for readers as well as for search.

A routine for a site that takes submissions

This is my reading of how the documented deterrents fit together. It is not a Google procedure. My beginner guide to keeping spam out of pages that visitors can submit to walks through it step by step.

  1. List every place visitors can submit something, and what happens to it afterwards.
  2. Decide what is published immediately, what is held and why, and who reviews it.
  3. Qualify every link in submitted content.
  4. Add a challenge to each form, and keep a record of when and from where submissions arrive.
  5. Each week, search Google with the site: operator for words that do not belong on your site, and look at the log for spikes.
  6. When you find one spam submission, look for others from the same source.

Mistakes that let user-generated spam through

  • Blocking held pages in robots.txt and expecting noindex to work.
  • Qualifying links in comments but not in submitted stories or questions.
  • Treating moderation as a one-off setup. Google lists monitoring as one of its steps, alongside the rest.
  • Assuming a small site will not be targeted. The documentation describes spammers using open forms against unsuspecting sites.

Where this fits at Cultured Digital: crawling, indexing and builds

My crawling and indexing guide covers the rules that decide what Google can read and keep, and the websites page covers building sites with search in the architecture. If spam has already reached a site, my article on security and spam problems as causes of a traffic drop covers how that can show up in the data.

Further reading on user-generated spam

A site that publishes what visitors type?

Share where visitors can submit content and what happens next. Dean replies with what Google documents as the sensible deterrents.