Anywhere a site publishes what visitors type, spam can arrive: a comment form, a question box, a story submission. Google treats this as the site owner’s problem to prevent, and it is specific about how. This article sets out what Google says about user-generated spam, and where its guidance gives options rather than requirements.
It came out of a build, ManFlu, which takes visitor questions and stories and holds off-topic questions and every submitted story for moderation before they appear. Where I give my own reading, I say so.
What Google counts as user-generated spam
Google’s Spam Policies for Google Web Search define user-generated spam as spammy content added to a site by users through a channel intended for user content. It adds that site owners are often unaware of it. The examples it gives are spammy accounts on hosting services anyone can register for, spammy posts on forum threads, comment spam on blogs and spammy files uploaded to file hosting platforms. A question box or story form is a channel of the same kind.
The page Prevent User-Generated Spam on Your Site says spammers take advantage of open comment forms and other inputs to put spammy content on an unsuspecting site. It also says preventing abuse is usually not hard, and that even simple deterrents, such as an unusual challenge a user has to complete before interacting, may discourage spammers. It gives no thresholds and no required set of measures. It lists what a site can do.
The deterrents Google lists
- Say that spam is not allowed. Publish a clear abuse policy and communicate it, for example at sign-up, and let trusted users report content they consider spammy.
- Look for spam patterns. Keep a record of sign-ups and interactions, and look at form completion time, the number of requests from the same IP address range, user agents and the values submitted. Google says these signals can feed a user reputation system.
- Noindex content from new users. Many comment spammers want their content in search engines, so Google suggests a noindex robots meta tag on posts from new users with no reputation. Indexing can be allowed later, once they have earned it.
- Qualify links. Add nofollow or ugc to links in untrusted content.
- Approve manually. Moderation of suspicious interactions can decrease spam considerably. Google notes it adds overhead.
- Use a blocklist. Once you find one spammy profile, make it simple to remove others, for example by banning an IP address that produces several.
- Block automated submissions. Use reCAPTCHA or similar verification so that only human submissions get through.
- Monitor. Watch for redirects, large numbers of ad sections, spammy keywords and large sections of encoded JavaScript. Use the site: operator or Google Alerts to detect problems, and watch server logs for sudden traffic spikes. Google also suggests confidence checks: if you mainly target one country, how likely is a sudden flood of interactions from another?
The figure below shows an invented submissions page twice: once open to anyone, once with these deterrents in place.
A page that takes visitor submissions, open and protected
Switch between an open submissions page and one with Google's deterrents in place. Hover, focus or tap a numbered marker to see what Google says about that detail.
Open to anyone
Share your story no rules shown, no way to report a post
Note 1Form: name, story, link, Send
anything is accepted from anyoneNo record kept of when or from where posts arrive
New post: "Cheap watches, click here" by a first-time visitor
live at onceHeld pages blocked in robots.txt
the noindex rule can never be seenNobody checks the site: search or the server log
- No policy, no reports. Google says to publish a clear abuse policy and tell users, and to let trusted users report content they consider spammy. Prevent User-Generated Spam on Your Site
- No challenge on the form. Google says even an unusual challenge before users interact may discourage spammers. It also suggests reCAPTCHA or similar verification to allow only human submissions. Prevent User-Generated Spam on Your Site
- No record of patterns. Google says to keep a record of sign-ups and interactions and look for spam patterns: form completion time, requests from the same IP address range, user agents and the values submitted. Prevent User-Generated Spam on Your Site
- Posts go live instantly. Google says manual approval for certain interactions can decrease spam considerably, by stopping spammers from instantly creating content. Prevent User-Generated Spam on Your Site
- New contributors indexable at once. Many comment spammers want their content in search engines. Google suggests noindex on posts from new users with no reputation. Prevent User-Generated Spam on Your Site
- Links not qualified. Spammers are often motivated by leaving a link to their site. Google suggests nofollow or ugc on all links in untrusted content. Prevent User-Generated Spam on Your Site
- Noindex behind a robots.txt block. For noindex to work, the page must not be blocked by robots.txt. If it is, the crawler never sees the rule and the page can still appear in search results. Block Search Indexing with noindex
- No monitoring. Google lists monitoring as one of its steps, including the site: search operator and watching server log files for sudden traffic spikes. Prevent User-Generated Spam on Your Site
Protected
Share your story Spam is not allowed. Report a post.
Note 1Form: name, story, link, a challenge, Send
only a person gets throughKept: form completion time, IP range, values submitted
New post: "Cheap watches, click here" by a first-time visitor
held until a person approves itHeld pages open to crawlers, with noindex
the rule can be seenRegular checks: site: search, server log spikes
- A policy and a way to report. A clear abuse policy, communicated to users, and a way for trusted users to report spammy content. Prevent User-Generated Spam on Your Site
- A challenge before sending. A challenge or similar verification, so that only human submissions get through. Prevent User-Generated Spam on Your Site
- Signals recorded. Google says these signals may help create a user reputation system, which can also help identify spammers. Prevent User-Generated Spam on Your Site
- Held for approval. Moderation adds overhead to daily workflows, Google says, but it is a very effective way of fighting spam. Prevent User-Generated Spam on Your Site
- Noindex until reputation is earned. Once the user gains reputation, their content can be allowed to be indexed. Google says this will greatly demotivate spammers. Prevent User-Generated Spam on Your Site
- Links marked ugc. Google recommends the ugc value for user-generated links. Links marked this way will generally not be followed. Qualify Outbound Links for SEO
- Noindex on a crawlable page. When Googlebot crawls the page and extracts the rule, Google drops the page entirely from Search results. Block Search Indexing with noindex
- Regular checks. The site: operator or Google Alerts can help detect problems, and server logs can show sudden traffic spikes. Prevent User-Generated Spam on Your Site
On links, Qualify Outbound Links for SEO recommends marking user-generated content links, such as those in comments and forum posts, with rel=”ugc”. It says you might remove the attribute from links posted by members who have consistently made high-quality contributions over time. It also says links marked with these attributes will generally not be followed, and that the linked pages may still be found by other means. My article on what you vouch for with outbound links covers the values in full.
How noindex works, and the trap in it
Google’s Block Search Indexing with noindex says the rule can be a meta tag or an HTTP response header. Once Googlebot crawls the page and extracts the rule, Google drops the page entirely from Search results. The trap is in the next part. For noindex to work, the page must not be blocked by robots.txt. A crawler that cannot fetch the page never sees the rule, and the page can still appear in results if other pages link to it. Specifying noindex in robots.txt is not supported.
So for content you are holding back from search, the documented route is noindex on a page the crawler can reach. My reading is that anything not yet reviewed should also be hidden from visitors, but Google does not say that. The anti-abuse page also mentions blocking crawling in robots.txt, for user-created pages meant for direct sharing rather than search. That is a different case from a page you want Google to see and drop.
Google’s Q&A markup guidelines (Schema for Q&A Pages (QAPage)) say question and answer content may not appear as a rich result if it contains obscene, profane, sexually explicit, graphically violent or hateful language, or promotes dangerous or illegal activities. I cover those in my piece on QAPage markup. My reading: a filter on what visitors submit protects the page for readers as well as for search.
A routine for a site that takes submissions
This is my reading of how the documented deterrents fit together. It is not a Google procedure. My beginner guide to keeping spam out of pages that visitors can submit to walks through it step by step.
- List every place visitors can submit something, and what happens to it afterwards.
- Decide what is published immediately, what is held and why, and who reviews it.
- Qualify every link in submitted content.
- Add a challenge to each form, and keep a record of when and from where submissions arrive.
- Each week, search Google with the site: operator for words that do not belong on your site, and look at the log for spikes.
- When you find one spam submission, look for others from the same source.
Mistakes that let user-generated spam through
- Blocking held pages in robots.txt and expecting noindex to work.
- Qualifying links in comments but not in submitted stories or questions.
- Treating moderation as a one-off setup. Google lists monitoring as one of its steps, alongside the rest.
- Assuming a small site will not be targeted. The documentation describes spammers using open forms against unsuspecting sites.
Where this fits at Cultured Digital: crawling, indexing and builds
My crawling and indexing guide covers the rules that decide what Google can read and keep, and the websites page covers building sites with search in the architecture. If spam has already reached a site, my article on security and spam problems as causes of a traffic drop covers how that can show up in the data.
