Learn · Technical SEO · Beginner

How to find which stage of Google Search a page is stuck at

Learn to check one page against Google’s three stages, crawling, indexing and showing in results, so you know what kind of problem you have.

Published
Last reviewed
Time
30 minutes to do

The three stages of Google Search, explained

Google Search works in three stages: crawling, where Google fetches a page, indexing, where it analyses and stores the page, and serving, where it shows pages in results. Not every page makes it through all three.

When a page is not appearing as you expect, the first job is to find which stage it stopped at. This guide shows you how, using a browser and Search Console.

Why finding the stage comes before any fix

Google says a page is eligible to be indexed if Googlebot is not blocked, the page returns an HTTP 200 status and it has indexable content, and that indexing is not guaranteed even then. It also says a page can show as indexed in Search Console without appearing in results, because the content is irrelevant to the queries, its quality is low, or robots meta rules prevent serving. Each stage needs a different fix. See the technical requirements and how Search works.

What you need to find a page’s stage

  • Access to Google Search Console with the site added as a property
  • One page that is not doing what you expect, with its full address
  • A web browser, including a private or incognito window
  • About half an hour

Find which stage a page is stuck at, step by step

  1. 1

    Choose one page and say what you expected

    Pick a single page and write down its full address, including https://. Note what you expected, for example that it would appear in results for its main topic.

    You will know it worked when You have one full address and one sentence describing what you expected.

  2. 2

    Check anyone can open it

    Open the page in a private window without signing in. Google says that if a page is private, such as one that needs a login, Googlebot will not crawl it.

    You will know it worked when The page opens for a visitor who is not signed in.

  3. 3

    Check robots.txt does not block it

    Type your site address followed by /robots.txt into the browser. Look for a Disallow line that matches the start of your page’s path. Google says pages blocked by robots.txt are unlikely to show in results.

    You will know it worked when You found no matching Disallow line, or you noted the line that matches.

  4. 4

    Inspect the page in Search Console

    Paste the full address into the inspection bar at the top of Search Console. This shows what Google knows about the page. See the URL Inspection help.

    You will know it worked when You can see the inspection result for the page.

  5. 5

    Read the crawling and indexing results

    Look for whether Google could fetch the page and what status it received. Google only indexes pages served with a 200 status. Then look for whether the page is indexed, whether a noindex rule is present, and which address Google selected as the canonical. Google says the canonical is found only in the indexed data.

    You will know it worked when You know whether Google fetched the page with a 200 status, and whether it is indexed.

  6. 6

    Check whether it is being shown

    If the page is indexed, find out whether it appears in results at all. The Performance report, filtered to the page, shows its impressions. Our guide to reading the Performance report explains how.

    You will know it worked when You know whether the page has any impressions in the period you looked at.

  7. 7

    Name the stage and choose the next step

    If the page could not be fetched or is blocked, the problem is at crawling. If it was fetched but is not indexed, it is at indexing. If it is indexed but has no impressions, Google lists irrelevant or low-quality content and robots meta rules as reasons. Our article on what technical SEO is sets out which problems are technical.

    You will know it worked when You can say in one line which stage the page stopped at.

Common mistakes when you diagnose a stuck page

  • Using robots.txt to keep a page out of search. Google says robots.txt blocks crawling, and a URL can still appear; use noindex and allow crawling.
  • Assuming a page that meets the requirements will be indexed. Google says indexing is not guaranteed.
  • Expecting the live test in URL Inspection to show which canonical Google will choose. Google says that is found only in the indexed data.
  • Fixing the wrong stage, for example rewriting content on a page Google cannot fetch.

Terms used when you diagnose a stuck page

Crawling
Google fetching a page from your server so it can read it.
Indexing
Google processing a crawled page and storing it, from where it may be shown in results.
Serving
Google showing pages in search results when someone searches.
URL Inspection
A Search Console tool that shows Google's indexed version of a page and can test the live one.

Sources for diagnosing a stuck page

Page stuck and the stage still unclear?

Tell Dean which page, what you expected and what Search Console shows for it. A reply from Dean is how it starts, not a sales sequence.