Cultured Digital logo Cultured Digital Talk to an SEO specialist

Learn · Technical SEO · Beginner

How to check for technical errors after a traffic drop

Learn to use Search Console and your browser to check whether server errors, a broken robots.txt file, a stray noindex tag or soft 404s could be stopping Google crawling or indexing your pages.

By Dean Cruddace · 45 minutes to do · Updated · Last reviewed

What technical errors can cause a drop

Google describes technical issues as errors that can prevent it from crawling, indexing or serving your pages. Crawling is Google fetching a page; indexing is Google storing it so it can appear in results. Its examples are server availability, robots.txt fetching and "page not found" errors.

Faults can be site-wide, as when your website is down, or limited to some pages, as with a misplaced noindex tag (an instruction telling Google not to keep a page in results). This guide shows how to check with free tools.

Why technical checks follow a confirmed drop

Google says a page-level fault such as a misplaced noindex tag depends on Google crawling the page, so the drop in traffic would be slower than for a site-wide fault. Knowing which kind you have shapes where you look. See Google's guide to debugging traffic drops. Do this after you have checked that the drop is real.

What you need to check for technical errors after a traffic drop

  • Access to Google Search Console for the site
  • The date the drop began, and the pages that lost the most clicks
  • Someone who can edit the site or its server settings, if you find a fault

Check for technical errors after a traffic drop, step by step

  1. 1

    Find out how widespread the drop is

    In the Performance report, compare the drop period with an earlier one and sort the Pages table by Clicks Difference. Google says to find out whether the whole site, a group of pages or one page was hit, and points to the Page indexing report for site-wide drops and URL Inspection for a group of pages.

    You will know it worked when You have a short list of lost pages marked site-wide, group or single page.

  2. 2

    Look at your robots.txt file

    Type your site address plus /robots.txt into the browser. Per Google's robots.txt specification, if the file cannot be fetched because of a server error (5xx), Google stops crawling the site for the first 12 hours while it keeps trying, then uses the last good version for up to 30 days. A 4xx error other than 429 is treated as if there were no robots.txt file.

    You will know it worked when The address shows a readable text file or a plain not-found page, not a server error.

  3. 3

    Open the Crawl Stats report

    In Search Console open Settings and choose Crawl stats. Read Host status, then the robots.txt, DNS and server connectivity details, and Crawl responses for 5xx, DNS, timeout and redirect errors. Per Google's help page, the report is aimed at advanced users, needs a root-level property, and sites under a thousand pages should not need it.

    You will know it worked when You know whether host status is green and whether errors rose on or before your drop date.

  4. 4

    Check the Page indexing report

    Open Page indexing and compare the indexed and not indexed chart with your drop date. Google says a fall in indexed pages without a rise in errors suggests blocking by robots.txt, noindex or a login, and an error spike may come from a template change. Click rows in Why pages aren't indexed, such as Server error (5xx), Soft 404 or Blocked by robots.txt.

    You will know it worked when You have noted which reasons grew around your drop date.

  5. 5

    Inspect a sample of lost pages

    Paste a lost page's full address into the bar at the top of Search Console. URL Inspection shows whether the page is on Google, and under Page indexing how it was found and any crawl obstacles. Click Test live URL to check the current page; the indexed result is from the last crawl, not the live page.

    You will know it worked when For each sampled page you can read its status and crawl result.

  6. 6

    Check for a stray noindex tag

    On a lost page, choose View page source and search for noindex. A <meta name="robots" content="noindex"> tag in the head, or an X-Robots-Tag: noindex header, makes Google drop the page. Headers are not in the page source, but Google says URL Inspection's View crawled page shows the response and HTML Googlebot received. See Google's noindex documentation.

    You will know it worked when You know whether each sampled page carries noindex, and whether you meant it to.

  7. 7

    Fix, then ask Google to recheck

    Have the fault fixed. Google's status code guide says persistent 5xx errors eventually drop indexed URLs, and 4xx URLs are removed from the index. Then use Validate fix on the issue in Page indexing; Google says validation typically takes up to about two weeks. For one page you can request a recrawl in URL Inspection, though a noindex page may take months to be revisited.

    You will know it worked when The issue count falls, or URL Inspection shows the page indexed after a recrawl. Google does not guarantee indexing.

Common mistakes when you check for technical errors after a traffic drop

  • Blocking a page in robots.txt and adding noindex. Google's crawler cannot see the noindex rule on a blocked page.
  • Assuming clean server logs mean no crawl problem. Some Crawl Stats errors never reach your server.
  • Treating every not indexed page as a fault. Google says not indexed is not necessarily bad.
  • Using the live test to predict the canonical. Google says it appears only in indexed data.
  • Expecting a quick recovery after a fix.

Terms used when you check for technical errors after a traffic drop

Crawling
Google fetching a page from your server so it can read it.
Indexing
Google processing a crawled page and storing it, from where it may be shown in results.
noindex
A meta tag or HTTP header telling Google not to keep a page in its search results.
Soft 404
A page that is empty or shows a not-found message but returns a success status.
Status code
A number a server sends with each response, such as 200 for success or 500 for a server error.

Go deeper

Technical SEOTechnical causes of a traffic drop: crawling, indexing and serving errorsGoogle defines a technical cause of a traffic drop as an error that stops it crawling, indexing or serving your pages. This article goes through the…Read the in-depth article → Crawling and indexingHow crawling, robots.txt, sitemaps, canonicals and logs decide which pages Google can reach and index, explained from Google’s own documentation.See the service guide → Implementation and QAWhat makes an SEO ticket buildable and testable, what implementing directly means for search, and why releases are checked before and after launch.See the service guide → Next in the pathHow to check a traffic drop against a Google updateLearn to line up a fall in search traffic with Google's announced ranking updates, judge whether the drop is small or large,…Open the guide →
Errors in Crawl Stats or Page indexing after a drop?

Describe which reports show a spike and when it began. Crawling, indexing and implementation checks are part of technical SEO.

Talk to an SEO specialistTechnical SEO: implementation and QA →