Search Console tells you what Google chooses to report. A server log tells you what actually arrived at your server, line by line, and that is why log analysis is a standard part of advanced technical SEO. It is also easy to misread, because a log records everyone, including requests that only claim to be Google.
This article sets out what Google documents about verifying its crawlers, how its Crawl Stats report relates to a log, and what status codes mean for crawling. Where we give our own reading rather than Google’s wording, we say so.
What a log records that Search Console does not
A server log has a line for each request your server handles: who asked, for what, when, and what the server answered. Google’s Crawl Stats report covers similar ground, with total requests, download size, response times, host status, response codes, file types, crawl purpose and Googlebot type. It has limits. It is only available for root-level properties, and Google says sites with fewer than a thousand pages should not need it. It also states a known issue: the report counts most crawl requests but not all, so you may see slight differences between your logs and its numbers.
It counts each hop of a redirect chain as a separate request, shows the actual URLs requested, not the canonical ones, and its example URLs are representative, not complete. Our reading: when a log and the report disagree slightly, that is expected. A large disagreement is worth investigating.
Who is really asking: verifying Googlebot
Google’s guide to verifying requests from Google crawlers exists because spammers and other troublemakers can claim to be Google. It describes three kinds of Google request. Common crawlers, such as Googlebot, always respect robots.txt rules for automatic crawls. Special-case crawlers serve specific products and may or may not respect robots.txt. User-triggered fetchers act when a person asks, and ignore robots.txt. The crawler overview gives AdsBot as an example of a special-case crawler that ignores the global user agent rule with the publisher’s permission.
To verify a request, Google describes a manual check. Run a reverse DNS lookup on the IP address from your log, confirm that the domain is googlebot.com, google.com or googleusercontent.com, run a forward lookup on that domain, and confirm it returns the original IP address. For large volumes, match addresses against the JSON lists of Google’s IP ranges. Google also says its crawlers are spread across many datacentres, so your logs may show several IP addresses.
Reading status codes in a log
Google’s crawl budget guide ties status codes to crawling. If a site slows down, responds with server errors or sends rate-limiting signals such as HTTP 429, the crawl capacity limit goes down and Google crawls less. A 404 or 410 for permanently removed pages is a strong signal not to crawl the URL again, though Google will not forget a URL it knows. Soft 404 pages keep being crawled and waste budget. Supporting 304 Not Modified responses lets Google reuse a cached page, and long redirect chains have a negative effect on crawling.
Google’s technical requirements add a status-code rule for indexing: only pages served with a 200 status are indexed. So a log full of 200s for the pages you care about is necessary, but it is not proof of indexing.
What “discovered” and “crawled” mean next to your log
The Page indexing report help defines two statuses that matter here. “Discovered – currently not indexed” means Google found the page but has not crawled it yet, typically because crawling it was expected to overload the site, so its last crawl date is empty. “Crawled – currently not indexed” means Google crawled the page but did not index it, and it may or may not be indexed in future. Our reading: a page with no Googlebot request in your log is probably in the first group, and one with requests but no indexing is in the second.
What a log cannot tell you
A log shows requests. It does not show whether Google rendered the page correctly, indexed it, or shows it in results. Google says that for Search not every page that is crawled will necessarily be indexed, since each page must still be evaluated, consolidated and assessed for the index. Use the URL Inspection tool for what Google holds about a page.
When log analysis is worth the effort
Google calls its crawl budget guide advanced, and aims it at large sites with a million or more unique pages that change weekly, medium or larger sites with ten thousand or more pages that change daily, and sites with many URLs classed as discovered but not indexed. It says that for others, keeping the sitemap up to date and checking the Page Indexing report regularly is adequate. Our article on when crawl budget matters covers the same boundary.
A first pass over a log
A routine we would use, as our reading rather than a Google procedure: take two to four weeks of logs; keep only requests that pass verification as Googlebot; group them by status code and by site section; check which of your important URLs were never requested; look at how many requests went to redirects and to URLs you did not mean to expose; look for runs of 5xx or 429 responses; then compare totals with Crawl Stats. Our beginner guide, how to check what Googlebot is requesting on your site, walks through it.
Where this fits at Cultured Digital: logs as evidence
On our technical SEO page, the crawl step asks whether a page is allowed and affordable to crawl, and lists robots.txt, status codes, crawl budget and logs. Log-file analysis sits in the crawling and indexing guide. Logs are one source of evidence for the hypothesis-and-test approach we describe there. Related reading: technical causes of a traffic drop.