Learn · Technical SEO · Beginner
Learn to read a server log for Googlebot, confirm which requests really came from Google, and spot errors and missing pages.
Every time anything asks your website for a page, your server can record the request in a server log: the address asked for, the time, who asked and what the server answered. Some of those requests come from Googlebot, Google’s crawler.
This guide shows you how to find Googlebot’s requests in a log, confirm they really came from Google, and read what they say about how your site is being crawled.
Google says its crawlers can be impersonated, so the name in a log line is not proof. It says server errors and rate-limiting signals such as HTTP 429 make it crawl less, and that its own Crawl Stats report counts most but not all requests, so a log can show detail the report does not. A log shows crawling, not indexing. See how to verify requests from Google and the crawl budget guide.
Ask your host or developer for the raw access logs for the last two to four weeks. Each line should be one request.
You will know it worked when You have a file in which each line is one request, with an address, a time, a status code and a user agent.
Search the file for Googlebot in the user agent part of each line. Treat these as claims only, because any program can use that name.
You will know it worked when You have a count of lines that claim to be Googlebot, and a copy of them in a separate file.
Take the IP addresses from a handful of those lines. For each, run a reverse DNS lookup, confirm the name ends in googlebot.com, google.com or googleusercontent.com, then run a forward lookup on that name and confirm it returns the same IP address. See Google’s verification steps.
You will know it worked when Each sampled IP address either passes both lookups or fails, and you have noted which.
Group the verified requests by the status code the server returned. Google only indexes pages served with a 200 status, and says server errors and HTTP 429 make it crawl less.
You will know it worked when You know roughly what share of requests got 200, a redirect, 404 and a 5xx error.
Search the verified requests for five pages you care about. A page Google has found but not requested may simply not have been crawled yet. See the Page indexing report help for what “discovered” means.
You will know it worked when For each important page you know whether Googlebot requested it in the period.
List the most requested addresses that returned a redirect or an error. Google says long redirect chains have a negative effect on crawling and that soft 404 pages keep being crawled.
You will know it worked when You have a short list of addresses Googlebot keeps requesting that return redirects or errors.
In Search Console, open the Crawl Stats report from the property settings. Google says it counts most but not all requests, so slight differences from your log are expected. It is only available for root-level properties. See the Crawl Stats report help.
You will know it worked when You have noted the difference between the report’s total and your log’s count, and whether it is small.
Tell Dean which pages are missing from the log and what status codes you see. A reply from Dean is how it starts, not a sales sequence.
Go deeper
