Diagnose Googlebot Access Errors: 401, 403, 429, and 5xx

Collect evidence before you touch a firewall rule: the URL, the time, the status code Google reported, and whether that request ever reached your server. Confirm the requester was really Google with a reverse and forward DNS lookup. Then find the layer that produced the error, whether authentication, a CDN or firewall rule, rate limiting, or WordPress itself, and make the smallest change at that layer. No SEO plugin runs on a request that was blocked before WordPress loaded.

This guide covers infrastructure access failures. Rules in robots.txt are covered in Fix a robots.txt block.

What each status means to Google

The report descriptions are from Google’s Page Indexing report help, and the 429 behavior is from Google’s HTTP status documentation, both checked September 27, 2026. The status documentation adds three points that shape the fix:

• “All 4xx errors, except 429, are treated the same”: Google is told the content does not exist. A 403 served to Googlebot on a real page tells Google the page is not there.
• “5xx and 429 server errors prompt Google’s crawlers to temporarily slow down with crawling,” and “Google’s indexing pipeline removes from the index URLs that persistently return a server error.”
• “Don’t use 401 and 403 status codes for limiting the crawl rate.”

Confirm the request came from Google

Any client can put “Googlebot” in its user-agent string, and security tools block impostors for that reason. Before deciding a block was a mistake, verify the IP address in your log. Google’s verification documentation, checked September 27, 2026, gives four steps:

1. Run a reverse DNS lookup on the IP address with host.
2. Check that the name ends in googlebot.com, google.com, or googleusercontent.com.
3. Run a forward lookup on that name.
4. Check that it returns the original IP address.

host 66.249.66.1
host crawl-66-249-66-1.googlebot.com

The IP address above is the example from Google’s page. For automated checks, Google publishes its crawler ranges as JSON files, starting with common-crawlers.json for Googlebot. A request that fails verification is not Google; blocking it is fine and needs no change.

The Search Console URL Inspection live test fetches from Google, using its Google-InspectionTool crawler, so it is the easiest way to see what Google gets right now. Google’s URL Inspection help, checked September 27, 2026, notes that server errors can be transient: a live test can hit an error that crawling did not, or succeed where crawling failed. Sending a Googlebot user-agent string with curl from your own machine does not reproduce a block based on IP address or reputation.

Find the layer that answered

Take the time and URL from Search Console or the live test, then look for that request in the origin server’s access log. Your host’s control panel usually links to the logs.

grep -E "Googlebot|Google-InspectionTool" access.log | awk '$9 >= 400 {print $4, $7, $9}' | tail -50

Live test requests identify as Google-InspectionTool, and that user agent string does not contain the word Googlebot, so the pattern matches both. Google lists both user agents on its common crawlers page, checked September 27, 2026. The command also assumes the combined log format, nginx’s default and a standard Apache format, where the ninth field is the status code and the user agent is recorded at the end of the line. The plain Common Log Format has no user agent, so the grep finds nothing there. The Apache documentation defines both formats, and the nginx documentation defines combined as its default, both checked September 27, 2026. Where the request shows up tells you who answered:

Search Console’s Crawl Stats report, checked September 27, 2026, adds context. Its Host status shows failures fetching robots.txt, resolving DNS, and connecting to the server, and its response breakdown lists 401, other 4xx, 5xx, DNS, and timeout errors among the bad response codes. Google says sites with fewer than a thousand pages should not need the report for crawl detail, but host status is still useful here.

WordPress causes to check

These are causes worth checking on WordPress sites, not a ranking of how often each happens:

• Security plugin rules that rate-limit fast visitors, block countries, or block anything claiming to be Googlebot without verifying it.
• CDN bot protection or challenge pages that a crawler cannot complete, returned as a 403 or 503.
• HTTP authentication left on after a staging site became the live site, which produces 401 on every URL.
• Host resource limits on shared hosting, where extra traffic returns 503 or another 5xx while the account is throttled.
• PHP fatal errors after a plugin or theme update. These can affect a single template, so the home page works while single posts return 500.
• Database connection failures, where WordPress cannot reach its database and every page fails.
• A failing robots.txt. Google’s robots.txt specification, checked September 27, 2026, says a server error on robots.txt makes Google stop crawling the site for 12 hours while it retries, then fall back to its last cached copy for up to 30 days.

Make the minimum correction

Change the one rule responsible, not the whole security setup.

1. Firewall or CDN block on verified Google traffic. If the vendor offers a setting that allows verified search engine crawlers by checking more than the user agent, use it. Do not allow requests by user-agent string alone; that lets impostors through.
2. Authentication on a live site. Remove it from the public site. If part of the site must stay private, keep only that part behind a login.
3. Rate limiting. Exempt verified crawlers, or raise the threshold. Google’s crawling troubleshooting page, checked September 27, 2026, says you can return 503 or 429 temporarily when the server is overloaded, and that Googlebot retries those URLs for about 2 days. It also warns that returning 503 or 429 “for more than 2 days will cause Google to drop those URLs from the index,” and that “no availability” codes returned “for more than a few days will cause Google to permanently slow or stop crawling URLs on your site.”
4. Application errors. Read the PHP error log, identify the plugin or theme, and roll it back or update it. Test the affected template afterward.
5. Server capacity. If errors line up with traffic peaks, the fix is caching, fewer expensive requests, or a larger plan. Your host can tell you which limit was hit.

WP Visibility, like any SEO plugin, runs only when WordPress handles the request. It cannot see or change a block at the CDN, the host firewall, or the web server, and a 500 raised before its code loads never reaches it.

Build an evidence bundle for your host

When the block is outside WordPress, send the host or CDN support team one message with:

• The affected URLs and the exact times of failed requests, with time zone.
• The status codes from Search Console and from the URL Inspection live test.
• Log lines for those requests from the origin, or a note that none exist.
• The result of your reverse and forward DNS checks on the requesting IP address.
• Any security, CDN, or firewall rule names shown in the block log.
• What you already changed, and when.

After the fix, run Test live URL on an affected page, watch Host status in Crawl Stats, and click Validate fix for the reason in the Page Indexing report. Google says validation typically takes up to about two weeks. After 5xx or 429 errors, Google’s HTTP status documentation says that once the server returns 2xx again, “Google gradually increases the crawl rate,” so recovery is gradual rather than immediate. For the other reasons Google can report, see the indexing triage guide, and for URLs Google has found but not yet crawled, Discovered, currently not indexed.