Crawl Issues
When the crawler cannot access pages.
Updated August 2026
Overview
If the crawl reaches fewer pages than expected, something is preventing access.
Possible causes
- robots.txt disallow rules.
- Authentication or password protection.
- Pages that are unavailable at crawl time.
- Redirects that break discovery.
- Server errors under crawl load.