site.robots
- Default
- warn
- Plugin
site
A robots.txt exists, and does not block every crawler.
Why: Crawlers read robots.txt first. A missing file is fine for them, but often means it was left out of the build. A Disallow: / for User-agent: * blocks the whole site, which is often left over from a staging site.
When off: a missing or blocking robots.txt passes.
# ✗ Fails
User-agent: *
Disallow: /
# ✓ Passes
User-agent: *
Allow: /
References
- Google: Introduction to robots.txt: What robots.txt does. A site can work without one, so the rule warns.
- Google: How Google interprets robots.txt: How Google reads robots.txt, as seolint does.
- RFC 9309: Robots Exclusion Protocol: The robots.txt standard.