- Class
- C4 Informational
- Weight
- 0
- Severity on fail
- info
- Applies to
Test this rule on a live site with the robots.txt checker. Free, no account needed.
Scope and evidence
Evaluated on each in-scope origin. The book marks this requirement SEARCH-ENGINE-SPECIFIC.
Bing said in 2012 that bingbot honors crawl-delay (B-ROBOTS-2012), so the info text names the engine. noindex lines are reported by SEO-IDX-03.
Observations
Each condition has a stable code and a reason template. The evaluator fills the placeholders from observed values only; no sentence in a report is generated.
ignored_linesGoogle ignores {n} lines in robots.txt on {origin}: {lines}.
Sources, quoted
These are the passages the rule rests on, stored once in the knowledge base and emitted exactly as written. A citation marked for specific conditions appears only on findings with those codes.
“other fields such as crawl-delay aren't supported”
[G-ROBOTS-SPEC] How Google interprets the robots.txt specification · Syntax · last updated 2026-08-31
https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec“Google ignores invalid lines in robots.txt files, including the Unicode Byte Order Mark (BOM) at the beginning of the robots.txt file, and use only valid lines.”
[G-ROBOTS-SPEC] How Google interprets the robots.txt specification · File format · last updated 2026-08-31
https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec