- Class
- C1 Mechanism
- Weight
- 8
- Severity on fail
- critical
- Applies to
Test this rule on a live site with the robots.txt checker. Free, no account needed.
Scope and evidence
Evaluated on each in-scope origin with a 2xx robots.txt. The book marks this requirement REQUIRED, SEARCH-ENGINE-SPECIFIC.
Fail conditions
Each condition has a stable code and a reason template. The evaluator fills the placeholders from observed values only; no sentence in a report is generated.
invalid_utf8robots.txt on {origin} is not valid UTF-8 (first invalid byte on line {line}). Google requires a UTF-8 encoded plain text file and ignores invalid lines, so rules on the affected lines may not apply.
The documented fix
Save and serve robots.txt as UTF-8.
Sources, quoted
These are the passages the rule rests on, stored once in the knowledge base and emitted exactly as written. A citation marked for specific conditions appears only on findings with those codes.
“The robots.txt file must be a UTF-8 encoded plain text file and the lines must be separated by CR, CR/LF, or LF.”
[G-ROBOTS-SPEC] How Google interprets the robots.txt specification · File format · last updated 2026-08-31
https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec“Google ignores invalid lines in robots.txt files, including the Unicode Byte Order Mark (BOM) at the beginning of the robots.txt file, and use only valid lines.”
[G-ROBOTS-SPEC] How Google interprets the robots.txt specification · File format · last updated 2026-08-31
https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec