SEOscanner.app

Rule · robots.txt

robots.txt summary

SEO-ROBOTS-01C4 Informational· w0informational

Class
C4 Informational
Weight
0
Severity on fail
info
Applies to
Any engine

Test this rule on a live site with the robots.txt checker. Free, no account needed.

Scope and evidence

Evaluated on each in-scope and variant origin. The book marks this requirement REQUIRED, STANDARD (location).

{interpretation_clause} is Google's interpretation of the status: for a 4xx other than 429 it is " Google's crawlers treat all 4xx errors, except 429, as if a valid robots.txt file didn't exist"; otherwise it is empty. The location requirement is satisfied by construction because the scanner only reads /robots.txt.

Observations

Each condition has a stable code and a reason template. The evaluator fills the placeholders from observed values only; no sentence in a report is generated.

  • summary

    robots.txt on {origin} returned {outcome} ({bytes} bytes, {hops} redirect hops, {groups} groups at lines {group_lines}, {sitemaps} Sitemap lines).{interpretation_clause}

Sources, quoted

These are the passages the rule rests on, stored once in the knowledge base and emitted exactly as written. A citation marked for specific conditions appears only on findings with those codes.