SEOscanner.app

Rule · robots.txt

robots.txt is smaller than 500 KiB

SEO-ROBOTS-03C1 Mechanism· w8

Class
C1 Mechanism
Weight
8
Severity on fail
critical
Applies to
Google

Test this rule on a live site with the robots.txt checker. Free, no account needed.

Scope and evidence

Evaluated on each in-scope origin with a 2xx robots.txt. The book marks this requirement SEARCH-ENGINE-SPECIFIC, STANDARD.

Fail conditions

Each condition has a stable code and a reason template. The evaluator fills the placeholders from observed values only; no sentence in a report is generated.

  • over_limit

    robots.txt on {origin} is {bytes} bytes. Google enforces a 500 KiB limit and ignores content after it; {n} rules from line {line} onward fall beyond the limit.

The documented fix

Consolidate rules with wildcards, or move unneeded rules out.

Sources, quoted

These are the passages the rule rests on, stored once in the knowledge base and emitted exactly as written. A citation marked for specific conditions appears only on findings with those codes.