SEOscanner.app

Rule · robots.txt

Named crawler groups replace the * group

SEO-ROBOTS-07C4 Informational· w0informational

Class
C4 Informational
Weight
0
Severity on fail
info
Applies to
Google and Bing

Test this rule on a live site with the robots.txt checker. Free, no account needed.

Scope and evidence

Evaluated on each in-scope origin. The book marks this requirement SEARCH-ENGINE-SPECIFIC.

Observations

Each condition has a stable code and a reason template. The evaluator fills the placeholders from observed values only; no sentence in a report is generated.

  • shadowed_rules

    robots.txt has a group for {token} (line {line}). {crawler} follows only that group, so these rules from the * group do not apply to it: {rules}. If they should, repeat them in the {token} group.

Sources, quoted

These are the passages the rule rests on, stored once in the knowledge base and emitted exactly as written. A citation marked for specific conditions appears only on findings with those codes.