SEOscanner.app

Rule · robots.txt

Staging hosts require authentication

SEO-ROBOTS-09C2 Recommended· w4

Class
C2 Recommended
Weight
4
Severity on fail
major
Applies to
Google

Scope and evidence

Evaluated on each configured staging host. The book marks this requirement DOCUMENTED RECOMMENDATION. It needs config; without that input the rule reports as not evaluated rather than guessing.

Fail conditions

Each condition has a stable code and a reason template. The evaluator fills the placeholders from observed values only; no sentence in a report is generated.

  • staging_public

    {host} answered {status} without credentials. robots.txt is not a mechanism for keeping pages out of Google; Google's guidance is to block indexing with noindex or to password-protect the page.

The documented fix

Put HTTP authentication or SSO in front of every non-production environment.

Sources, quoted

These are the passages the rule rests on, stored once in the knowledge base and emitted exactly as written. A citation marked for specific conditions appears only on findings with those codes.