robots.txt and noindex: diagnose crawling and indexing blocks
Separate crawl permission from indexing instructions and inspect the exact URL.
check.uk.app technical team · Reviewed 26 September 2026
Decide what you want to control
robots.txt manages crawling by cooperating bots. noindex instructs supporting search engines not to index a resource. Google must be able to fetch a page to discover its noindex; blocking that page in robots.txt can prevent this. Neither mechanism protects private data: use authentication for restricted content.
Test a narrow robots rule
This example disallows /drafts/ but allows the more specific /drafts/public/ path. Replace the sitemap address with your own. It is a rule demonstration, not a recommendation to publish confidential drafts. Rules apply to a particular protocol, host and port.
User-agent: *
Disallow: /drafts/
Allow: /drafts/public/
Sitemap: https://example.com/sitemap.xmlRead the matched rule in the report
Submit the complete page URL, then open robots.txt and inspect the Googlebot or Bingbot result, matched directive and source line. Test both /drafts/private-page and /drafts/public/page. A homepage check does not tell you whether a nested URL is allowed.
The service simulates rules for the submitted URL. It does not simulate crawler caches or HTTP-error fallback policies. An unavailable or incomplete robots response yields unknown. The final redirected URL can have different rules.
Inspect noindex separately
In Indexability, compare HTML robots directives with X-Robots-Tag and any crawler-specific values. noindex belongs in a supported HTML meta tag or HTTP header, not a Google robots.txt directive. The checker reads returned HTML without executing JavaScript.
Do not remove a restriction before checking why it exists. Test a public page intended for search, not an account or private document. Save the old rule and repeat the same URL after changing it.
Confirm search-engine evidence
An allowed crawl and no detected noindex do not prove that a page is indexed. Use URL Inspection in Search Console for a property you control and compare its fetched page with the public response. Rechecking our report does not request indexing.
The example is tested against the service’s rule evaluator: the private path is disallowed and the public exception is allowed for both supported bots. This is a parser test, not a live Googlebot visit.
