Robots.txt Checker

Robots.txt Basics

The robots.txt file tells search engine crawlers which pages they can or cannot access.

# Example robots.txt
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/

Sitemap: https://example.com/sitemap.xml