Login

Robots.txt & Sitemap Inspector

Check your robots.txt and sitemap directives instantly. Catch wildcard blocks and missing sitemaps before they hurt SEO.

Run your first inspection to visualize robots.txt rules and sitemap coverage here.

What are robots.txt and sitemaps?

robots.txt is a text file that tells search engine crawlers which pages or sections of your website they can or cannot access. It uses directives like User-agent, Disallow, Allow, and Sitemap. A sitemap is an XML file that lists all important pages on your website, helping search engines discover and index your content faster. Both files work together to control how search engines crawl and index your site.

Why use this robots.txt and sitemap checker?

This free inspector analyzes both your robots.txt file and sitemap in one check. It detects dangerous wildcard blocks like Disallow: /, verifies that your sitemap is referenced in robots.txt, checks if the sitemap URL is accessible, and ensures search engines can access all important pages. Fix crawl and indexing issues instantly to maintain visibility in search results.

Robots.txt Directives

  • User-agent — Specifies which crawler
  • Disallow — Blocks specific paths
  • Allow — Overrides Disallow rules
  • Sitemap — Points to XML sitemap location

Sitemap & Common Issues

  • Sitemap.xml — Lists all important URLs
  • Wildcard blocks — Disallow: / blocks everything
  • Missing sitemap — No sitemap reference or inaccessible
  • Accidental blocks — Important pages blocked
Complete Website Analysis

Check everything at onceand much more

Launch a free analysis in under 30 seconds. Get performance, SEO, and trust scores with actionable fixes—all in one dashboard.

All SEO checks
Performance
Trust scoring
Daily monitoring
Certified reports
Start Website Analysis
FAQ

Everything you need tounderstand

Get answers to the most common questions about robots.txt, sitemaps, and crawl control

Paste your domain into this robots.txt tester. It retrieves the live file, parses User-agent sections, Allow/Disallow rules, crawl-delay directives, and sitemap references so you see exactly what Googlebot reads.
A missing (404) or blocked (403) robots.txt means crawlers default to full access but you lose control. The tool flags those responses and recommends hosting a simple file—even if it only lists your sitemap.
A wildcard block with `Disallow: /` prevents every crawler from indexing your site. Our inspector highlights this in red so you can remove the directive or add specific Allow rules before rankings disappear.
Always reference your XML sitemap at the bottom of robots.txt with `Sitemap: https://yourdomain.com/sitemap.xml`. It accelerates discovery for Google, Bing, Yandex, and every SEO crawler.
Check robots.txt any time you ship a redesign, move folders, launch a new CMS, or notice sudden indexing drops. A two-minute audit prevents critical sections from being blocked accidentally.
Yes. Keep production pages open with `User-agent: *` plus `Disallow: /admin/` or `/staging/`. The report shows every blocked path so you can confirm only private areas are excluded.
Most major engines ignore crawl-delay, but some third-party bots respect it. We surface the values so you know when legacy directives might slow down helpful crawlers.
Absolutely. The tool lists every User-agent section (Googlebot, Bingbot, AdsBot, etc.) so you can confirm customized rules fire only for the crawler you intend.
Use a broad Disallow such as `/private/` and pair it with `Allow: /private/document.pdf`. Our rule breakdown shows whether the exception is in place and properly ordered.
No. Each robots.txt and sitemap fetch happens on demand, displayed in your browser, and discarded immediately. Nothing is persisted on our servers.