Robots.txt checker
Check your site’s robots.txt file against the robots.txt checks in an Evisio SEO audit report: whether it exists, whether it can be read, whether it blocks your whole site or AI crawlers, and whether it points to your sitemap. Enter your website to see the results in seconds.
What is a robots.txt file?
robots.txt is a plain-text file at the root of your site, such as https://www.example.com/robots.txt. It tells crawlers, the bots that search engines and AI tools send to read websites, which parts of the site they may visit:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.example.com/sitemap.xml
Each group starts with User-agent, naming the crawler it applies to (* means all of them), followed by Disallow and Allow lines listing paths. A Sitemap line tells crawlers where to find the list of your pages.
Why robots.txt matters for SEO
robots.txt is small, but one wrong line can hide a whole site from Google. Disallow: / under User-agent: * tells every crawler to stay out of everything. It is a common leftover from a site that was built behind a “discourage search engines” setting and launched without switching it off.
The file also decides whether AI assistants such as ChatGPT, Claude and Perplexity can read your site. Some plugins and hosts block their crawlers by default, which can keep your business out of AI answers without you knowing.
How to use this checker
- Enter your website’s address. The checker reads
/robots.txtat the root of the site. - Read each result. The checker shows the paths disallowed for all crawlers, the sitemaps the file declares and any AI crawlers it blocks.
- After you change the file, run the check again to confirm the new version is live. Caching plugins and CDNs can serve an old copy for a while.
What Evisio checks
Robots.txt found
Your site should answer /robots.txt with a file that has at least one rule. Without one, crawlers assume they may visit everything, which is fine for many sites, but you lose the chance to point them to your sitemap or keep them out of admin and search pages.
Blocked from crawling in robots.txt
Fails when the file disallows /, the whole site, for all crawlers. The result lists every path disallowed for all crawlers, so you can also spot folders you didn’t mean to block.
Sitemap not referenced in robots.txt
A Sitemap: line helps every search engine find your sitemap, not only the ones you have submitted it to. The result lists up to five declared sitemaps.
Invalid robots.txt format
Typos, stray characters or an HTML page served in place of the file can make it unreadable. Crawlers may then ignore rules you rely on.
AI crawlers blocked in robots.txt
Fails when the file blocks GPTBot, ClaudeBot, Google-Extended, PerplexityBot or CCBot from the whole site, and names the ones it blocks.
How to fix robots.txt issues
- Not found: on WordPress, a virtual robots.txt is built in, and Yoast SEO lets you edit it under Tools > File editor. Elsewhere, create a plain-text file named robots.txt in your site’s root folder.
- Whole site blocked: remove
Disallow: /. On WordPress, also check Settings > Reading and untick “Discourage search engines from indexing this site”. - No sitemap: add
Sitemap: https://www.yoursite.com/sitemap.xmlon its own line, using your sitemap’s real address. - AI crawlers blocked: remove the groups for the crawlers you want to allow, unless blocking them is a deliberate choice.
More detail: how to fix issues with robots.txt files, how to fix an invalid robots.txt format and robots.txt file not found.
Check the rest of your site
robots.txt is one file, but what it blocks affects every page. A free Evisio SEO audit report checks robots.txt along with your sitemap, your pages and more than 80 other checks, and turns each problem into a task with step-by-step instructions.
Questions
Does Disallow stop a page from appearing in Google?
Not reliably. Disallow stops crawling, but Google can still list a blocked URL if other sites link to it. To keep a page out of search results, let it be crawled and add a noindex tag. The indexability checker tests both.
Do I need a robots.txt file?
It isn’t required, but it is the simplest way to point crawlers to your sitemap and keep them out of pages that don’t belong in search.
Should I block AI crawlers?
That’s a business decision. Blocking them keeps your content out of AI training and AI answers. For most small businesses, being mentioned in AI answers is worth more, so they allow them.