KNOW WHO CAN GET IN
Check the rules at your website’s front door.
Inspect AI crawler robots rules, indexability and public website access without confusing search with training.
Your robots.txt can express different rules for search crawlers, training crawlers and user-directed fetchers. Knowing the difference helps you make an informed publishing decision.
Search and training controls serve different purposes
CMB distinguishes OAI-SearchBot, PerplexityBot and Claude-SearchBot from training controls such as GPTBot, ClaudeBot and Google-Extended. Training opt-outs are respected and are not treated as visibility failures.
A robots rule is not an access test
Robots rules describe permissions for compliant crawlers. A CDN challenge, authentication requirement or firewall can still prevent access. CMB requests pages using its own identified scanner agent, not by pretending to be an AI provider.
Read rules in the context of a page
Specific agent groups, wildcard paths and Allow exceptions can change the result. The report records the requested URL and the rules evaluated, and marks the result unknown when robots.txt could not be obtained reliably.
A common question
Will you bypass Cloudflare or my security protections?+
No. If your website presents an access challenge, CMB reports that limitation. You can decide whether your public crawler policy needs review.
Keep exploring
MAKE YOUR NEXT STEP A CLEAR ONE
Find out what your
website is saying to AI.
Start with a free check. Leave with a little more clarity.