Softrecs AI Crawler Access Check

AI answer engines can only cite pages they can read. This plugin checks, from inside your dashboard, whether they can read yours.

Go to Tools › AI Crawler Access and press Run the check. It looks at:

• robots.txt for search and answer bots. Googlebot, Bingbot, OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User. Rules are matched the way RFC 9309 says (the longest rule wins, Allow wins a tie), against your home page, newest post and newest page.
• A live request as each bot. robots.txt can allow a bot while a firewall, security plugin or CDN still refuses it. The plugin requests your home page with each bot’s user agent and compares the answer with a normal browser request. It also spots bot challenge pages.
• Indexing and snippet controls. noindex, nosnippet and max-snippet:0 in meta tags and the X-Robots-Tag header, a canonical that points to another site, and the “Discourage search engines” setting. Google says a page must be indexed and eligible for a snippet to appear in AI Overviews and AI Mode.
• XML sitemap, and whether robots.txt lists it.
• Text in the raw HTML. Most AI crawlers do not run JavaScript, so text added by scripts is invisible to them.
• HTTPS, and whether http:// redirects to https://.

Two things are reported but never scored:

• AI training bots (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent). Blocking these is your choice. It does not remove you from search results or AI answers, so it does not count against you.
• llms.txt. Google has said its Search features do not use it, and no major AI engine has said it reads the file when choosing what to cite. The plugin tells you whether you have one and leaves it at that.

What it cannot tell you

Whether any engine actually cites you. Being reachable is the first step, not the result. The plugin also runs from your own server, so blocks based on the bots’ IP addresses will not show. A clean live test means no user-agent rule is blocking them.

Privacy

Every request the plugin makes goes to your own site. It sends nothing to any other server, sets no cookies on visitors and adds nothing to the front end. The last result is saved in your user profile and deleted when you uninstall the plugin.