Phylax AI Crawler Inspector

Phylax AI Crawler Inspector is a read-only diagnostic plugin for WordPress administrators who need clear evidence of how their site is presented to AI-related crawlers, search bots, training bots and user-triggered fetchers.

It inspects the served robots.txt, evaluates known AI-related user agents, checks the optional /llms.txt file, follows same-site redirect chains, reviews HTTP headers and HTML signals (canonical, meta robots, X-Robots-Tag), and can compare a neutral request with selected bot user agents.

Findings keep crawl access, HTTP access and indexing directives separate. The plugin explains evidence and limitations; it does not claim that any AI product has indexed, trained on, cited or ranked your site.

Product principle: inspect, explain and export. Do not modify, block, optimise or track. This is not an llms.txt generator.

Features

• Site overview audit of robots.txt, /llms.txt and the home page (bounded request budget)
• Robots Exclusion Protocol evaluation for a curated AI-related bot registry
• Structural validation of optional /llms.txt (neutral when absent)
• Same-site URL inspector for headers, redirects, canonicals and HTML robots signals
• Bounded bot response comparison (status, redirects, headers and HTML signals — never raw bodies)
• Sitemap discovery from robots.txt, HTML/HTTP Link signals and WordPress core sitemap declarations (no deep crawl)
• Structured findings with severity, status and expandable evidence
• Client-side JSON and CSV export from the current audit result
• Tools → Phylax AI Crawler Inspector admin screen with accessible status updates
• No custom database tables, no SaaS calls and no crawler-visit logging

What the plugin does not do

• Edit, inject or generate robots.txt or llms.txt
• Block crawlers with PHP, server or firewall rules
• Log crawler visits, IPs or analytics
• Contact external SaaS APIs or require an account
• Produce overall numerical scores, graded rankings or ranking guarantees
• Crawl the entire website or schedule WP-Cron scans
• Accuse cloaking or claim to know what an external crawler sees from another network location
• Create custom database tables or modify site SEO settings

Privacy

Phylax AI Crawler Inspector does not collect telemetry, analytics or promotional data. It does not call external SaaS APIs, load remote fonts or third-party admin scripts, or set its own cookies.

Audits use the WordPress HTTP API to request the administrator’s own site (same-site only). Response bodies are parsed in memory and are not permanently stored. Exported JSON/CSV reports are generated in the current admin browser session from the structured result and are not saved on the server by the plugin.

The only short-lived server-side data is an optional per-user audit cooldown transient (aici_audit_cd_{user_id}). Uninstall removes only plugin-owned aici_ options and transients. The plugin does not create custom database tables and does not log crawler visits.