Clownsteins Rogue Agent Defender

Clownsteins Rogue Agent Defender lets a site administrator choose which documented AI crawler tokens to block and which inference services WordPress itself may call.

The user-agent check returns 403 when a request header contains a selected token that the operator sends as a user agent. Google-Extended and Applebot-Extended are robots.txt tokens only, because those names are never sent as a User-Agent. Requests for the site’s robots.txt address are not blocked.

The robots.txt option adds one Disallow group to the robots.txt file WordPress generates. A robots.txt file already stored in the web root is left unchanged.

The HTTP API option blocks requests WordPress makes with wp_remote_* and the other WordPress HTTP API functions when the destination is a selected inference service. Programs that open their own connections are not affected.

Creator: Albert Clownstein.