All-IN Crawler & AI Agent Manager

Much of your traffic is bots. Some bring you customers: Google, Bing, ChatGPT search, social previews. Others only cost you server resources: aggressive AI training crawlers, SEO tools, scrapers and vulnerability scanners. All-IN Crawler & AI Agent Manager lets you decide, per category or per bot, who gets in and how fast, without hiding your site from search engines or AI assistants.

• About 120 known bots in 12 categories: search engines, AI search & assistants, AI training crawlers, SEO tools, social previews, monitoring, feed readers, archives, scripts & HTTP libraries, vulnerability scanners, unidentified bots and empty User-Agents.
• Allow, rate-limit or block per category, with per-bot overrides and custom User-Agent rules (text or regex).
• One-click presets: Balanced, Maximum AI visibility, Protect content from AI training, Lockdown.
• IP / CIDR allow and block lists (IPv4 and IPv6), and excluded paths for webhooks and payment callbacks.
• Protect content from AI training, stay in AI answers: training crawlers (GPTBot, ClaudeBot, CCBot…) are handled separately from AI search and assistant agents (OAI-SearchBot, ChatGPT-User, Claude-User, Perplexity-User…).
• Real search engines only: forward-confirmed reverse DNS plus optional official crawler IP lists (Google, Bing, Apple, OpenAI, Perplexity; off until you turn them on) catch fake Googlebots and GPTBots.
• Overload protection: rate limits per IP or per bot with 429 + Retry-After, load shedding with 503 + Retry-After when server load per CPU core crosses your threshold, and an optional per-IP limit for “human” visitors (logged-in users are exempt). SEO-safe status codes only, never cached.
• robots.txt and llms.txt: manage robots.txt rules (including Google-Extended, Applebot-Extended and an optional Crawl-delay) and generate an llms.txt that skips noindex content.
• Block before PHP: export rules to .htaccess (Apache/LiteSpeed) or nginx. Inside WordPress, requests are decided while plugins load, before the theme runs.
• Dashboard and reports: 7/14-day statistics, top denied IPs with one-click blocking, a denied-request log, a User-Agent tester and weekly or monthly email reports.
• Optional bot-list feed that adds new AI crawlers between releases (add-only and safety-checked). Off by default, and it only downloads from a URL you enter.
• Fast storage: APCu, then Redis/Memcached, then the database.
• Fails open: if anything goes wrong inside the plugin, visitors are let through and the error is logged.
• Full Hebrew translation, RTL ready.

Full documentation: see docs/USER-MANUAL.md (site owners) and docs/DEVELOPER.md (hooks & architecture).

More from A-in

This plugin is complete on its own: no licence key, no locked features. Premium plugins, add-ons and support from the same developer are available at a-in.co.il.

External services

All-IN Crawler & AI Agent Manager works without any external service. These features contact third parties:

• Official crawler IP lists (Bot Lists tab → “Download official crawler IP lists”, off by default, opt-in). Nothing is downloaded until you tick the box and save; the “Download official IP lists” button is also disabled until then. Once enabled, the site downloads the public IP range files published by the crawler operators once a day, to verify that a request claiming to be their bot really comes from them. No data about your site or visitors is sent: it is a plain HTTPS GET with a generic all-in-crawler-ai-agent-manager/<version> User-Agent, so the publisher only sees your server’s IP address. Untick the box to stop the downloads at any time. Sources:
  • Google: https://developers.google.com/static/search/apis/ipranges/ (Terms, Privacy)
  • Bing: https://www.bing.com/toolbox/bingbot.json (Terms, Privacy)
  • Apple: https://search.developer.apple.com/applebot.json (Terms, Privacy)
  • OpenAI: https://openai.com/gptbot.json, searchbot.json, chatgpt-user.json (Terms, Privacy)
  • Perplexity: https://www.perplexity.com/perplexitybot.json, perplexity-user.json (Terms, Privacy)
• Bot-list feed (Lists → Bot list updates, off by default). The plugin ships its extra bot definitions in data/bots.json and needs no download. If you enter a feed URL of your own (for example your agency’s feed), the site requests that URL once a day, sending your site URL in the User-Agent and, if you set one, your access key as a Bearer token. There is no default remote feed.
• DNS verification of search engine bots uses reverse and forward DNS lookups through your server’s resolver. No third-party API is called.