Globetrotters AI Presence

0.5.0

• The AI crawler list your robots.txt names now matches the one Globetrotters itself publishes and grades sites against: 20 named crawlers instead of 11, covering OpenAI, Anthropic, Google, Microsoft, Apple, Meta, Amazon, Perplexity, DuckDuckGo, Common Crawl, ByteDance and Cohere.
• Listing those crawlers no longer changes what they may access on your site. Earlier versions gave each listed crawler its own "allow everything" entry, and a crawler with an entry of its own ignores your site's general rules — so each was released from WordPress's standard block on `/wp-admin/`, and from a site-wide block on a staging site. Now a crawler your robots.txt already names keeps exactly the rules you gave it, and every other one follows the same rules as all other crawlers on your site. Your robots.txt is read the way Google reads it, so a line such as `Content-Signal` or `Crawl-delay` placed between two `User-agent` lines does not change which rules those crawlers follow.
• Each listed crawler carries a `Content-Signal` line stating that your content may be used for search and for answering questions. Nothing is said about AI training unless you choose to — and if your robots.txt already has its own `Content-Signal` line, that one is used instead, including when you give the listed crawlers access to everything.
• Two new options under **Settings → AI Presence**, both off by default because each is a decision about your whole site: give the listed AI crawlers access to everything, including the paths closed to other crawlers; and tell them your content may be used to train AI models.
• If a refresh receives a web page where a discovery file should be — a maintenance page, a login screen or a proxy error that still answers "OK" — the refresh is now refused and your site keeps serving the files it already had. Previously the page was saved in place of the file and the refresh reported success.
• A value in your presence content containing HTML-like text such as `<!--<script>` could make browsers read the rest of your homepage as part of the structured data in its head, hiding the page. Those characters are now written in an escaped form that structured-data readers decode straight back, so nothing in the content can affect how your page is displayed.
• Your robots.txt now points search engines at a sitemap **on your own domain** instead of at Globetrotters. The old line named another host, which search engines ignore, so the discovery files had no sitemap entry anywhere that counted.
• The plugin serves that sitemap itself, at `/ai-sitemap.xml`. It lists your homepage and every discovery file this site actually serves, so it can never point at an address that isn't there. Your existing sitemap is untouched: nothing is served at `/sitemap.xml`, WordPress's own `/wp-sitemap.xml` is left alone, and an SEO plugin keeps whichever address it uses — robots.txt simply lists both.
• Plugin updates now reach your site. Every earlier version answered WordPress's update check in a form WordPress silently discards, so no update was ever offered in wp-admin. Install this version by hand once; updates after it arrive normally.
• Clearing the Globetrotters website URL now stops publishing straight away: the discovery files, the homepage structured data and the search-engine verification file are all removed until you connect again. Pointing the plugin at a different website URL likewise clears the previous one's files, and the new ones are served as soon as the first refresh completes. Previously the last files kept being served indefinitely.
• Agent traffic reporting only ever sends to an `https://` endpoint and never follows a redirect, so your ingest token and the visitor addresses it reports cannot travel unencrypted, or to another address.
• Trusted proxies are now a list. Tick **This site sits behind Cloudflare**, or enter the addresses of your own proxies or load balancers: the visitor address is read from forwarding headers only on requests that actually come from one of them. Previously a single checkbox trusted those headers from anyone, so on a site still reachable directly, a visitor could pass for a known AI crawler. A ticked checkbox now reads as the Cloudflare option — if your proxy is something else, add its addresses.
• Reporting bookkeeping and refreshes no longer overwrite each other: a busy moment can no longer lose count of dropped requests, or undo a refresh that finished at the same time.