AI Training Opt-Out in robots.txt: What to Do Now

Four of the biggest names in AI just agreed on one line of robots.txt. The line tells their crawlers not to train on your site. None of them honors it today. If you read the announcements closely, the part that can cost you money is not the

Four of the biggest names in AI just agreed on one line of robots.txt. The line tells their crawlers not to train on your site.

None of them honors it today. If you read the announcements closely, the part that can cost you money is not the opt-out. It is a Bing deadline hidden underneath it.

On October 30, 2026, Google, OpenAI and Microsoft each published their own post on the same idea: a shared AI training opt-out in robots.txt. The signal comes from an individual IETF draft written by Kevin Kelley of Anthropic. Here is what it means if you own a business website, and what to do this week.

The new signal is three lines in your robots.txt

The draft defines one token, __usage_ai-training. You use it like a crawler name in a normal robots.txt group.

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That example comes straight from the draft. It says: no AI training on anything, except the press kit.

The draft adds no new directives and no new grammar. It says the token "is not a directive for crawling behavior", so it does not stop a bot from visiting. It states a preference about what the bot may do with what it reads.

The definition of training is narrow. The draft uses "using an asset to modify the learned parameters of a generative AI model". Search indexing is not in that definition.

Nobody honors it yet, and it is not a standard

Read the dates before you celebrate. OpenAI says it aims to implement the signal "in the coming months" and will publish details when it does. Google says it aims to support the control in 2027. Microsoft's date is the most concrete: February 1, 2027.

The document itself is an individual Internet-Draft, marked as not endorsed by the IETF. It expires on May 3, 2027 unless someone renews it or the working group adopts it. That is a proposal with serious backers, not a ratified rule.

Two more limits matter:

• Existing controls stay. Google and OpenAI both say their current crawler controls keep working next to the new signal. Adding the token does not replace a GPTBot or Google-Extended rule.
• Old data stays. An opt-out is a preference about future use. Nothing in these posts says content already collected gets pulled out of anything.

So if your goal is "stop AI companies using my content", the honest status is this. Add the token if you want to be early, and keep everything else you already have.

Training and grounding are different decisions

This is where owners get hurt. Training is a slow background harvest. Grounding is a live fetch: someone asks an assistant a question, the assistant pulls your page, and you get cited.

Google's post says that when a site uses the token to opt out and does not mention Google-Extended, Google plans to treat it as a Google-Extended disallow. It also says: "Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps."

Read that twice. Our reading: a blanket training block could take you out of Gemini Apps answers too. Google does not say that outright, but it ties eligibility to allowing the control you would be switching off.

Bing draws the line more cleanly. In Microsoft's post, training is set in robots.txt, and grounding is a separate page-level preference. Bing says neither affects eligibility for traditional search results. It also says allowing grounding makes content eligible for links and citations, though "neither choice guarantees inclusion, traffic, or payment."

Bing's March 1 deadline is the part that bites

Most coverage stopped at the robots.txt token. The Bing post has a second, quieter change.

Some sites use noarchive or nocache tags to keep content out of AI answers. Microsoft says those tags currently work as a preference, and after March 1, 2027 they "will no longer be read as a preference for training or grounding." Bing is replacing them with explicit settings.

The schedule from Microsoft's post:

1. December 15, 2026: temporary grounding preferences open in Bing Webmaster Tools, only for sites that use noarchive or nocache.
2. February 1, 2027: the new preferences take effect as pages are crawled.
3. March 1, 2027: the deadline. Sites relying on the old tags must have published page-level grounding preferences and recorded a temporary preference in Bing Webmaster Tools.
4. September 30, 2027: the temporary grounding preferences expire.

If you miss it, Microsoft says affected pages default to Allow for grounding until they are recrawled. Training has no temporary setting at all.

Who is exposed? Anyone whose site was built years ago and picked up a noarchive or nocache tag along the way, from a freelancer, a theme or a plugin, then forgot about it. You can check in a minute: view the page source and search for noarchive and nocache, or look at the response headers.

What we actually run on our own blog

Here is first-party data, and it is not what you might expect. The robots.txt on this blog explicitly allows GPTBot, ClaudeBot, PerplexityBot, Google-Extended and Meta-ExternalAgent. No training block.

That is deliberate. This blog is a young domain, and its job is to get cited when a business owner asks an assistant a question. Being unread by AI costs us more than being read.

Client sites are a different call. Our default there is allow Search, allow live AI fetches, block Training, because most owners have no reason to donate their copy to a model they will never see a cent from. It is the owner's decision, and the setting is easy to flip.

Block training if you like. Do not block the live fetch. That is the one that gets you cited when a customer asks.

— our rule for service-business sites

If you sell original photography, paid courses, premium reports or anything a model could reproduce in place of buying it, blocking training is an easy call. If you are a local clinic, a law firm or an HVAC company, the live fetch is the part with revenue attached, and it deserves protecting even when you opt out of training.

Three ways this goes wrong on a small business site

Your robots.txt is not yours to edit. Many site builders and CDNs generate robots.txt for you. Cloudflare, for example, can write content signals into robots.txt automatically from your dashboard settings, which we flagged in the earlier crawler post. A hand-edited file and a generated file can disagree. Whoever ran your setup last may not remember which one wins.

You copy a rule from a forum and block more than you meant to. A single stray Disallow: / under the wrong user agent can cut a search crawler off. The token group is separate from the * group, so mind which lines sit under which header.

You set it once and never test. Support arrives in stages, from the coming months to 2027. Add a calendar entry for February 2027, when Bing's preferences start to apply, and check the file again then.

For the Meta side of the same question, our post on what advertisers must do about Meta's new Commercial Terms covers Meta-ExternalAgent and a robots.txt that matches what you want. If you are still working out how to get cited at all, start with the AEO playbook for ChatGPT, Perplexity and Gemini and check whether your pages can survive an agent visiting them in is your website ready for AI agents.

A 20-minute check you can do this week

1. Open yourdomain.com/robots.txt. Read it top to bottom. Note every bot named, and whether the file looks hand-written or generated.
2. Search the page source for noarchive and nocache. If you find them, find out who added them and why.
3. Decide the three questions separately. Search: allowed. Live AI answers: allowed, unless you have a reason. Training: your call, based on whether your content is the product.
4. If you want training blocked, add the token group and keep your existing rules. Do not delete the GPTBot or Google-Extended lines.
5. Put February 1, 2027 and March 1, 2027 in your calendar. If step 2 found anything, those are your deadlines.

Most owners can do this alone. The point where it stops being a 20-minute job is when you have several sites, a CDN in front, a builder that controls the file, or a Bing Webmaster account nobody can log into. That is the work we do for clients on every website build: crawl rules, bot settings and page-level directives set once, written down, and checked again when the rules change.