One Line in robots.txt to Opt Out of AI Training? What Small Business Websites Should Do Now

What was announced

On 30 October 2026, Google and Microsoft each published plans to support a shared way for websites to say "don't use my content to train AI models". It lives in robots.txt, the plain-text file at the root of a website that tells automated visitors what they may fetch. It's based on work at the Internet Engineering Task Force (IETF), the body that develops internet standards.

• Google says: "Instead of managing separate instructions for every AI developer, website owners will be able to use a shared 'policy token' within robots.txt to set broad preferences across participating AI providers." It adds: "We aim to support this new control along with other AI providers in 2027."
• Microsoft says that starting 1 February 2027 it "will honor the new preferences as it crawls".
• OpenAI says it will honour the signal. In its own post it says: "We aim to implement the signal in the coming months", and that its existing robots.txt controls will continue to apply.
• Anthropic is reported to have started implementing it. PPC Land quotes the draft's author, who is listed with an Anthropic affiliation, writing that Anthropic "has already started implementing support for this draft". He didn't give a date.
• Meta: The Verge says a Meta spokesperson signalled support but didn't say whether the company will honour the signal.
• The draft itself says: "This I-D is not endorsed by the IETF and has no formal standing in the IETF standards process." It expires on 3 May 2027.

So this is a draft with big names behind it, not yet a finished standard.

What the line looks like

The IETF draft's example uses a special name, called a policy token, in place of a crawler's name:

User-Agent: __usage_ai-training

Disallow: /

You can also allow some parts, for example Allow: /press-kit/, according to the draft. Microsoft's guide shows a path-level example too: Disallow: /example-path/.

Two details from the draft matter:

• A general rule for every crawler doesn't count. The draft says wildcard groups (User-Agent: *) "MUST NOT" be read as matching the token, and if no group names the token, "no preference has been expressed."
• It's a statement of preference, not a lock. The draft says "This policy token is not a directive for crawling behavior." It also says "Private agreements MAY override preferences."

Microsoft makes a similar point: its settings "govern use of content, not crawl access", and they "do not establish licensing or payment arrangements".

Training is not the same as appearing in AI answers

Microsoft separates two things, and says its grounding choice is a page-level setting only, with no site-wide robots.txt version. Training is "using an asset to modify the learned parameters of a generative AI model", a definition it takes from IETF work. Grounding is using your page as input to help inform an AI-generated answer, with links back to the source. Microsoft says the two choices are independent, and both default to Allow if you publish nothing.

Microsoft's own warning is worth reading before you opt out of anything: "If you select DisallowExtended or Disallow for grounding, you will see reduced brand visibility in AI experiences." It also says neither choice "guarantees inclusion, traffic, or payment", and that these settings don't change eligibility for traditional Bing search results.

Google's post says that when a site uses the shared token to opt out of training (and doesn't mention Google-Extended), Google "plans to interpret that as if the publisher had disallowed Google-Extended". It also says: "Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps." PPC Land reads this as meaning that opting out through the shared token would also keep a site out of Gemini answers that cite web pages. Google's post doesn't spell that out, so check Google's current documentation for Google-Extended before you decide.

Our take

Don't copy-paste a "block all AI" snippet today. Not from us, not from anyone. For most small businesses a website exists so that people find you. Microsoft says that disallowing grounding reduces brand visibility in AI experiences. Whether that matters to you is a business decision, and you should make it on purpose.

Decide now, add the line when it's live. It's a draft, nothing obliges anyone to read it, and the only firm date so far is Microsoft's 1 February 2027. Adding the line today may do nothing for companies that aren't reading it yet. Writing down your decision costs nothing.

Don't treat it as protection. It's a preference signal. The draft says it isn't a crawling directive and that private agreements can override it. We aren't lawyers and we take no position on copyright or on what a signal means legally.

What to do this week

1. Find out who controls your robots.txt. Ask your web host or whoever built your site whether you can edit it. We haven't checked what each website builder allows.
2. Make two decisions, separately. Do you allow your content to be used for training? Do you want to appear in AI answers?
3. Write them down in a note with today's date.
4. Set a reminder for January 2027 to check what Google and OpenAI have published, since both dates are still loose.
5. If your site uses NoArchive or NoCache tags, read Microsoft's guide. It says the old tags stop being read as training or grounding preferences after 1 March 2027, and a Bing Webmaster Tools transition step opens on 15 December 2026 for sites using those tags.

This is about public website content, not your customers' data.

If you also write your own site copy with AI, our FAQ page draft playbook shows a checked-draft approach using ChatGPT. Google's side of this touches Gemini, and Anthropic, which makes Claude, is among the companies backing the signal.

Get the next guide by email. One email when we publish a new AI workflow. Join the AgentShed newsletter.

How we made this

AI-assisted, and checked against Google's, Microsoft's and OpenAI's announcements, the IETF draft, The Verge and PPC Land, on 31 October 2026. We haven't tested whether any crawler honours the signal. Dates are the companies' own plans, and the opinions are ours. This isn't legal advice. See how we research a playbook.

FAQ

What is a policy token? It's a special name used in robots.txt in place of a crawler's name, so one rule can express a preference to every participating AI provider.

Will it stop AI companies using my website? Not by itself. It's a voluntary preference. The draft says it isn't a directive for crawling and that private agreements may override it. Each company says it will honour it on its own timetable. Anthropic hasn't given a date.

Do I have to add it? No. Both training and grounding default to allowed on Bing if you publish nothing, according to Microsoft.

Will it affect my Google search ranking? Google's post doesn't address that. Microsoft says its settings don't change eligibility for traditional Bing search results. Check Google's own documentation before changing anything.

Is it legally binding? We can't advise on that. The draft calls itself a preference signal with no formal standing, and we'd take advice before relying on it for anything legal.

Your next step

Ask whoever looks after your website whether you can edit robots.txt, and write down your two decisions: training, and appearing in AI answers.

---

Sources (all read 31 October 2026): Google Search Central Blog · Bing Blog · IETF draft-kelley-aipref-training-preference-signal-00 · OpenAI · The Verge · PPC Land