Google adds AI training signal to robots.txt

Google announced that it will support a common signal that lets publishers express their preference about AI training with a single rule. The announcement was published on the Google Search Central blog on October 30, 2026. The same day, Microsoft and OpenAI made similar commitments.

Until now, each AI provider offered its own separate control: Google-Extended, OpenAI’s crawler settings, and Microsoft’s NoArchive and NoCache tags. Site administrators had to manage all of them one by one.

One signal, multiple providers

In the new approach, a common policy token written in robots.txt applies to all participating providers. Google says that if a site writes a training ban for this token but does not separately specify Google-Extended, it will treat that as a Google-Extended ban. Other providers can read the signal in a similar way, so a publisher can either opt out of all of them at once or make choices one by one.

Provider-specific controls such as Google-Extended remain in effect. Google notes that sites allowing Google-Extended will continue to appear as sources and receive clicks in answers in Gemini apps. The company aims to support the new signal in 2027.

What the IETF draft says

The mechanism is defined in an Internet-Draft titled “Machine-Readable Training Preference Signal” dated October 30, 2026, by Kevin Kelley of Anthropic. The draft is progressing on the standards track and expires on May 3, 2027; according to IETF records, this is an individual submission and does not mean official approval by the organization.

The method does not introduce a new command or grammar. Site owners write ordinary robots.txt groups using the __usage_ai-training token in the User-Agent line; path syntax, the longest-match rule, and group structure remain as in RFC 9309. Unknown groups are ignored by existing parsers, the wildcard (*) group does not match this token, and if no such group exists in the file, no preference is considered to have been expressed.

The token’s meaning comes from the AIPREF vocabulary: using an entity to change a generative model’s learned parameters. Models trained only for non-generative tasks fall outside this definition. The draft emphasizes that this is not a command about crawling behavior, but a preference about content use.

Microsoft and OpenAI joined the same day

Microsoft announced on the Bing blog that it will support the common training preference at site and path level, and also bring it to page level with meta tags and HTTP headers. The company is also adding a new “AI Grounding” preference; the DisallowExtended option limits grounding use to its own surfaces such as AI Mode, Copilot, and Microsoft 365, excluding grounding services provided to third parties. When neither preference is published, permission is assumed.

At Microsoft, the new preferences will take effect during crawling starting February 1, 2027. Temporary transition preferences for sites using the old tags will open in Bing Webmaster Tools on December 15, 2026; those sites must publish the new preferences by March 1, 2027. OpenAI said it will honor the common signal, implement it in the coming months, and keep its existing crawler controls.

Practical takeaway for publishers

Google describes the announcement as aligned with the code of practice for general-purpose model providers under the EU AI Act; in that code, robots.txt is considered the main way for a publisher to express a machine-readable preference. On paper, a publisher will be able to tell all participating providers “do not use for training” with a single line. But the draft is not yet a standard and implementation dates stretch into 2027; in other words, nothing is final yet, but the direction has been set.

Source: Unite.AI