Google, Anthropic, OpenAI and Microsoft commit to a unified IETF robots.txt directive that lets publishers block AI training crawlers with one rule

Overview

Google, Anthropic, OpenAI and Microsoft have committed to honour a single robots.txt user-agent that lets website owners opt out of generative AI model training with one site-wide rule. Engadget reported the development on 31 October 2026. The four developers will recognise a shared directive instead of requiring publishers to maintain separate entries for each proprietary crawler, such as GPTBot, ClaudeBot and Google-Extended.

The proposal is under review at the Internet Engineering Task Force (IETF), the body that develops many of the technical standards underpinning the internet. Its author is Kevin Kelley, technical lead for standards at Anthropic. Because the proposal has not been finalised, the syntax and scope could still change before it becomes a formal standard. The four companies have nonetheless set out their own implementation dates, which means the directive will have practical effect for at least some major AI developers before the IETF process concludes.

For Australian professional services firms, including environmental consultancies, law firms, planning practices and councils that publish technical material online, this changes a question that has mostly been left to IT contractors. Deciding which published content AI developers may train on has been a technical chore involving a growing list of crawler names. It is now becoming a single policy decision that can be documented, reviewed and approved by whoever is responsible for information governance and intellectual property.

Key details

The proposed standard reserves a common user-agent identifier, __usage_ai-training, which covers ingestion of web content for generative AI model pre-training and fine-tuning. A website operator adds this identifier to their robots.txt file, the plain text file at the root of a domain that tells automated crawlers which parts of a site they may access. As reported, an entry under the current draft could look like the example below. Because the proposal is not final, this format is illustrative only:

• User-Agent: __usage_ai-training
• Disallow: /
• Allow: /press-kit/

In that example, the Disallow line opts the entire site out of AI training crawls, and the Allow line keeps one folder open. The same pattern can be used to permit training on public relations material, product documentation or promotional content while excluding everything else. This granular control matters because many organisations want their public profile and contact details to appear accurately in AI-generated answers, while keeping detailed technical analysis, white papers and client advisories out of training datasets.

The implementation timeline differs between the four developers. Anthropic has already implemented support for the directive. OpenAI has pledged to roll it out over the coming months. Microsoft has committed to enforcement from 1 February 2027. Google has committed to an early 2027 rollout across its Google-Extended infrastructure. Until all four are live, a robots.txt file that relies only on the new identifier will not yet be honoured by every participating developer.

Google has confirmed that blocking the AI training user-agent will govern Google-Extended only and will not affect general Google Search crawling. This addresses the main risk that discouraged many organisations from blocking AI crawlers in the past. Under the individual-agent approach, some site operators blocked the wrong crawler or used overly broad rules and harmed their organic search visibility as a result. The unified directive explicitly separates general indexing from generative AI model training.

Two limits are built into the arrangement. First, the directive only governs future crawls. It cannot retroactively remove content that has already been collected and incorporated into existing foundational training sets. Second, it is a voluntary commitment from four developers. Other AI companies, data brokers and scrapers have not made the same commitment and are under no technical obligation to read or respect the instruction.

Australian context

A robots.txt file is a technical signal, not a legal control, and Australian firms should not treat it as a substitute for copyright, contractual or confidentiality protections. Copyright in published reports, technical notes and guidance documents arises under the Copyright Act 1968 (Cth) regardless of whether a robots.txt entry exists. In October 2025, the Australian Government said it would not introduce a text and data mining exception into Australian copyright law, a position that was welcomed by creative industries and publishers. The new directive does not alter that legal position. It gives publishers a clearer way to state their preference to the developers who have agreed to listen.

For environmental consultancies in particular, much of the most sensitive material about a site does not sit on the consultant’s own website. State regulators maintain public registers that hold notices, audit outcomes and site records. Examples include the NSW EPA record of notices under the Contaminated Land Management Act 1997, the Queensland Environmental Management Register and Contaminated Land Register under the Environmental Protection Act 1994, site audit information held by EPA Victoria under the Environment Protection Act 2017, and the South Australian EPA public register under the Environment Protection Act 1993. A consultancy’s robots.txt file has no bearing on how those registers are crawled. Any information governance review should acknowledge that the directive only covers the domains a firm controls.

Privacy is a separate consideration. Where web pages contain personal information, such as staff profiles, project contacts or landholder details in case studies, the Privacy Act 1988 (Cth) and the Australian Privacy Principles continue to apply to how that information is collected, used and disclosed. Opting out of AI training through robots.txt does not change those obligations. Firms should still decide what personal information belongs on a public website in the first place, because a crawler instruction cannot recall information once it has been published or collected.

What firms can do now

Firms that want to use the new directive can take a few practical steps while the standard is finalised:

• Review the current robots.txt file and record which crawlers are already blocked or allowed.
• Keep existing entries for individual crawlers such as GPTBot, ClaudeBot and Google-Extended until all four developers have implemented the shared directive.
• Decide which folders, if any, should remain open to AI training, such as media material or general service descriptions.
• Document the decision and who approved it, so it can be revisited if the IETF changes the syntax or scope.
• Keep confidential client reports and sensitive project material off public web pages, since robots.txt only applies to developers that choose to respect it.

References and related sources

• Primary source: www.engadget.com
• dehub.io
• ground.news
• https://www.engadget.com/2303225/google-openai-anthropic-ai-crawling-controls/`
• NEPM Assessment of Site Contamination

How iEnvi can help

iEnvi integrates technology and data-driven approaches into environmental consulting. We monitor AI and technology developments that affect how environmental professionals deliver services to clients.

• Our capabilities
• About iEnvi
• Talk to the team

---

This is an iEnvi Machete news summary. Prepared by iEnvi to summarise the source article for environmental professionals tracking AI, data, and technology developments that affect consulting and project delivery.

Published: 03 Nov 2026

Need advice on this topic? Speak to an iEnvi expert at info@ienvi.com.au or 1300 043 684, or contact us online.