Google joins OpenAI and Anthropic in promising new scraping controls

The new robots.txt directive will prevent the companies from scraping content for generative AI training.

By Mariella Moon Oct. 31, 2026 6:34 am EST tadamichi/Shutterstock Add Engadget on Google: Google Discover

Google, Anthropic, OpenAI and Microsoft have committed to honoring a new robots.txt directive, which will allow publishers to specify their preferences when it comes to the scraping of their content for generative AI training. The companies have decided to adopt a proposal under consideration at IETF, the open standards organization for the internet, written by Anthropic technical lead for standards Kevin Kelley.