Good news!
Were previous, similar measures successful against webcrawlers?
"Google, Microsoft, Anthropic and OpenAI announced they will support a crawling standard that lets site operators block all generative AI training scrapers with one robots.txt entry. Today, a site must add a separate entry for every AI provider, and for each scraper a provider runs.
The companies have submitted a draft to the Internet Engineering Task Force, which maintains the Robots Exclusion Protocol (the format robots.txt follows), and the setting would not cover models that perform only non-generative tasks.
Microsoft also plans an AI Grounding setting that blocks requests made to answer a user’s question rather than to gather training data, including from third-party tools built on Microsoft technology, and OpenAI plans similar controls.
Microsoft and OpenAI expect to add support in the coming months and Google sometime next year, while Meta has voiced support for the idea but has not committed to implementing it.
Until the standard is final and shipping, site owners who want to opt out of AI training still need per-crawler entries, and the pending AI Grounding controls suggest training and answer-time access may soon be blockable separately."
Google, Anthropic and others to support new AI crawling opt-out standard
No comments:
Post a Comment