diff --git a/robots.txt.dist b/robots.txt.dist index 0a56bd6d356cd..c14fe769e0818 100644 --- a/robots.txt.dist +++ b/robots.txt.dist @@ -27,3 +27,9 @@ Disallow: /logs/ Disallow: /modules/ Disallow: /plugins/ Disallow: /tmp/ + +# GPTBot is OpenAI’s web crawler. This rule prevents the GPTBot from crawling +# the web site to use its content when building new models. + +User-agent: GPTBot +Disallow: /