User-agent: * Crawl-Delay: 10 Disallow: /preview/ User-agent: CCBot User-agent: GPTBot User-agent: anthropic-ai Disallow: /