robots.txt lines · only if AI crawlers are blocked
Let the assistants' crawlers in.
The exact lines to change, and what each one lets in. If nothing is blocked, we say so and leave it alone.
FILE example-accountants.co.uk/robots.txt
Before · as found
Two AI crawlers turned away
User-agent: *
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
After · the fix
Two lines changed, nothing else touched
User-agent: *
Allow: /
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
Illustrative example · not a real firm's file