Free · instant
Which AI crawlers does your robots.txt let in?
Enter a domain or paste a robots.txt. Every AI crawler token gets evaluated the way that crawler would read the file: its own group if it has one, otherwise the * group, longest rule wins.
or paste the file:
Enter a domain or paste a file.
The tokens
Search crawlers build the index an answer engine queries. User crawlers fetch a page because someone asked about it right now. Training crawlers collect corpora. Blocking training is a policy choice. Blocking search or user crawlers means you can't be cited by that assistant. Vendor documentation for each token is linked from the methodology.