Controlling how AI crawlers access your site · Local AI Visibility Central

Controlling how AI crawlers access your site

Hi everyone,

Just logged into my Cloudflare account (after a very long time away from it) to add a new client's site. I'm seeing a new setting regarding AI crawlers in the robots.txt file. (see attached)

I'm trying to get a consensus opinion on the pros and cons of at least some blocking (full or partial) AI crawlers, for (in this case) a local contractor site.

Thanks

2 comments