Community
Classroom
Calendar
Members
Leaderboards
About
Aug 13 • Algorithm Updates & SEO News
IMPORTANT! Update your robots.txt file on your website to allow crawling by the LLM platforms
Here is what you should have in your robots.txt file IF you want Googlebot, social and AI platforms to be able to crawl your site content/pages. This is super important for AEO!
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
User-agent: Twitterbot
Allow: /
User-agent: facebookexternalhit
Allow: /
User-agent: GPTBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: CCBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: anthropic-ai
Allow: /
User-agent: Applebot
Allow: /
User-agent: Applebot-Extended
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: DuckDuckBot
Allow: /
User-agent: LinkedInBot
Allow: /
User-agent: facebot
Allow: /
User-agent: *
Allow: /
Sitemap: https://targetdomain.com/sitemap.xml
2
0 comments
Lane Houk
5
IMPORTANT! Update your robots.txt file on your website to allow crawling by the LLM platforms
SEO Success Academy
skool.com/seo-success-academy
Welcome to SEO Success Academy – the ultimate destination for business owners, digital marketers and agencies to master the art and science of SEO.
Powered by