January 31, 2026 ยท 12 min read
Before AI can recommend you, it needs to find and understand your content. This technical checklist ensures nothing blocks AI discovery.
Crawlability Checklist
- robots.txt allows AI crawlers (GPTBot, anthropic-ai, PerplexityBot)
- No critical content behind JavaScript-only rendering
- XML sitemap is current and submitted
- Internal linking connects all important pages
- No orphan pages with valuable content
- Redirect chains are minimal (under 2 hops)
- 404 errors are fixed or redirected
AI Crawler Access
Check your robots.txt for these user agents:
# AI Crawlers to allow
User-agent: GPTBot
Allow: /
User-agent: anthropic-ai
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /Note: Blocking AI crawlers means AI systems can't learn about your content. Only block if you have specific reasons.
Structured Data Audit
- Organization schema on homepage
- Article/BlogPosting schema on content pages
- Product schema on product pages
- LocalBusiness schema for local businesses
- FAQPage schema for Q&A content
- BreadcrumbList for navigation
- Schema validates without errors
Content Accessibility
- Important content is in HTML, not images or PDFs
- Images have descriptive alt text
- Tables have proper headers
- Content hierarchy uses proper heading tags (H1, H2, H3)
- Lists use proper list markup
- No content hidden behind accordions or tabs by default
Page Speed and Performance
AI crawlers have time budgets. Slow sites get less thorough indexing.
- Time to First Byte under 200ms
- Largest Contentful Paint under 2.5s
- No render-blocking resources
- Images are optimized and lazy-loaded
- Server responds reliably (99.9%+ uptime)
Entity Signals
- Company name is consistent across all pages
- About page clearly describes what you do
- Author pages exist for content creators
- Contact information is easily findable
- Social profiles are linked and verified
Common Technical Blockers
JavaScript Rendering
AI crawlers may not execute JavaScript. Ensure critical content is in the initial HTML response.
Login Walls
Content behind logins can't be crawled. Consider making key content publicly accessible.
Infinite Scroll
Paginated content with proper URLs is better than infinite scroll for AI discovery.
Full Technical AI Audit
Get a comprehensive technical audit of your AI discoverability.
Request Audit