Make Your Site Show Up in AI Search
I asked Claude Code to scrape my own website this week and the scraper got blocked at the door. Your site is probably invisible to AI search the same way, and there is a 15 minute fix.
Ryan FrizellePublished May 21, 2026What you'll walk away with
- The prompt that finds every block keeping AI crawlers out
- The robots and llms files your site is probably missing
- A quick check that ChatGPT, Claude and Perplexity can read you
Turns out my own hosting settings were quietly blocking every AI crawler from reading the site. If ChatGPT, Claude, and Perplexity cannot read you, they will never recommend you, no matter how good your site is.
The fifteen minute fix
Here is what you will do:
- 1Ask Claude Code to scan your live site and check if AI crawlers can actually read it.
- 2Have Claude fix every block in one shot. Your robots.txt, your llms.txt, your hosting rules, all of it.
- 3Verify ChatGPT, Claude, and Perplexity can now pull from your site.
The audit prompt
I want my website to actually show up in AI searches like ChatGPT, Claude, Perplexity, and Google AI Overviews. Right now I am not even sure if AI crawlers can read my site. Here is what I need you to do.
1. Try to fetch my live site at [PASTE YOUR DOMAIN HERE] as if you were an AI crawler. Tell me if anything is blocked, returning an error, or missing.
2. Check my robots.txt file (or create one at the root of my project if I do not have one) and confirm every major AI user agent is explicitly allowed. That means GPTBot (OpenAI), ClaudeBot and Claude-Web (Anthropic), PerplexityBot, Google-Extended (Google's AI crawler), CCBot (Common Crawl), Applebot-Extended, and Bytespider. If any are missing, add them.
3. Create or update an llms.txt file at the root of my site. This is the file AI tools look for to understand who I am and what my site is about. Include a clean summary of my name, what I do, and links to my most important pages.
4. Check my hosting setup for any firewall rules, Cloudflare bot protection, or middleware that might be blocking AI scrapers. If you find any, walk me through allowing AI bots through.
5. After everything is in place, scan the site one more time and confirm AI can now read it end to end.
Walk me through every step. Explain what each file does in plain English so I learn the why. I am not a developer.The Claude Code Course
Liked this? There's 50x more in the course.
- Getting Started setup guide + prompts
- Quick Wins library (growing monthly)
- Step-by-step website and dashboard builds
- All future sections + updates forever
$47 once. Keep forever. Full refund within 14 days.
What the prompt catches
That one prompt is the whole fix. Claude scans your live site like an AI crawler would. Finds every block, every misconfigured file, every firewall rule that is silently shutting AI out. Then it writes the right robots.txt and llms.txt for your site and walks you through any hosting fixes.
The bots you want in
Here are the AI bots you actually want crawling your site:
- 1GPTBot. OpenAI's crawler. This is what feeds ChatGPT search results.
- 2ClaudeBot. Anthropic's crawler. Feeds Claude, plus Claude Code when it browses for you.
- 3PerplexityBot. Perplexity's crawler. The fastest growing AI search engine right now.
- 4Google-Extended. Google's separate AI crawler for Gemini and AI Overviews in search.
- 5CCBot. Common Crawl. The open dataset behind a huge chunk of every AI model training.
Check that it worked
Test that my robots.txt and llms.txt are live and correct. Fetch https://[YOUR DOMAIN]/robots.txt and https://[YOUR DOMAIN]/llms.txt directly. Confirm both files load, list every AI user agent that is currently allowed, and flag anything missing. Then simulate a request from GPTBot, ClaudeBot, and PerplexityBot to my homepage and tell me if each one gets a clean response.Why this compounds
That is the whole thing. Run the audit. Let Claude fix the blocks. Verify it works. Now when someone asks ChatGPT or Claude about your business, your site is actually in the answer. Not even exaggerating.. this is a 15 minute fix that pays you back for years.
Want more 15 minute fixes like this that turn Claude Code into your personal site operator?
The Claude Code Course
Liked this? There's 50x more in the course.
- Getting Started setup guide + prompts
- Quick Wins library (growing monthly)
- Step-by-step website and dashboard builds
- All future sections + updates forever
$47 once. Keep forever. Full refund within 14 days.